Apache Nifi | Aviation Data Flow | Data Trimming with JoltTransform Processor | Part 2

Please Subscribe for New Upload Notifications!

Resources: Nifi - https://nifi.apache.org/download.html

Aviationstack - https://aviationstack.com/

Apache NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Some of the high-level capabilities and objectives of Apache NiFi include:

Web-based user interface
    Seamless experience between design, control, feedback, and monitoring
Highly configurable
    Loss tolerant vs guaranteed delivery
    Low latency vs high throughput
    Dynamic prioritization
    Flow can be modified at runtime
    Back pressure
Data Provenance
    Track dataflow from beginning to end
Designed for extension
    Build your own processors and more
    Enables rapid development and effective testing
Secure
    SSL, SSH, HTTPS, encrypted content, etc...
    Multi-tenant authorization and internal authorization/policy management

What is Apache NiFi?

Put simply NiFi was built to automate the flow of data between systems. While the term 'dataflow' is used in a variety of contexts, we use it here to mean the automated and managed flow of information between systems. This problem space has been around ever since enterprises had more than one system, where some of the systems created data and some of the systems consumed data.

Processors

AttributeRollingWindow 
AttributesToCSV 
AttributesToJSON 
CaptureChangeMySQL 
CompareFuzzyHash 
CompressContent 
ConnectWebSocket 
ConsumeAMQP 
ConsumeAzureEventHub 
ConsumeEWS 
ConsumeGCPubSub 
ConsumeIMAP 
ConsumeJMS 
ConsumeKafka 
ConsumeKafka_0_10 
ConsumeKafka_0_11 
ConsumeKafka_1_0 
ConsumeKafka_2_0 
ConsumeKafkaRecord_0_10 
ConsumeKafkaRecord_0_11 
ConsumeKafkaRecord_1_0 
ConsumeKafkaRecord_2_0 
ConsumeMQTT 
ConsumePOP3 
ConsumeWindowsEventLog 
ControlRate 
ConvertAvroToJSON 
ConvertAvroToORC 
ConvertAvroToParquet 
ConvertCharacterSet 
ConvertExcelToCSVProcessor 
ConvertJSONToSQL 
ConvertRecord 
CountText 
DeleteDynamoDB 
DeleteGridFS 
DeleteHBaseCells 
DeleteHBaseRow 
DeleteHDFS 
DeleteMongo 
DeleteRethinkDB 
DeleteS3Object 
DeleteSQS 
DetectDuplicate 
DistributeLoad 
DuplicateFlowFile 
EncryptContent 
EnforceOrder 
EvaluateJsonPath 
EvaluateXPath 
EvaluateXQuery 
ExecuteGroovyScript 
ExecuteInfluxDBQuery 
ExecuteProcess 
ExecuteScript 
ExecuteSparkInteractive 
ExecuteSQL 
ExecuteSQLRecord 
ExecuteStreamCommand 
ExtractAvroMetadata 
ExtractCCDAAttributes 
ExtractEmailAttachments 
ExtractEmailHeaders 
ExtractGrok 

... https://www.youtube.com/watch?v=mGH2equ9amw