Course contentsShow
← System Design
Lesson 775 of 1,91920. Data Pipelines and ETLFree lesson

Apache NiFi for Data Flow

Learn NiFi's visual dataflow design, processors, and guaranteed delivery for data ingestion pipelines.

Apache NiFi for Data Flow

You'll learn: How Apache NiFi uses a visual dataflow design to reliably ingest and route data with guaranteed delivery.

What is Apache NiFi?

Apache NiFi is a visual data pipeline tool that lets you design, control, and monitor data flows through drag-and-drop. Think of it as "wiring diagrams for data" — you connect boxes (processors) on a canvas to move data from sources to destinations, transforming it along the way.

Unlike writing ETL code in Spark or Airflow DAGs, NiFi's web UI lets non-programmers build complex pipelines by connecting pre-built components.

Core Components

Processors

These are the building blocks — reusable units that perform specific tasks:

  • GetFile / PutFile: Read/write local files
  • ConsumeKafka / PublishKafka: Integrate with Kafka
  • InvokeHTTP: Call REST APIs
  • QueryRecord: Transform data formats (JSON, CSV, Avro)

You chain processors together, and data flows between them as FlowFiles (metadata + content wrapper).

Visual Dataflow Design

The drag-and-drop canvas shows your entire pipeline at a glance. You connect processors with relationships (success, failure, retry), making error handling and routing explicit. This visual model makes troubleshooting easier — you see exactly where data bottlenecks or fails.

Guaranteed Delivery

NiFi tracks every FlowFile through the system using a write-ahead log (provenance repository). If a processor crashes mid-processing, NiFi automatically retries or routes failures to designated paths. Data isn't lost — it's persisted until successfully delivered, similar to Kafka's acknowledgment mechanisms.

When to Use NiFi

Best for ingestion pipelines where you need to reliably collect data from diverse sources (sensors, APIs, logs) and route it to destinations (data lakes, Kafka). Its strength is flexibility and visibility, not heavy computation — leave big transformations to Spark or Flink.

Key Takeaway:
Apache NiFi provides visual dataflow design with drag-and-drop processors and built-in guaranteed delivery, making it ideal for building flexible, monitored data ingestion pipelines.