Course contentsShow
← Cloud and Infrastructure
Lesson 2650 of 2,98957. Serverless Data ProcessingFree lesson

ADF Pipeline Activities and Control Flow

Building pipelines with copy activities, transformations, and control flow constructs like loops and conditionals.

ADF Pipeline Activities and Control Flow

You'll learn how to assemble Azure Data Factory pipelines using copy activities, transformations, and control flow logic to orchestrate complex data workflows.

What Are Pipeline Activities?

In Azure Data Factory, activities are the building blocks that perform actual work. Think of a pipeline as a recipe: activities are the individual steps like "mix ingredients" or "bake for 30 minutes." Common activity types include:

  • Copy Activity: Moves data between sources and destinations (databases, files, APIs)
  • Data Flow Activity: Executes visual transformations on data at scale
  • Stored Procedure Activity: Runs SQL logic in databases
  • Web Activity: Calls external REST endpoints

Each activity has inputs, outputs, and dependencies that define execution order.

Control Flow Constructs

Control flow activities let you build intelligent, dynamic pipelines rather than just linear sequences. Key constructs:

  • ForEach Loop: Iterate over a collection (e.g., process files in an array)
  • Until Loop: Repeat activities until a condition is met
  • If Condition: Branch execution based on expressions (e.g., if file exists, copy it)
  • Switch: Multi-way branching like a case statement
  • Wait: Pause pipeline execution for a specified duration

These work like programming logic but in a visual, declarative pipeline designer.

Building Orchestrated Workflows

Combine activities with control flow to handle real-world scenarios: check if yesterday's data arrived (If Condition), loop through multiple datasets (ForEach), apply transformations (Data Flow), then copy results to a warehouse (Copy Activity). Dependencies between activities create the execution graph—activity B waits for activity A to succeed before starting.

Key Takeaway: ADF pipeline activities perform discrete work units (copy, transform, execute), while control flow constructs (loops, conditionals) add programming-like logic to orchestrate complex, dynamic data processing workflows.