Course contentsShow
← Machine Learning and Deep Learning
Lesson 2860 of 3,53862. ML Pipelines and OrchestrationFree lesson

Pipeline Orchestration vs Execution

Distinguish between orchestration (workflow coordination) and execution (running individual steps on compute resources).

Pipeline Orchestration vs Execution

What you'll learn: You'll understand the crucial distinction between coordinating workflow steps (orchestration) and actually running computations (execution) in ML pipelines.

The Core Distinction

When building ML pipelines, two separate concerns often get conflated: orchestration and execution.

Orchestration is the "conductor" of your pipeline—it decides when and in what order pipeline steps should run, manages dependencies between steps, handles retries on failure, and coordinates the overall workflow. Think of it as the control plane that says "data preprocessing is done, now trigger model training."

Execution is the "worker" that actually performs the computational work—running your preprocessing script on a Spark cluster, training your model on GPUs, or evaluating metrics. It's where the heavy lifting happens on actual compute resources.

Why This Matters

Separating these concerns provides crucial flexibility:

  • Your orchestrator (like Airflow, Kubeflow, or Prefect) can run on a lightweight VM, making scheduling decisions
  • Your execution environments can be heterogeneous: CPUs for data prep, GPUs for training, distributed clusters for inference
  • You can scale execution resources independently from orchestration logic
  • Failed steps can be retried without re-running the entire workflow coordination

A Real-World Analogy

Think of a restaurant: the head chef (orchestrator) coordinates the kitchen, deciding which dishes to prepare when and calling out orders. The line cooks (execution environments) actually prepare the food on their stations. The head chef doesn't cook everything themselves—they coordinate specialized workers with the right tools.

Key Takeaway: Orchestration manages workflow coordination and dependencies, while execution performs the actual computational work on appropriate resources. Keeping these separate enables scalable, maintainable ML pipelines.