Course contentsShow
← Machine Learning and Deep Learning
Lesson 2872 of 3,53862. ML Pipelines and OrchestrationFree lesson

Airflow Operators for ML Workflows

BashOperator, PythonOperator, DockerOperator, KubernetesPodOperator, and custom operators for different execution environments.

Airflow Operators for ML Workflows

What you'll learn: How to use different Airflow operators to execute ML tasks in various computing environments.

Core Concept

Airflow operators are the building blocks that define what actually gets executed in each node of your DAG. Think of operators as specialized workers—some handle shell commands, others run Python functions, and some even orchestrate containerized workloads. For ML pipelines, choosing the right operator means matching your task's requirements (dependencies, resources, isolation) with the appropriate execution environment.

Common Operators for ML

BashOperator

Executes shell commands directly. Useful for calling CLI tools, running scripts, or simple file operations.

from airflow.operators.bash import BashOperator

preprocess_data = BashOperator(
    task_id='preprocess',
    bash_command='python preprocess.py --input data.csv'
)

PythonOperator

Runs Python functions within the Airflow worker. Best for lightweight tasks that don't need heavy dependencies.

from airflow.operators.python import PythonOperator

def train_model():
    # Your training logic here
    pass

train = PythonOperator(
    task_id='train_model',
    python_callable=train_model
)

DockerOperator

Executes tasks in isolated Docker containers. Perfect when you need specific dependencies or GPU access without polluting the Airflow environment.

KubernetesPodOperator

Launches tasks as Kubernetes pods. Ideal for scaling ML workloads across clusters, with fine-grained resource control (CPU, memory, GPUs).

Custom Operators

You can extend BaseOperator to create reusable operators for repeated patterns (e.g., a ModelTrainingOperator that handles your standard training workflow).

Key Takeaway: Choose operators based on your execution requirements—use BashOperator and PythonOperator for simple tasks, DockerOperator for dependency isolation, and KubernetesPodOperator for distributed, resource-intensive ML workloads.