Airflow Operators for ML Workflows
What you'll learn: How to use different Airflow operators to execute ML tasks in various computing environments.
Core Concept
Airflow operators are the building blocks that define what actually gets executed in each node of your DAG. Think of operators as specialized workers—some handle shell commands, others run Python functions, and some even orchestrate containerized workloads. For ML pipelines, choosing the right operator means matching your task's requirements (dependencies, resources, isolation) with the appropriate execution environment.
Common Operators for ML
BashOperator
Executes shell commands directly. Useful for calling CLI tools, running scripts, or simple file operations.
from airflow.operators.bash import BashOperator
preprocess_data = BashOperator(
task_id='preprocess',
bash_command='python preprocess.py --input data.csv'
)
PythonOperator
Runs Python functions within the Airflow worker. Best for lightweight tasks that don't need heavy dependencies.
from airflow.operators.python import PythonOperator
def train_model():
# Your training logic here
pass
train = PythonOperator(
task_id='train_model',
python_callable=train_model
)
DockerOperator
Executes tasks in isolated Docker containers. Perfect when you need specific dependencies or GPU access without polluting the Airflow environment.
KubernetesPodOperator
Launches tasks as Kubernetes pods. Ideal for scaling ML workloads across clusters, with fine-grained resource control (CPU, memory, GPUs).
Custom Operators
You can extend BaseOperator to create reusable operators for repeated patterns (e.g., a ModelTrainingOperator that handles your standard training workflow).
Key Takeaway: Choose operators based on your execution requirements—use BashOperator and PythonOperator for simple tasks, DockerOperator for dependency isolation, and KubernetesPodOperator for distributed, resource-intensive ML workloads.