XCom for Inter-Task Communication
What you'll learn: How to pass data between Airflow tasks using XCom, its limitations, and when to switch to external storage.
What is XCom?
XCom (short for "cross-communication") is Airflow's built-in mechanism for sharing small pieces of data between tasks in a DAG. Think of it as a shared mailbox where one task can drop off a message and another task can pick it up later—perfect for passing metadata, file paths, or small results.
How XCom Works
When a task returns a value or explicitly pushes data using xcom_push(), Airflow stores it in its metadata database. Downstream tasks can retrieve this data using xcom_pull(). This creates a simple communication channel without needing external dependencies.
# Task 1: Push data
def train_model(**context):
accuracy = 0.95
context['ti'].xcom_push(key='model_accuracy', value=accuracy)
return accuracy # Also automatically pushed with key 'return_value'
# Task 2: Pull data
def evaluate_model(**context):
ti = context['ti']
accuracy = ti.xcom_pull(key='model_accuracy', task_ids='train_model_task')
print(f"Retrieved accuracy: {accuracy}")
Critical Limitations
XCom stores data in Airflow's database, which has strict size limits (typically 48KB in MySQL, 64KB in PostgreSQL). Trying to pass large objects like trained models or datasets will fail or degrade performance.
When to Use External Storage
For anything beyond simple metadata (like file paths, metrics, or model IDs), use external storage:
- Store trained models in S3/GCS and pass the URI via XCom
- Save datasets to cloud storage and exchange paths
- Use MLflow Model Registry and pass model version numbers
Key Takeaway: XCom is ideal for lightweight metadata like file paths, metrics, or IDs, but always use external storage (S3, GCS, model registries) for large artifacts like trained models or datasets to avoid size limits and performance issues.