XComs and Passing Data Between Tasks
What you'll learn: How to share small pieces of data between Airflow tasks using XComs, and why they're not suitable for large datasets.
What Are XComs?
XComs (short for "cross-communications") are Airflow's built-in mechanism for tasks to pass messages or small amounts of data to downstream tasks. Think of XComs like sticky notes passed between coworkers in an office—perfect for quick messages like "file processed: success" or a filename, but terrible for handing over the entire manuscript.
How XComs Work
When a task returns a value or explicitly pushes data, Airflow stores it in its metadata database with a key. Downstream tasks can then pull this data by referencing the task ID and key. This creates loose coupling: tasks communicate through the orchestrator rather than directly.
Common use cases:
- Passing file paths or S3 keys between tasks
- Sharing small configuration values or counts
- Communicating processing status or metadata
- Returning API response IDs to subsequent tasks
Understanding the Limitations
XComs are stored in Airflow's metadata database (typically PostgreSQL or MySQL), which has strict size limitations. Most databases limit individual values to a few megabytes at most. More importantly, serializing and deserializing large objects through the database creates massive performance bottlenecks.
What NOT to pass via XComs:
- DataFrames or large datasets
- Model files or trained models
- Large JSON responses or bulk data
- Anything over a few kilobytes
For large data, use intermediate storage (S3, databases, file systems) and pass only the location via XCom.
Key Takeaway: XComs enable tasks to share small metadata like file paths or counts, but should never carry large datasets—always use external storage and pass references instead.