Course contentsShow
← Data Science
Lesson 1838 of 2,14543. Data Pipelines and OrchestrationFree lesson

XComs and Passing Data Between Tasks

Using XComs to share small amounts of data, understanding their limitations for large datasets.

XComs and Passing Data Between Tasks

What you'll learn: How to share small pieces of data between Airflow tasks using XComs, and why they're not suitable for large datasets.

What Are XComs?

XComs (short for "cross-communications") are Airflow's built-in mechanism for tasks to pass messages or small amounts of data to downstream tasks. Think of XComs like sticky notes passed between coworkers in an office—perfect for quick messages like "file processed: success" or a filename, but terrible for handing over the entire manuscript.

How XComs Work

When a task returns a value or explicitly pushes data, Airflow stores it in its metadata database with a key. Downstream tasks can then pull this data by referencing the task ID and key. This creates loose coupling: tasks communicate through the orchestrator rather than directly.

Common use cases:

  • Passing file paths or S3 keys between tasks
  • Sharing small configuration values or counts
  • Communicating processing status or metadata
  • Returning API response IDs to subsequent tasks

Understanding the Limitations

XComs are stored in Airflow's metadata database (typically PostgreSQL or MySQL), which has strict size limitations. Most databases limit individual values to a few megabytes at most. More importantly, serializing and deserializing large objects through the database creates massive performance bottlenecks.

What NOT to pass via XComs:

  • DataFrames or large datasets
  • Model files or trained models
  • Large JSON responses or bulk data
  • Anything over a few kilobytes

For large data, use intermediate storage (S3, databases, file systems) and pass only the location via XCom.

Key Takeaway: XComs enable tasks to share small metadata like file paths or counts, but should never carry large datasets—always use external storage and pass references instead.