In this lesson, you will learn how to use distributed tracing to visualize the flow of data tasks across your infrastructure and identify precisely where latency bottlenecks occur.
While standard logging tells you that a task failed or succeeded, distributed tracing tells you why a workflow might be sluggish by tracking a single "trace" or request as it travels through various executors, workers, and external systems. In a complex Airflow architecture—especially one using a KubernetesExecutor or remote task workers—a single DAG run might involve communication between the Scheduler, the Celery/Kubernetes broker, and multiple worker nodes.
Distributed tracing assigns a unique Trace ID to every workflow execution. As a task is dispatched, the orchestrator injects this ID into the metadata. When a worker receives the task, it carries that ID forward. By visualizing these spans in tools like Jaeger or Honeycomb, you can spot "gaps" in the timeline. For instance, if you notice a 30-second delay between "Task Scheduled" and "Task Running," you aren't looking at a slow data process; you are looking at an infrastructure bottleneck, such as worker autoscaling latency or pool contention.
Consider this conceptual snippet of how you might wrap a custom operator to emit span data:
from opentelemetry import trace
tracer = trace.get_tracer(__name__)
def execute(self, context):
with tracer.start_as_current_span("custom_task_execution") as span:
span.set_attribute("dag_id", context['dag'].dag_id)
# Perform your data transformation here
result = self.process_data()
span.set_status(trace.StatusCode.OK)
return result
By adding these spans to your operators, you gain a continuous timeline of infrastructure health. You can distinguish between time spent waiting for a free slot in a Resource Pool, time spent pulling a Docker image for a KubernetesPodOperator, and actual execution time. Instead of guessing why a pipeline is running slow, you can pinpoint whether the issue is internal compute resource exhaustion or an external network delay during a database call.
Key Takeaway: Distributed tracing transforms opaque workflow logs into a unified timeline, allowing you to isolate infrastructure-level latency from data processing delays.