Monitoring Resource Utilization via Task Callbacks

In this lesson, you will learn how to automate observability by triggering custom alerts and metrics reporting whenever a task succeeds or fails.

The Role of Task Callbacks

In complex workflows, simply relying on the Airflow UI to tell you if a task failed is often insufficient. To build a robust data platform, you need to push signals to external monitoring tools (like PagerDuty, Slack, or Datadog) the moment a task reaches its final state. Airflow facilitates this through on_success_callback and on_failure_callback.

These parameters allow you to pass a function that executes whenever the task finishes. The function receives a context dictionary containing the task instance, execution date, and even the captured logs, allowing you to extract performance metrics like duration or memory usage to push to your observability platform.

Implementation Pattern

The best practice is to define these functions in a central module so they can be reused across different DAGs. Here is how you might implement a simple failure alert:

def notify_failure(context):
    task_instance = context.get('task_instance')
    duration = task_instance.duration
    error_msg = context.get('exception')
    
    # Logic to send to observability platform
    print(f"Alert: Task {task_instance.task_id} failed after {duration}s. Error: {error_msg}")

# Using the callback in an operator
task_1 = PythonOperator(
    task_id='process_data',
    python_callable=my_func,
    on_failure_callback=notify_failure,
    on_success_callback=log_execution_time,
    dag=dag
)

Why This Matters

By embedding these callbacks, you transform your workflow from a "black box" into a proactive system. Instead of waiting for a stakeholder to report missing data, your pipeline notifies you of resource-heavy tasks or persistent failures immediately. You can capture metadata—such as the task_instance.duration or task_instance.try_number—to calculate long-term performance trends or identify tasks that are consistently pushing the boundaries of your resource allocation.

Key Takeaway: Task callbacks provide a centralized hook to push real-time health signals and execution metrics to your observability stack, enabling proactive management of your data pipeline's performance.