Course contentsShow
← Data Science
Lesson 1866 of 2,14543. Data Pipelines and OrchestrationFree lesson

Handling Failed Quality Checks

Strategies for pipeline behavior when checks fail: halt, quarantine, alert, or continue with warnings.

Handling Failed Quality Checks

You'll learn: How to configure pipeline behavior when data quality checks detect problems, choosing between stopping execution, quarantining bad data, sending alerts, or continuing with warnings.

The Problem

When a data quality check fails (like detecting null values in a required field or finding duplicates), your pipeline must decide what to do next. The wrong choice can either break critical downstream processes or let corrupt data silently pollute your systems.

Four Core Strategies

1. Halt (Fail Fast)

Stop the pipeline immediately. Use this for critical checks where proceeding would be dangerous—like a primary key violation or completely missing required data. This prevents cascading failures but blocks all downstream tasks.

2. Quarantine

Route failed records to a separate location (a "quarantine table" or file) while allowing clean records to proceed. Think of it like airport security: suspicious items get pulled aside for inspection while other passengers continue. This preserves good data and lets you investigate bad data later.

3. Alert and Continue

Log the failure, send notifications (email, Slack, PagerDuty), but let the pipeline finish. Use this for non-critical issues—like slightly higher-than-usual null rates—where manual review is needed but stopping isn't justified.

4. Continue with Warnings

Proceed silently but log the issue. Reserve this for informational checks during development or very minor quality concerns that shouldn't block production.

Choosing Your Strategy

Match severity to action: critical checks → halt, moderate issues → quarantine, minor concerns → alert, informational → warn. Often you'll combine strategies—quarantining bad records and alerting the team.

Key Takeaway: Design your pipeline's failure behavior intentionally based on data criticality; defaulting to either "always stop" or "always continue" creates operational risk or unnecessary downtime.