Routing Late Events to Side Outputs

Capture and redirect events that fall outside the allowed lateness threshold to secondary streams for auditing or manual reconciliation.

Routing Late Events to Side Outputs

In this lesson, you will learn how to preserve data integrity by capturing events that arrive after your watermark’s "allowed lateness" threshold has expired, redirecting them to a dedicated sink for manual investigation.

Why Drop Data When You Can Audit It?

When performing stateful aggregations (like windowing), you define a watermark to handle late-arriving events. Once an event arrives after the watermark plus your configured allowedLateness, the engine typically drops it to keep the state size manageable. However, in mission-critical pipelines, simply discarding these records can lead to reconciliation nightmares.

"Side outputs" (or side-logging) allow you to treat these "too late" records as a distinct stream. Instead of losing the data, you divert it to an external storage layer, such as a "dead-letter" table in your data lake or a secondary message queue. This enables your primary dashboard to remain performant while keeping an audit trail of anomalies.

The Mechanism

In practice, you implement this by checking the event timestamp against the watermark during your processing logic. While modern stream processing engines handle internal state expiration, a pattern for side outputs often looks like this:

val lateEvents = stream
  .filter(event => event.timestamp < (watermark - allowedLateness))

// Direct late events to a secondary sink for auditing
lateEvents.writeStream
  .format("delta")
  .option("checkpointLocation", "/path/to/late_data_checkpoint")
  .start("/path/to/audit_log")

// Proceed with your primary aggregation logic
val mainOutput = stream
  .filter(event => event.timestamp >= (watermark - allowedLateness))
  .groupBy(window($"time", "10 minutes"))
  .count()

Strategic Implementation

By separating your pipeline into a primary processing path and an audit path, you gain the ability to perform manual reconciliations. If you notice a high volume of events landing in your side output, it serves as a "canary in the coal mine," alerting you that your watermark settings may be too aggressive or that your upstream source is experiencing significant latency issues.

Key Takeaway: Routing late data to a side output prevents silent data loss and provides a necessary safety net for auditing records that exceed your windowing thresholds.