Course contentsShow
← System Design
Lesson 695 of 1,91919. Event Streaming PlatformsFree lesson

Stream Retention and Replay

Configurable retention policies allow consumers to replay historical events, enabling reprocessing and recovery.

Stream Retention and Replay

What you'll learn: How event streaming platforms let you keep historical events and "rewind" to replay them whenever needed.


The Core Idea

Unlike traditional message queues that delete messages after consumption, event streaming platforms retain events for a configurable period (hours, days, or even indefinitely). This creates a persistent, replayable history of everything that happened in your system.

Think of it like a DVR for your data: you can rewind and watch what happened yesterday, last week, or even last year. Any consumer can jump back to any point in the stream and process events from that moment forward.


How Retention Works

When events are written to a stream (commit log), they're stored on disk with a timestamp. You configure retention policies like:

  • Time-based: Keep events for 7 days, then delete
  • Size-based: Keep the last 100 GB of data
  • Indefinite: Never delete (storage permitting)

Consumers track their own position (offset) in the stream. To replay, a consumer simply resets its offset to an earlier point and starts reading again.


Why Replay Matters

Recovery: If a consumer crashes or has a bug, replay from before the problem occurred and reprocess correctly.

New consumers: Launch a new analytics service? It can replay months of historical events to build its initial state.

Testing and debugging: Replay production events in a test environment to reproduce issues safely.

Multiple perspectives: Different consumers can process the same events at different times for different purposes—one for real-time alerts, another for weekly reports.


Key Takeaway:
Configurable retention policies turn event streams into a replayable historical record, enabling recovery, reprocessing, and new consumer onboarding without data loss.