Course contentsShow
← System Design
Lesson 737 of 1,91920. Data Pipelines and ETLFree lesson

What is Stream Processing?

Stream processing handles continuous data flows in real-time, prioritizing low latency over absolute throughput.

What is Stream Processing?

You'll learn: How stream processing enables systems to analyze and act on data continuously as it arrives, rather than waiting to collect it in batches.

The Core Idea

Stream processing is about handling data in motion—processing records as they flow through your system rather than storing them first and analyzing them later. Unlike batch processing (which waits to accumulate data before running), stream processing treats every incoming event as something to act on immediately.

Think of it like watching a live sports game versus watching the highlights at the end. Stream processing is the live commentary—reacting to every play as it happens, even if you miss some nuances. Batch processing is the post-game analysis—more thorough but hours behind.

Characteristics of Stream Processing

Low Latency Priority: Results appear in milliseconds or seconds, not hours. The goal is fast reaction, not perfect completeness.

Continuous Computation: Your processing logic runs continuously, waiting for the next event rather than starting and stopping on a schedule.

Unbounded Data: Streams have no defined "end"—they keep flowing. Your system must handle data that never stops arriving.

Windowing: Since streams are infinite, you often analyze chunks of time (e.g., "events in the last 5 minutes") rather than "all events ever."

When Stream Processing Matters

Use stream processing when the freshness of insights matters more than absolute accuracy or completeness. Real-time fraud detection, live dashboards, immediate notifications, and instant search indexing all benefit from processing events as they arrive rather than waiting for batch jobs.

Key Takeaway: Stream processing sacrifices some throughput efficiency to achieve near-instant results by analyzing continuous data flows immediately upon arrival.