The Stateful Nature of distinct
You'll learn why distinct() requires internal memory and how that affects stream processing behavior.
Why distinct Must Remember
Unlike operations like filter() or map() that process each element independently, distinct() has a problem: to know if an element is a duplicate, it must remember every element it has already seen. This makes it a stateful intermediate operation.
Think of it like a bouncer at an exclusive club checking IDs. The bouncer can't just look at each person in isolation—they need to maintain a mental (or written) list of everyone who's already entered. Without that memory, they'd let the same person in multiple times.
How Stateful Behavior Works
When you call distinct() on a stream, the operation internally maintains a Set (or similar collection) to track unique elements encountered so far. For each element flowing through:
- Check if it exists in the internal set
- If not present, add it to the set and pass it downstream
- If already present, skip it (don't pass downstream)
This internal storage accumulates until the stream completes.
Why This Matters
Memory consumption: If you call distinct() on a stream of 1 million elements, the operation might store up to 1 million unique values in memory.
Ordering sensitivity: In parallel streams, distinct() can be expensive because threads must coordinate to maintain a shared state of "seen" elements.
Pipeline position: Placing distinct() early in a pipeline (after narrowing filters) can reduce memory usage compared to placing it late.
Key Takeaway:
distinct() is a stateful operation that must internally track all unique elements seen so far, consuming memory proportional to the number of distinct values in the stream.