In this lesson, you will learn how to distinguish between Keyed State and Operator State, the two fundamental ways to persist intermediate data across a distributed streaming topology.
The Problem of Distributed Memory
In a distributed streaming job, your logic is spread across multiple parallel worker nodes. If you need to "remember" information—like a running total or the last seen event—that data must be stored somewhere. Since nodes can fail and restart, this memory must be fault-tolerant. We classify this persistent memory into two categories based on how the data is scoped.
1. Keyed State
Keyed State is strictly bound to a specific key in your data (e.g., user_id or device_id). When you use functions like MapGroupsWithState, the engine automatically partitions the state by key.
- Behavior: The state is automatically redistributed if you change the parallelism of your job. Because it is partitioned by key, the system knows exactly which node holds the data for a specific user.
- Use Case: Ideal for user-centric calculations, such as tracking the number of items in a shopping cart for a specific customer.
2. Operator State
Operator State (also called Non-Keyed State) is scoped to the operator instance itself, regardless of the data values. It does not know anything about keys.
- Behavior: If you have an operator with a parallelism of 3, you have 3 distinct "buckets" of operator state. This state is not distributed by key, but rather by the instance index.
- Use Case: Perfect for global metadata or patterns that don't depend on specific keys. A classic example is the Broadcast State, where you maintain a list of rules or configuration filters that are pushed to all operator instances simultaneously.
Illustration
If you are calculating the "Average Session Duration," you would use Keyed State because the average depends on the individual user_id. However, if you need to maintain a list of "Blocked IP Addresses" to filter incoming traffic across the whole cluster, you would use Operator State (specifically Broadcast State), as the list is uniform across all nodes and not tied to individual event keys.
Key Takeaway: Use Keyed State when your data logic depends on specific identity keys to allow for scalable distribution, and use Operator State when you need to maintain global context that remains uniform across all parallel processing tasks.