Course contentsShow
← System Design
Lesson 703 of 1,91919. Event Streaming PlatformsFree lesson

Partitioning Strategies and Key Selection

Explore partitioning algorithms (hash, round-robin, custom) and how key selection impacts message ordering and distribution.

Partitioning Strategies and Key Selection

What you'll learn: How choosing partition keys and algorithms affects message distribution and ordering in event streaming platforms.

When you publish messages to a topic with multiple partitions, the streaming platform needs a strategy to decide which partition receives each message. This decision has huge implications for throughput, ordering guarantees, and load distribution.

Common Partitioning Algorithms

Hash Partitioning
The producer extracts a key from the message and applies a hash function to determine the target partition. For example, partition = hash(key) % partition_count. Messages with the same key always go to the same partition, preserving ordering for that key.

Round-Robin Partitioning
Messages without keys are distributed evenly across partitions in rotation. This maximizes throughput and load balancing but provides no ordering guarantees between messages.

Custom Partitioning
You can implement your own logic based on business rules—perhaps routing all "premium user" events to specific partitions or distributing geographically.

The Power of Key Selection

The key you choose determines your ordering guarantees. If you key messages by user_id, all events for the same user land in the same partition in order. If you key by product_id, product-related events stay ordered. Choose incorrectly—like using random keys—and you lose ordering entirely.

Keys also affect distribution. If one customer generates 80% of your events and you key by customer, that partition becomes a hotspot while others sit idle.

Practical Tradeoffs

  • No key (null): Maximum parallelism, no ordering
  • Natural keys (user_id, order_id): Per-key ordering, risk of skew
  • Composite keys: Balance distribution and ordering needs

Key Takeaway: Your partition key choice determines both message ordering (same key = same partition = ordered) and load distribution—pick keys that align with your ordering requirements while avoiding hotspots.