Scatter-Gather Pattern
What You'll Learn
You'll learn how applications retrieve and merge data from multiple shards when a query spans across shard boundaries.
The Core Problem
In a sharded database, your data is split across multiple servers based on your shard key. But what happens when you need data that doesn't align with a single shard? For example, if you've sharded users by user_id but need to query "all orders placed last week" across all users?
This is where scatter-gather comes in.
How Scatter-Gather Works
The pattern has three phases:
1. Scatter
The application sends the same query to all relevant shards (or sometimes all shards if you can't determine which ones hold relevant data). Each shard processes the query independently against its local data.
2. Gather
The application collects partial results from each shard as they respond. Some shards might return results faster than others.
3. Merge
The application combines all partial results into a single, unified result set. This might involve sorting, filtering duplicates, or aggregating values.
Real-World Analogy
Think of searching for a book in a library system with multiple branches. You (the application) call all branches simultaneously asking "Do you have books about penguins?" Each branch checks its shelves independently (scatter), tells you what they found (gather), and you compile the complete list (merge).
The Performance Cost
Scatter-gather is expensive. Your query is only as fast as the slowest shard, and you're multiplying network calls and processing across all shards. This is why choosing a good shard key that minimizes cross-shard queries is so critical.
Key Takeaway: Scatter-gather allows cross-shard queries by sending requests to multiple shards and merging results, but it's significantly slower than single-shard queries—minimize its use through smart shard key selection.