Course contentsShow
← System Design
Lesson 255 of 1,9197. Database Scaling: ShardingFree lesson

Scatter-Gather Pattern

Learn how applications query all shards, gather results, and merge them for cross-shard operations.

Scatter-Gather Pattern

What You'll Learn

You'll learn how applications retrieve and merge data from multiple shards when a query spans across shard boundaries.

The Core Problem

In a sharded database, your data is split across multiple servers based on your shard key. But what happens when you need data that doesn't align with a single shard? For example, if you've sharded users by user_id but need to query "all orders placed last week" across all users?

This is where scatter-gather comes in.

How Scatter-Gather Works

The pattern has three phases:

1. Scatter

The application sends the same query to all relevant shards (or sometimes all shards if you can't determine which ones hold relevant data). Each shard processes the query independently against its local data.

2. Gather

The application collects partial results from each shard as they respond. Some shards might return results faster than others.

3. Merge

The application combines all partial results into a single, unified result set. This might involve sorting, filtering duplicates, or aggregating values.

Real-World Analogy

Think of searching for a book in a library system with multiple branches. You (the application) call all branches simultaneously asking "Do you have books about penguins?" Each branch checks its shelves independently (scatter), tells you what they found (gather), and you compile the complete list (merge).

The Performance Cost

Scatter-gather is expensive. Your query is only as fast as the slowest shard, and you're multiplying network calls and processing across all shards. This is why choosing a good shard key that minimizes cross-shard queries is so critical.

Key Takeaway: Scatter-gather allows cross-shard queries by sending requests to multiple shards and merging results, but it's significantly slower than single-shard queries—minimize its use through smart shard key selection.