Cross-Partition Queries
You'll learn: Why fetching data from multiple partitions simultaneously is costly and how to reduce that overhead.
The Core Problem
When your data is distributed across partitions (or shards), some queries need information from only one partition—these are local queries and they're fast. But when a query needs data from multiple partitions—a cross-partition query—the system must:
- Route requests to multiple nodes
- Wait for all responses (limited by the slowest node)
- Merge/aggregate results
- Return the combined answer
This is expensive because you're paying network latency per partition touched, and the query latency becomes the max of all individual partition response times, not the average.
Real-World Analogy
Imagine asking "Who has read more than 100 books?" in a library system where books are partitioned by genre. You must ask every partition, wait for all answers, then combine them. Compare this to "Show me sci-fi readers with 100+ books"—you only ask the sci-fi partition.
Optimization Strategies
Design to avoid them:
- Choose partition keys matching your most common query patterns
- Denormalize data so related information lives together
- Cache aggregated results
When unavoidable:
- Use scatter-gather with timeouts
- Implement result streaming instead of waiting for all partitions
- Pre-compute cross-partition aggregates asynchronously
- Use secondary indexes (though they introduce their own cross-partition complexity)
- Limit fan-out with query constraints (date ranges, top-N limits)
Key Takeaway: Cross-partition queries multiply network latency and coordination overhead; optimal partitioning strategies align data placement with query access patterns to keep most queries single-partition.