Distributed Query Coordination
What You'll Learn
You'll learn how to orchestrate parallel queries across multiple index shards and merge their results while gracefully handling timeouts and failures.
The Scatter-Gather Pattern
When your search index is horizontally partitioned across multiple shards (as covered in your earlier lessons), a single user query must fan out to all relevant shards simultaneously, then merge their results into a unified response.
Think of it like asking librarians in different library branches to each find the top 10 books matching "quantum physics." You wait for all responses, then combine and re-rank them to produce the final top 10.
Core Coordination Responsibilities
The query coordinator (or aggregator node) handles:
- Query Broadcasting: Send the identical search request to all shards in parallel
- Partial Result Collection: Each shard returns its local top-K results with scores
- Global Merge-Sort: Combine results from all shards, re-rank by score, and return the global top-K
- Timeout Management: Set aggressive per-shard timeouts (e.g., 100-200ms) to maintain p99 latency
- Partial Failure Handling: If one shard times out or fails, decide whether to return incomplete results or fail the query
Handling Failures Gracefully
Timeout strategy: Use circuit breakers per shard. If shard-3 repeatedly times out, temporarily exclude it and alert operators.
Degraded mode: Return results from N-1 shards with a warning flag rather than failing completely—users get 95% coverage instead of nothing.
Retry logic: For critical queries, retry failed shards once with a shorter timeout, but never block the response.
Monitoring Essentials
Track per-shard latency percentiles, timeout rates, and result contribution to detect hot shards or failing replicas early.
Key Takeaway: Distributed query coordination transforms multiple partial results into a unified response through parallel scatter-gather execution, aggressive timeouts, and graceful degradation when shards fail.