Secondary Indexes in Partitioned Systems
What you'll learn: How distributed databases maintain indexes when data is spread across multiple shards, and the trade-offs between local and global indexing strategies.
When you partition data across multiple nodes, querying by the partition key is straightforward—you know exactly which shard to check. But what if you need to search by another attribute, like finding all users with status = "active" when your partition key is user_id? This is where secondary indexes come in.
Two Indexing Approaches
1. Local Indexes per Shard (Document-Partitioned)
Each shard maintains its own secondary index covering only the documents it stores.
Analogy: Imagine a library with multiple buildings. Each building has its own card catalog, but only for the books in that building.
- Writes: Fast! You only update one shard's index.
- Reads: Slow. You must query all shards (scatter-gather), then combine results. Even if only a few documents match globally, you still contact every partition.
2. Global Partitioned Indexes (Term-Partitioned)
The secondary index itself is partitioned across nodes, independent of how the primary data is partitioned. For example, terms A-M might be indexed on Node 1, N-Z on Node 2.
Analogy: A centralized library catalog system where different letters of the alphabet are managed by different catalog servers.
- Reads: Fast! You know which index partition to query based on the search term.
- Writes: Slower and more complex. A single document write may update indexes on multiple nodes, requiring distributed transactions or accepting eventual consistency.
The Trade-off
Local indexes optimize writes but penalize reads with scatter-gather queries. Global indexes optimize reads but complicate writes with cross-partition updates.
Key Takeaway: In partitioned systems, local indexes keep writes simple but require querying all shards; global partitioned indexes speed up reads but make writes distributed operations—choose based on your read/write patterns.