Real-Time Indexing and Updates
What you'll learn: How search systems incrementally add new documents to indexes without rebuilding everything from scratch.
The Challenge of Fresh Data
When a new blog post is published or a product is added to your catalog, users expect to find it immediately. But you've already built massive inverted indexes across distributed shards. Rebuilding those indexes for every new document would be impossibly slow and resource-intensive.
In-Memory Buffers: The Speed Layer
Real-time indexing uses a two-tier architecture:
-
In-memory buffer – New documents first go into a small, fast in-memory index structure. This buffer accepts writes quickly and can be queried immediately alongside the main index.
-
Main index on disk – The larger, durable index already built from existing documents.
Think of it like a restaurant: the main kitchen (disk index) handles bulk meal prep, while a small expediting station (memory buffer) handles last-minute orders that need to go out immediately.
Periodic Merges: Consolidation Strategy
When the in-memory buffer reaches a threshold (size or time-based), it's flushed to disk as a new index segment. Search queries must now look across:
- The main index segments
- Recently flushed segments
- The current in-memory buffer
Periodically, smaller segments are merged into larger ones to prevent having too many segments (which would slow query time). This merge process also removes deleted documents and optimizes storage.
Query-Time Orchestration
When a query arrives, the search system:
- Checks the in-memory buffer first
- Queries all disk segments in parallel
- Merges results using posting list intersections
- Returns unified rankings
Key Takeaway: Real-time indexing uses in-memory buffers for instant write acceptance and periodic segment merging to balance query performance with fresh content availability—enabling searches to see new documents within seconds rather than hours.