Course contentsShow
← System Design
Lesson 1741 of 1,91942. Real-World System: Search and TypeaheadFree lesson

Search Latency and Response Time

Meeting sub-100ms latency requirements through index partitioning, caching, and efficient query execution.

Search Latency and Response Time

You'll learn: How modern search engines achieve sub-100ms response times through strategic partitioning, caching, and query optimization.

Why Speed Matters

When users type a query and hit enter, they expect results instantly. Research shows that delays beyond 100 milliseconds degrade user experience noticeably. For a system handling millions of queries per second, this is an enormous engineering challenge—especially when you're scanning billions of documents with TF-IDF scoring.

The Three-Pronged Strategy

Index Partitioning (Sharding)

Instead of one massive inverted index, split it across multiple servers. Each partition holds a subset of documents. Query routers send requests to all relevant shards in parallel, then merge results. Think of it like having 100 librarians searching different sections of a library simultaneously instead of one person checking every shelf.

Aggressive Caching

Cache frequently-accessed posting lists and pre-computed results for popular queries. If "weather today" is searched 10,000 times per hour, serve 9,999 of those from cache. Use tiered caching: hot queries in memory (Redis/Memcached), warm queries in SSD-backed stores.

Query Execution Optimization

  • Early termination: Stop scoring after finding the top K results
  • Skip pointers: Jump over irrelevant sections of posting lists
  • Pruning: Ignore low-scoring documents early using approximate scoring
  • Parallel processing: Score multiple documents simultaneously across CPU cores

The inverted index structure you've already learned enables these optimizations—its sorted posting lists allow binary search and efficient intersection.

Key Takeaway: Sub-100ms search latency requires horizontal scaling through index partitioning, multi-layer caching of hot queries and posting lists, and query execution tricks that avoid scoring every matching document.