Course contentsShow
← System Design
Lesson 1731 of 1,91942. Real-World System: Search and TypeaheadFree lesson

Search Requirements and Scale Estimation

Estimating queries per second, document count, index size, and storage needs for a large-scale search system.

Search Requirements and Scale Estimation

What you'll learn: How to estimate the computational and storage demands of a large-scale search system before designing it.

Why Scale Estimation Matters

Before building a search engine, you need to size your infrastructure. A search system for 1,000 documents differs radically from one serving billions of queries daily. Understanding your scale dictates every architectural decision—from hardware to database choices to caching strategies.

Core Metrics to Estimate

Queries Per Second (QPS)
Start with daily active users and their search behavior. If 100 million users search 3 times daily, that's 300 million queries/day ≈ 3,500 QPS average. Peak traffic might be 3-5× higher, so plan for 10,000-15,000 QPS.

Document Count
How many web pages, products, or posts will you index? A mid-sized e-commerce site might have 10 million products. Google indexes hundreds of billions of documents. This number determines your index structure complexity.

Index Size
Each document produces an inverted index entry. A typical document might generate 100-500 index terms. If each term reference takes ~20 bytes (term ID + document ID + position), a 10M document corpus with 200 terms/doc averages 40GB just for the inverted index (10M × 200 × 20 bytes).

Storage Needs
You'll store: original documents, inverted indices, metadata, and caching layers. Apply a 3-5× multiplier over raw index size for redundancy, auxiliary structures, and growth buffer. Our 40GB index becomes 120-200GB minimum.

The Planning Process

Think like a city planner: count your population (users), estimate traffic (queries), measure land area (documents), and allocate infrastructure (storage/compute) accordingly. Underestimating by 10× means outages; overestimating wastes millions.

Key Takeaway: Scale estimation combines user behavior predictions with document corpus size to calculate QPS, index storage, and infrastructure capacity—the foundation for all subsequent search architecture decisions.