Course contentsShow
← System Design
Lesson 443 of 1,91912. NoSQL: Wide-Column StoresFree lesson

BigTable Overview and Motivation

Understanding Google's BigTable design goals: scalable structured storage for petabytes of data across thousands of machines.

BigTable Overview and Motivation

What You'll Learn

You'll understand why Google built BigTable and what design goals it aimed to achieve.

Why BigTable Was Created

In the early 2000s, Google faced a unique challenge: their services (Search, Gmail, Maps, Earth) generated petabytes of structured data that needed to be stored, accessed quickly, and scaled across thousands of commodity servers. Traditional relational databases couldn't handle this scale efficiently, and existing NoSQL solutions didn't meet their requirements.

Google needed a storage system that could:

  • Scale horizontally across thousands of machines without performance degradation
  • Handle sparse, semi-structured data where rows might have vastly different columns
  • Provide high throughput for both reads and writes
  • Support strong consistency at the row level
  • Automatically distribute and rebalance data as servers were added or failed

The BigTable Solution

BigTable was designed as a distributed, scalable structured storage system built on top of Google's infrastructure (GFS for storage, Chubby for coordination). Think of it as a massive, sorted map distributed across a cluster—simple enough to be flexible, yet powerful enough to handle Google-scale workloads.

Rather than forcing data into rigid tables with fixed schemas, BigTable allows dynamic columns grouped into families. Each row can have completely different columns, making it perfect for web crawling data, user profiles, and time-series logs where structure varies significantly.

The key innovation: BigTable treats storage and computation as separate, commodity resources that can scale independently.

Key Takeaway: BigTable was motivated by the need for a storage system that could handle petabytes of sparse, structured data across thousands of machines while maintaining high performance and automatic failover—solving problems traditional databases couldn't address at Google's scale.