Course contentsShow
← System Design
Lesson 1556 of 1,91938. Real-World System: PastebinFree lesson

Hybrid Storage: Metadata + Content References

Store metadata in SQL database with pointers to S3/blob storage for large paste content.

Hybrid Storage: Metadata + Content References

What You'll Learn

You'll discover how to efficiently split Pastebin data by storing metadata in a SQL database while keeping paste content in object storage with reference pointers.

The Core Idea

In a hybrid storage architecture, we separate what we query frequently from what we store in bulk. For Pastebin:

  • SQL Database holds paste metadata: paste_id, created_at, expiration_time, user_id, language, views, and crucially, a reference pointer to where the actual content lives
  • Object Storage (S3/Blob) stores the actual paste content using the paste ID as the key

Think of it like a library catalog system: the database is your card catalog with book titles, authors, and shelf locations—small, searchable records. The object storage is the warehouse of actual books. You search the catalog quickly, then use the shelf location to fetch the book.

Why This Split?

Relational databases excel at:

  • Indexed queries (find all pastes by user)
  • Range queries (expiration checks)
  • Transactional updates (view counts)

Object storage excels at:

  • Storing large blobs cheaply
  • High throughput for reads/writes
  • Durability and replication

Implementation Flow

When creating a paste:

  1. Generate unique paste_id
  2. Upload content to S3 with key paste_id
  3. Store metadata row with content_location = "s3://bucket/paste_id"

When retrieving:

  1. Query database for metadata by paste_id
  2. Use the content_location reference to fetch from S3
  3. Return both metadata and content to user

Key Takeaway: Hybrid storage leverages SQL for queryable metadata and object storage for cost-effective bulk content, connecting them via reference pointers—optimizing both access patterns and costs.