Building Social Media Feeds

Optimizing Document Retrieval for Linear Feed Aggregation

In this lesson, you will learn how to structure document models to ensure your social media feed renders instantly by minimizing compute-heavy joins at read time.

When building a social media feed in a document-oriented database, the "fan-out" problem is your biggest enemy. If a user follows 500 people, querying the database to find the latest post from each of those 500 sources and merging them in real-time will cause massive latency spikes. To build a truly performant linear feed, we move away from "pull" models (fetching data on demand) toward "push" models (pre-computing the feed).

The Denormalized Feed Approach

Instead of storing posts and followers separately, we create a dedicated collection for each user's feed. When an author publishes a post, the system "pushes" that post ID into the pre-computed feed document of every follower. This transforms a complex aggregation query into a simple, single-key lookup.

Consider this structure for a user's feed document:

{
  "user_id": "user_123",
  "feed_items": [
    { "post_id": "p_987", "author": "user_456", "timestamp": "2023-10-27T10:00:00Z" },
    { "post_id": "p_986", "author": "user_789", "timestamp": "2023-10-27T09:45:00Z" }
  ]
}

Why This Matters for Linear Feeds

By storing the necessary metadata (author ID and timestamp) directly inside the feed_items array, the application can display a scrollable list without needing to perform a secondary lookup for every single post. The document is retrieved as a contiguous block of data, which is ideal for the linear nature of a feed.

To keep this efficient:

1. Limit Size: Keep the feed_items array capped at a reasonable length (e.g., the 50 most recent items) to ensure the document doesn't exceed storage limits or memory buffers.
2. Capped Writes: Use a "push-to-head" operation so that the most recent content is always at the start of the array, allowing the database to read only the front of the document.

This approach trades storage space (duplicating post references) for read speed, which is a classic trade-off in high-scale social media systems.

Key Takeaway: By pre-computing feeds into localized, denormalized documents, you convert expensive cross-reference queries into a single, high-speed linear scan.