Change Data Capture (CDC)

In this lesson, you will learn how to use database change logs to stream real-time document modifications to downstream services, ensuring your entire ecosystem stays in sync.

The Problem: Decoupling Data Changes

In high-scale systems, your document database (like MongoDB) is often just one part of a larger architecture. If you update a user’s profile document, you might need to trigger a cache invalidation, update a search index (like Elasticsearch), or notify a microservice. Rather than manually writing "side-effect" code inside your application layer—which is error-prone and can fail—you can use Change Data Capture (CDC).

How CDC Works: The Oplog

Modern document stores maintain an internal, append-only log of every operation that modifies the state of the data (often called an oplog or change stream). CDC acts as a listener that tails this log. Whenever a document is inserted, updated, or deleted, the database emits an event containing the modification details.

For example, if you update a product document, the stream might emit an event like this:

{
  "operationType": "update",
  "documentKey": { "_id": "prod_123" },
  "updateDescription": {
    "updatedFields": { "price": 19.99 },
    "removedFields": []
  }
}

Downstream Synchronization

By subscribing a separate process (a "sink connector" or custom worker) to this stream, you can propagate the change to other systems asynchronously. This ensures that even if your application service crashes mid-operation, the change log remains the "source of truth."

Because the stream only provides the delta (the specific fields changed), your downstream services can apply targeted logic—like updating just the price in a cache—rather than re-syncing the entire document. This approach is highly efficient and decouples your database performance from the complexity of your integration logic. It effectively transforms your database from a static data store into an active event producer, perfect for building reactive, event-driven architectures.

Key Takeaway: CDC leverages the internal database change log to broadcast document mutations as events, enabling reliable, asynchronous synchronization between your database and downstream systems.