Implementing Idempotency in Writes

In this lesson, you will learn how to design application-level logic to ensure that processing the same event multiple times does not result in corrupted or duplicated data within your document store.

The Challenge of Distributed Retries

In distributed systems, networks are unreliable. Your application might send a write request to the database, but a network timeout prevents the acknowledgement from reaching you. You don't know if the write failed or if the database successfully processed it but the response was lost. Your default instinct might be to retry the write; however, if the first write actually succeeded, a blind retry will create a duplicate record or increment a counter twice.

The Idempotency Key Pattern

To handle this, we implement Idempotency Keys. Before sending a write request, the application generates a unique identifier for that specific operation (a UUID or a hash of the event contents). Instead of just inserting the document, we instruct the database to treat this key as a unique constraint or use an "upsert" operation.

Consider a system where you are processing payment events:

// Application logic: Only process if the eventId is unique
db.payments.updateOne(
  { eventId: "payment_tx_98765" },
  { 
    $set: { status: "processed", amount: 50.00 },
    $setOnInsert: { createdAt: new Date() }
  },
  { upsert: true }
);

Why This Works

By using eventId as a unique index in your document store, you turn a potentially destructive "insert" into a controlled "upsert." If the request is retried:

1. The first attempt succeeds: The document is created with the index payment_tx_98765.
2. The second attempt arrives: The database sees the unique index conflict. Instead of failing or creating a second document, the updateOne logic realizes the document already exists and either ignores the change or safely overwrites it with identical data.

This approach ensures that regardless of how many times you retry the request, the final state of the database remains consistent. By anchoring your write logic to a unique identifier provided by the event producer, you effectively decouple the "intent" of the write from the "execution" of the network call.

Key Takeaway: Always pair your write operations with a unique idempotency key and use upsert semantics to ensure that retrying a request has no additional side effects.