Idempotency Keys in Distributed Systems

In this lesson, you will learn how to use idempotency keys to ensure that retrying a failed network request does not result in duplicate state changes on your server.

The Problem: The "At-Least-Once" Trap

In distributed systems, network failures are inevitable. If a client sends a payment request but the connection drops before receiving a confirmation, the client doesn't know if the server processed the payment. If the client simply retries the request, and your server isn't designed for idempotency, you risk charging the customer twice.

The Solution: Idempotency Keys

An idempotency key is a unique identifier (usually a UUID) generated by the client and sent in an HTTP header (e.g., Idempotency-Key: <UUID>). The server uses this key to check if a request has already been successfully processed.

Here is a simplified approach using a database to track these keys:

async function processPayment(req, res) {
  const idempotencyKey = req.headers['idempotency-key'];

  // 1. Check if we have already processed this specific key
  const existingResult = await db.idempotency.findOne({ key: idempotencyKey });
  if (existingResult) {
    return res.status(200).json(existingResult.response);
  }

  // 2. Process the operation (e.g., charge the credit card)
  const result = await paymentGateway.charge(req.body.amount);

  // 3. Store the result associated with the key
  await db.idempotency.insert({ 
    key: idempotencyKey, 
    response: result,
    createdAt: new Date() 
  });

  res.status(201).json(result);
}

Critical Considerations

• Atomicity: You should ideally store the processed result within the same database transaction as the business operation itself to prevent a "split-brain" scenario where the database records the payment but fails to record the idempotency key.
• Expiration: Storing every key forever is a memory leak. Use a TTL (Time-to-Live) index in your database (e.g., MongoDB or Redis) to automatically clean up keys after 24–48 hours.
• Concurrency: If two identical requests arrive at the exact same time, you may need a distributed lock or a UNIQUE constraint on the idempotency key column to ensure only one worker proceeds.

Key Takeaway: By requiring a client-generated idempotency key, you transform potentially dangerous "retry" logic into a safe operation where the server guarantees that a specific action happens exactly once, regardless of how many times the request is received.