Course contentsShow
← System Design
Lesson 1026 of 1,91926. Idempotency and RetriesFree lesson

Retry on Which Errors

Identifying which errors are retryable (transient) versus non-retryable (permanent) to avoid wasting resources.

Retry on Which Errors

What you'll learn: How to distinguish between errors worth retrying (transient) and those that aren't (permanent), so you don't waste time and resources on doomed requests.

The Core Problem

Not all errors are created equal. When a distributed system call fails, you need to decide: should I retry this, or give up immediately? Retrying the wrong type of error wastes resources, delays failure detection, and can even make things worse.

Transient vs Permanent Errors

Transient errors are temporary hiccups that might succeed if you try again:

  • Network timeouts or connection resets
  • HTTP 503 Service Unavailable (server overloaded)
  • HTTP 429 Too Many Requests (rate limited)
  • Temporary database deadlocks
  • "Leader election in progress" messages

These are like calling a friend whose phone is busy — try again in a moment.

Permanent errors indicate something fundamentally wrong that won't fix itself:

  • HTTP 400 Bad Request (malformed input)
  • HTTP 401 Unauthorized (bad credentials)
  • HTTP 404 Not Found (resource doesn't exist)
  • Validation errors (invalid email format)
  • Business logic failures (insufficient funds)

These are like calling a disconnected number — retrying won't help.

The Strategy

Build an error classification system:

  • Retry: 5xx errors (except maybe 501), timeouts, connection failures, rate limits
  • Don't retry: 4xx errors (except 408 Request Timeout, 429 Too Many Requests)
  • Maybe retry with caution: 500 Internal Server Error (could be transient or a bug)

Some systems use allowlists (only retry known-safe errors) rather than denylists for extra safety.

Key Takeaway: Smart retry logic classifies errors as transient or permanent to avoid wasting resources on unrecoverable failures while giving temporary issues a chance to resolve.