Retry on Which Errors
What you'll learn: How to distinguish between errors worth retrying (transient) and those that aren't (permanent), so you don't waste time and resources on doomed requests.
The Core Problem
Not all errors are created equal. When a distributed system call fails, you need to decide: should I retry this, or give up immediately? Retrying the wrong type of error wastes resources, delays failure detection, and can even make things worse.
Transient vs Permanent Errors
Transient errors are temporary hiccups that might succeed if you try again:
- Network timeouts or connection resets
- HTTP 503 Service Unavailable (server overloaded)
- HTTP 429 Too Many Requests (rate limited)
- Temporary database deadlocks
- "Leader election in progress" messages
These are like calling a friend whose phone is busy — try again in a moment.
Permanent errors indicate something fundamentally wrong that won't fix itself:
- HTTP 400 Bad Request (malformed input)
- HTTP 401 Unauthorized (bad credentials)
- HTTP 404 Not Found (resource doesn't exist)
- Validation errors (invalid email format)
- Business logic failures (insufficient funds)
These are like calling a disconnected number — retrying won't help.
The Strategy
Build an error classification system:
- Retry: 5xx errors (except maybe 501), timeouts, connection failures, rate limits
- Don't retry: 4xx errors (except 408 Request Timeout, 429 Too Many Requests)
- Maybe retry with caution: 500 Internal Server Error (could be transient or a bug)
Some systems use allowlists (only retry known-safe errors) rather than denylists for extra safety.
Key Takeaway: Smart retry logic classifies errors as transient or permanent to avoid wasting resources on unrecoverable failures while giving temporary issues a chance to resolve.