Clock Skew and Timeout Drift
You'll learn why relying on absolute timestamps for timeouts across distributed services can fail, and how to use elapsed time instead.
The Clock Synchronization Problem
Even with NTP (Network Time Protocol), clocks on different machines are never perfectly synchronized. A service might think it's 2:00:00.500 PM while another believes it's 2:00:00.300 PM — a 200ms difference. This is clock skew.
When you propagate a timeout as an absolute deadline (e.g., "request must complete by 2:00:01.000 PM"), each service interprets that timestamp using its own clock. If Service B's clock is 200ms behind Service A's, B thinks it has 200ms more time than A intended to give it.
Why Elapsed Time Wins
Instead of saying "complete by timestamp X," communicate "you have Y milliseconds remaining." Each service:
- Receives the remaining time budget
- Tracks how much time it uses locally
- Subtracts its processing time
- Passes the reduced budget downstream
Analogy: Instead of telling a relay team "finish by 3:00 PM," you tell each runner "you have 10 seconds left." The runner's watch might be slightly wrong, but they're measuring duration, not checking an absolute time.
Timeout Drift
Even with elapsed time, small measurement errors accumulate. If each hop overestimates its processing time by 5ms, a 10-hop chain "loses" 50ms from the budget. This is timeout drift. Monitor actual vs. budgeted time and tune accordingly.
Key Takeaway: Use elapsed time (remaining milliseconds) rather than absolute timestamps when propagating timeouts across services. Clock skew makes absolute deadlines unreliable, while duration-based budgets work regardless of clock differences.