Course contentsShow
← System Design
Lesson 1106 of 1,91928. Timeouts and DeadlinesFree lesson

Timeout Propagation Observability

Logging and tracing timeout values at each service hop to debug timeout-related failures and tune budgets.

Timeout Propagation Observability

You'll learn how to make timeout values visible across service hops so you can debug failures and tune timeout budgets effectively.

The Visibility Problem

When timeouts propagate through a chain of services, the values shrink at each hop—but if you can't see what's happening, you're flying blind. A request that started with 5 seconds might arrive at the 3rd service with only 200ms remaining, but without observability, you won't know why it timed out or where the budget was consumed.

Core Practice: Log and Trace Timeout Values

At each service hop, you need to:

  1. Log the incoming timeout — what budget did this service receive?
  2. Log the outgoing timeout — what budget did we pass to downstream services?
  3. Include timeout context in traces — attach these values to distributed tracing spans
  4. Record timeout exhaustion events — when a request times out, log how much time was actually used

Think of it like a relay race where each runner records their split time. You can't improve the team's performance without knowing where time is being lost.

What to Capture

  • incoming_timeout_ms: The deadline received from upstream
  • outgoing_timeout_ms: The deadline sent downstream
  • elapsed_ms: Time consumed at this service
  • remaining_ms: Budget left when passing to next hop
  • timeout_exhausted: Boolean flag if this hop caused the timeout

This data feeds directly into debugging: "Service B received only 150ms but needs at least 500ms for its database query" immediately points to upstream services consuming too much budget.

Key Takeaway: Logging and tracing timeout values at every service hop transforms opaque timeout failures into debuggable data trails, enabling precise tuning of per-hop timeout budgets.