Partial Failures and Fault Detection
You'll learn why it's fundamentally impossible to reliably tell whether a remote node has crashed or is just running slowly.
The Core Problem
In a distributed system, when you send a request to another node and don't get a response, you face an ambiguous situation: Is the remote node dead, or is it just slow? Is the network between you broken, or congested? Did your message get lost, or did the reply get lost?
This is called the partial failure problem—some components fail while others continue working, and you can't easily distinguish the scenarios.
Why Detection Is So Hard
Unlike a single machine where the operating system knows definitively when a process crashes, distributed systems have no such luxury. Consider:
- Slow node: Processing your request but taking 10 seconds instead of 100ms
- Dead node: Crashed entirely, will never respond
- Network partition: Node is fine, but you can't reach it
- Lost message: Your request never arrived, or the response got dropped
From your perspective, all four scenarios look identical: silence.
The Timeout Dilemma
You might use timeouts to detect failures, but this creates a catch-22:
- Short timeouts: Risk declaring healthy-but-slow nodes as failed, potentially cascading failures
- Long timeouts: Slow to detect real failures, poor user experience
There's no perfect answer. You're making a probabilistic guess, not a definitive determination.
Real-World Implications
You've already learned about network partitions and latency. Partial failures combine these: you must design systems that gracefully handle ambiguity, using techniques like retries, idempotency, and assuming the worst-case scenario.
Key Takeaway: In distributed systems, you cannot reliably distinguish between a failed node and a slow node—you can only make educated guesses with timeouts, accepting that you'll sometimes be wrong.