Course contentsShow
← Computer Science
Lesson 2591 of 2,87258. Distributed Systems BasicsFree lesson

Partial Failures and Fault Detection

Learn why distinguishing between slow nodes and failed nodes is fundamentally difficult in distributed systems.

Partial Failures and Fault Detection

You'll learn why it's fundamentally impossible to reliably tell whether a remote node has crashed or is just running slowly.

The Core Problem

In a distributed system, when you send a request to another node and don't get a response, you face an ambiguous situation: Is the remote node dead, or is it just slow? Is the network between you broken, or congested? Did your message get lost, or did the reply get lost?

This is called the partial failure problem—some components fail while others continue working, and you can't easily distinguish the scenarios.

Why Detection Is So Hard

Unlike a single machine where the operating system knows definitively when a process crashes, distributed systems have no such luxury. Consider:

  • Slow node: Processing your request but taking 10 seconds instead of 100ms
  • Dead node: Crashed entirely, will never respond
  • Network partition: Node is fine, but you can't reach it
  • Lost message: Your request never arrived, or the response got dropped

From your perspective, all four scenarios look identical: silence.

The Timeout Dilemma

You might use timeouts to detect failures, but this creates a catch-22:

  • Short timeouts: Risk declaring healthy-but-slow nodes as failed, potentially cascading failures
  • Long timeouts: Slow to detect real failures, poor user experience

There's no perfect answer. You're making a probabilistic guess, not a definitive determination.

Real-World Implications

You've already learned about network partitions and latency. Partial failures combine these: you must design systems that gracefully handle ambiguity, using techniques like retries, idempotency, and assuming the worst-case scenario.

Key Takeaway: In distributed systems, you cannot reliably distinguish between a failed node and a slow node—you can only make educated guesses with timeouts, accepting that you'll sometimes be wrong.