Why Distributed Systems Cannot Trust the Network

Silence is not a failure signal

A request leaves one machine and no response returns. The remote service may be dead, temporarily paused, still working, or completely finished while its response disappeared. The caller sees the same thing in every case: silence.

That ambiguity is the heart of distributed systems. One component can fail while the rest keep working, and no node gets a complete view of what happened. Reliable software must therefore be built around partial failure rather than treating it as an exceptional surprise.

💡 This discussion initially assumes non-Byzantine faults. Components may crash, pause, or stop responding, but they are not assumed to return arbitrary false or contradictory answers.

Why distributed failure feels different

A single computer normally presents a clean model: an operation succeeds, reports an error, or the machine crashes. Distributed systems confront the physical world directly. A switch can fail, a rack can lose power, a route can break, or one direction of a link can stop working while the other remains healthy.

High-performance computing often handles a failed node by stopping the whole offline job and restarting from a checkpoint. An online service usually cannot stop every time one commodity machine fails. It must continue serving users, replace unhealthy machines, and support rolling maintenance.

Reliability comes from layers

A system can be more reliable than its individual parts. Error-correcting codes recover limited bit corruption. TCP creates an ordered byte stream over IP even though IP may lose, duplicate, delay, or reorder packets.

The higher layer cannot erase every lower-layer weakness. Error correction has a corruption limit, and TCP cannot guarantee when data will arrive or whether the remote application processed it. Reliability layers reduce the faults the next layer must handle; they do not create perfection.

One missing response, several possible realities

In a shared-nothing system, every machine owns its memory and disk. Another node can learn its state only through network messages. Most datacenter and internet networks promise neither delivery nor a maximum delivery time.

flowchart TD
  A[Client sends request] --> B{No response}
  B --> C[Request was lost]
  B --> D[Service is paused or dead]
  B --> E[Request is still queued]
  B --> F[Operation succeeded]
  F --> G[Response was lost or delayed]

A timeout proves only that no response arrived before a deadline. The original request may still be queued and execute later. A careless retry can therefore perform a payment, email, or job twice.

Network faults are operational reality

Network faults occur even in controlled datacenters. Redundant equipment cannot eliminate switch misconfiguration or other human errors. Failures can also be directional: node A reaching node B does not prove that B can reach A.

💡 A network partition, or netsplit, separates groups of nodes that can still work internally but cannot communicate across the split. It can happen inside one datacenter through failed switches, links, routing rules, firewalls, hypervisors, or SDN controllers. It is unrelated to a database partition or shard, which deliberately divides stored data.

Handling a fault does not always mean staying available. Returning a controlled error can be valid when the behavior is defined and the system recovers afterward. The dangerous choice is leaving fault behavior unspecified. Deliberately injecting network failures during testing exposes deadlocks, destructive recovery, and other surprises before production does.

Failure detectors collect evidence, not certainty

Load balancers must remove dead servers, and replicated databases may need to promote a follower after leader failure. Several signals can help:

• An operating system can reject or close a TCP connection.
• A local process supervisor can report a crash.
• A switch-management interface can report a physical link failure.
• A router can return ICMP Destination Unreachable.

None is complete proof. Switch information may be inaccessible. Routers have limited knowledge. A TCP acknowledgement proves that bytes reached a remote TCP stack, not that the application committed the operation. Knowing the operation succeeded requires a positive application-level response, which may also be lost.

FIN indicates an orderly connection close, while RST indicates rejection or an immediate reset. Neither proves whether an earlier application operation completed.

Timeouts trade slow detection for false suspicion

A long timeout delays failover and user feedback. A short timeout reacts quickly but can declare a slow, overloaded node dead.

That false declaration transfers work to other nodes. If the cluster is already overloaded, the extra work slows them too, causing more timeouts and a cascading failure.

flowchart TD
  A[System overloaded] --> B[Healthy node responds slowly]
  B --> C[Short timeout declares failure]
  C --> D[Work moves to other nodes]
  D --> E[Other nodes become overloaded]
  E --> C

A mathematically safe timeout would require a maximum one-way network delay d and maximum server processing time r. A response would then arrive within 2d + r. Real asynchronous networks and general-purpose servers provide neither bound, so there is no universally correct timeout.

Queueing creates unbounded delays

Long delays often come from functioning components under contention:

• Several senders target one output link and fill its switch queue.
• A busy destination leaves requests waiting in an operating-system queue.
• A virtual machine pauses while another VM uses the physical CPU.
• TCP delays sending through receiver flow control or network congestion control.
• TCP waits for a retransmission timeout before replacing an unacknowledged packet.

flowchart LR
  A[Input port 1] --> D[Switch fabric]
  B[Input port 2] --> D
  C[Input port 4] --> D
  D --> E[Output port 3 queue]
  E --> F[One destination link]
  E --> G[Queue fills and packets drop]

The bottleneck is one hot destination link, not necessarily the whole switch. Near maximum utilization, queues grow rapidly. Multi-tenant clouds add noisy neighbors because network links, NICs, and CPUs are shared with workloads the application cannot observe.

TCP and UDP value different outcomes

TCP retransmits missing packets and controls its sending rate. This helps when every byte matters, but recovery introduces variable delay. UDP avoids those guarantees, though it remains exposed to queueing, scheduling, and loss.

For live voice, a retransmitted audio packet may arrive after its playback moment. Silence followed by continued audio is more useful than delivering old sound late. For a file, late bytes remain valuable, so retransmission is appropriate.

Timeouts should follow measurements

Measure round-trip distributions across many machines and over long periods, including load spikes. Then choose the application-specific balance between detection speed and false suspicion.

A Phi Accrual detector continually compares a missing heartbeat with observed timing and produces a suspicion score rather than one permanent deadline. Akka and Cassandra use this approach.

💡 A low phi score suggests an ordinary delay. A high score means the silence is increasingly unusual. The application still chooses the threshold at which suspicion becomes action.

Predictable networks reserve resources

A traditional fixed-line telephone circuit reserves bandwidth across every hop. In the ISDN example, 4,000 frames per second with 16 reserved bits per frame provides a 64 kbps channel. Because capacity is reserved, unrelated traffic does not create queues and delay can be bounded.

ISDN means Integrated Services Digital Network. Its fixed capacity explains the predictable bandwidth; this is a fixed-line example, not cellular or VoIP networking.

Ethernet and IP instead use packet switching. A TCP connection consumes available bandwidth opportunistically and uses almost none while idle. This suits bursty web pages, email, and file transfers, but packets must compete for capacity.

Predictable latency and high utilization pull apart

A circuit may reserve bandwidth that remains unused. Packet networks dynamically reassign capacity and use hardware more efficiently, but queueing makes latency variable. CPUs and virtual machines make the same trade-off when many tasks share one core.

flowchart LR
  A[Static allocation] --> B[Reserved capacity]
  B --> C[Predictable delay]
  C --> D[Lower utilization and higher cost]
  E[Dynamic allocation] --> F[Shared capacity]
  F --> G[Queueing and variable delay]
  G --> H[Higher utilization and lower cost]

Quality of Service can prioritize accepted traffic, while admission control refuses or rate-limits new traffic when capacity is unavailable. These mechanisms can approximate bounded delay, but they are not generally available end to end across multi-tenant clouds or the public internet.

The takeaway

Distributed systems cannot convert silence into certainty. Failure detectors make decisions from incomplete evidence, and timeouts balance delayed recovery against false suspicion. Queueing and dynamic resource sharing make network delay unbounded in ordinary deployments.

The practical goal is not to guess perfectly whether a node is dead. It is to design retries, failover, and side effects so that either decision remains recoverable when the guess is wrong.

These are my personal learning notes from Designing Data-Intensive Applications by Martin Kleppmann.