Why Distributed Systems Cannot Trust the Clock

Why time becomes dangerous across machines

A clock reading feels objective, but in a distributed system it is only one machine's measurement. Messages arrive after unpredictable delays, quartz clocks drift, and a process can stop while the rest of the system continues. Those facts turn time into a source of lost writes, stale leaders, and inconsistent snapshots.

The first distinction that clarified this for me was that clocks answer two different questions. A time-of-day clock answers, "What calendar time is it?" A monotonic clock answers, "How much time has elapsed?" Publication dates and scheduled reminders need calendar time. Request timeouts and response durations need monotonic time.

Time-of-day and monotonic clocks solve different problems

Linux exposes time-of-day through CLOCK_REALTIME, while Java provides System.currentTimeMillis(). These clocks are normally adjusted using NTP so timestamps from different machines roughly agree.

They can still jump forward or backward, ignore or smear leap seconds, and display far more digits than their real accuracy justifies. That makes them unsafe for measuring elapsed time. A timeout can become longer, shorter, or even appear negative after a clock correction.

Monotonic clocks, such as CLOCK_MONOTONIC and System.nanoTime(), only move forward. Their absolute value is meaningless, but subtracting two readings on the same machine gives a useful duration. Values from different machines cannot be compared.

Monotonic does not mean perfectly accurate. NTP can gradually adjust its rate without making it jump. The OS also compensates for differences between underlying CPU timers, so implementation guarantees still matter.

NTP improves clocks but cannot make them exact

NTP compares a machine's clock with several time servers and normally rejects obvious outliers. It can gradually change the clock's rate, known as slewing, or directly correct a large difference. It cannot remove the uncertainty created by hardware drift and network delay.

Quartz drift changes with temperature. At an assumed drift of 200 parts per million, a clock can move about 6 milliseconds during a 30-second synchronization interval or about 17 seconds in one day. A firewall can silently block NTP. A server may report the wrong time. Congestion can make the reply too old to be useful, and a large difference may cause the client to refuse synchronization.

Virtual machines add another problem. A VM may stop running while another VM gets the CPU, making time appear to jump forward. Mobile and embedded devices are less trustworthy because users may deliberately set their clocks incorrectly.

Very accurate synchronization is possible with GPS, atomic clocks, Precision Time Protocol, careful deployment, and continuous monitoring. It is an engineering investment, not an NTP default. If correctness depends on synchronized clocks, clock offset must be monitored and a badly skewed node must be removed before it silently damages data.

Last-write-wins can discard the write that happened later

The clearest failure appears when a database uses wall-clock timestamps to resolve conflicting writes. Client A writes x = 1. After that value reaches another node, client B increments it to x = 2. Causally, B's write is later. Clock skew can still give it the earlier timestamp.

sequenceDiagram
    participant A as Client A
    participant B as Client B
    participant N1 as Node 1
    participant N3 as Node 3
    participant N2 as Node 2
    A->>N1: set x = 1
    Note right of N1: timestamp 42.004
    N1->>N3: replicate x = 1
    B->>N3: increment x to 2
    Note right of N3: timestamp 42.003
    N1->>N2: x = 1 at 42.004
    N3->>N2: x = 2 at 42.003
    Note right of N2: LWW keeps x = 1

Node 2 sees 42.004 as newer and discards x = 2. The clocks differ by less than three milliseconds, yet the increment disappears. Generating timestamps on clients instead of servers does not fix the assumption.

Last-write-wins has three separate weaknesses. A slow clock can make a node's writes lose until the skew passes. A timestamp cannot distinguish a rapid causal sequence from truly concurrent writes. Two writers may also generate the same millisecond timestamp, forcing an arbitrary tiebreaker. Logical clocks and explicit causality tracking are safer when the real question is whether one event happened before another.

💡 Snowflake-style IDs are globally unique and roughly time sortable, but they do not prove causal order. A server with a slow clock can observe an event and create its response with a numerically smaller ID.

A timestamp is better understood as an interval

A machine usually reports one timestamp, but the honest answer is a range: the earliest and latest time that could be correct. For server-derived time, that uncertainty includes clock drift since the last synchronization, the server's own uncertainty, and network round-trip uncertainty.

💡 Resolution and accuracy are different. A timestamp may display nanoseconds while still being milliseconds away from real time. Resolution tells us how finely a clock reports values; accuracy tells us how close those values are to reality.

flowchart TD
    A["Compare two uncertainty intervals"] --> B{"Do they overlap?"}
    B -- "No" --> C["Definite order"]
    B -- "Yes" --> D["Order remains unknown"]

Google Spanner's TrueTime makes this uncertainty explicit by returning [earliest, latest]. If two trusted intervals do not overlap, their order is certain. If they overlap, physical time alone cannot tell which event came first.

Spanner pays for a safe global snapshot

Snapshot isolation lets a long-running backup or analytics query see one consistent database state without blocking short read-write transactions. A write with a transaction ID greater than the snapshot's ID is invisible to that snapshot.

On a single-node database, ordered IDs can come from a counter. Across partitions and datacenters, one global counter requires coordination and can become a bottleneck. The order must also preserve causality: if transaction B reads A's write, B must be ordered after A.

Suppose transaction A creates an order with ID 20. Transaction B reads that order and records its payment, but receives ID 10. A snapshot at ID 15 would include the payment while excluding the order that caused it. The snapshot is internally inconsistent even though each partition followed its local order.

Spanner uses TrueTime intervals for this purpose. Before reporting a read-write transaction as committed, it waits for the remaining uncertainty interval to pass. A transaction that can read the committed data is then safely later. Better clock synchronization narrows the interval and shortens this commit wait, but it requires specialized infrastructure and adds latency.

At the source's publication point, this remained active research and was not mainstream outside Google.

💡 Reader context: TrueTime and MVCC solve different layers of the problem. TrueTime exposes bounded, globally comparable time. MVCC stores row versions and selects the versions visible at that time.

A lease can expire while its holder is asleep

Consider a database leader holding a time-limited lease. Other nodes grant the lease, only one leader should hold it, and periodic renewal keeps that leadership alive. If renewal stops, the lease expires and another node can take over.

The expiry timestamp may have been created on another machine, so comparing it with the holder's local time-of-day clock already assumes synchronization. Even after switching to a local monotonic clock, a leader might check that its lease is valid and then pause between authorization and action.

sequenceDiagram
    participant A as Leader A
    participant C as Cluster
    participant B as Leader B
    A->>A: lease valid with 12 seconds left
    Note over A: process pauses for 15 seconds
    C->>B: lease expires and B takes over
    A->>C: resumes and performs stale write
    Note over C: two nodes acted as leader

A local monotonic clock avoids wall-clock jumps, but it cannot close this gap. The paused process receives no warning that fifteen seconds passed. It may discover the expired lease only on its next loop iteration, after doing something unsafe.

Pauses can come from stop-the-world garbage collection, VM suspension or migration, laptop sleep, OS scheduling, hypervisor steal time, synchronous disk access, network-backed storage, paging, thrashing, or an accidental SIGSTOP. I/O and GC delays can compound. A distributed node must assume it can pause at any instruction while the rest of the system keeps moving.

Hard real-time means guaranteed deadlines

Ordinary server software cannot promise that every process runs within a fixed time. Hard real-time systems are built differently: missing a deadline is a system failure. An airbag controller is the intuitive example.

This is not the loose web meaning of "real-time," and it is not the same as high performance. A hard real-time system may sacrifice throughput for predictable latency. It needs a real-time operating system, bounded scheduling, known worst-case library work, restrictions on allocation and garbage collection, and extensive testing. That whole-stack cost makes it inappropriate for most server-side data systems.

Garbage collection can be treated as a planned outage

Applications can reduce the effect of GC without providing hard real-time guarantees. A runtime may warn that memory pressure is approaching. The application stops routing new work to that node, drains active requests, lets replicas carry the traffic, runs GC, and then rejoins.

Another approach collects short-lived objects normally but restarts stateless processes one at a time before long-lived objects force a full collection. Durable state must live elsewhere, and traffic must be drained first, much like a rolling upgrade.

These techniques reduce tail latency and contain pauses. They do not eliminate pauses or make prediction exact.

What I will remember

The right clock depends on the question. Monotonic time measures local durations. Time-of-day supports calendar meaning but carries synchronization error. Logical clocks express causal order. Systems that use physical time for global ordering must expose uncertainty and pay for it.

Most importantly, checking time does not freeze the world. Between a permission check and an action, the process may pause, leadership may change, and every assumption that was true at the check can become stale.

These are my personal learning notes from Designing Data-Intensive Applications by Martin Kleppmann.