How leader-follower replication behaves in production

Why putting data on more machines changes the problem

A single database machine eventually hits a limit: data may outgrow it, reads or writes may exceed its capacity, or users may be too far away for acceptable latency. More machines can also keep the service available when a node or an entire location fails.

Scaling up keeps one large shared-memory machine. It is operationally simple, but big machines cost more than their parts and bottlenecks keep capacity from rising in a straight line. Shared-disk systems give several machines separate CPUs and memory but a common disk array. They suit some warehouse workloads, although shared storage brings locking and contention.

Most distributed databases use shared-nothing architecture instead. Every node owns its CPU, memory, and disk, and nodes coordinate over an ordinary network. That makes geographic distribution and commodity hardware practical, but coordination becomes an application concern. A small, focused single-machine program can still beat a large cluster for the right workload.

Replication copies data. Partitioning divides it

These are different tools. Replication keeps the same data on several nodes, which helps availability, nearby reads, and read throughput. Partitioning, also called sharding, divides a dataset into subsets so different nodes carry different parts. Production systems often use both: each partition has replicas.

flowchart TB
  D[Distributed data] --> R[Replication]
  D --> P[Partitioning]
  R --> R1[Same data on several nodes]
  R1 --> R2[Redundancy and read capacity]
  P --> P1[Different subsets on different nodes]
  P1 --> P2[More total storage and throughput]

A leader gives writes one ordered path

In leader-follower replication, one replica is the leader. Clients send writes there. The leader commits each change locally, appends it to a replication log, and followers apply that log in the same order. Followers can serve reads, but client writes go only to the leader.

flowchart LR
  C[Client] -->|writes| L[(Leader)]
  C -->|reads| L
  C -->|reads| F1[(Follower 1)]
  C -->|reads| F2[(Follower 2)]
  L -->|replication log| F1
  L -->|replication log| F2

The pattern appears in relational databases such as PostgreSQL, MySQL, Oracle, and SQL Server, and also in systems such as MongoDB, Kafka, RabbitMQ, and replicated storage. The hard part is never copying a static dataset once. It is keeping copies useful while writes keep arriving.

Durability depends on whether the leader waits

With synchronous replication, the leader waits for a follower acknowledgement before confirming the write to the client. That follower has an up-to-date copy if the leader dies, but an unavailable synchronous follower can block writes.

With asynchronous replication, the leader confirms once it has committed locally and followers catch up later. The usual lag is small, but it has no hard upper bound. If an unrecoverable leader fails before a follower receives an acknowledged write, that write can disappear.

sequenceDiagram
  participant C as Client
  participant L as Leader
  participant F as Synchronous follower
  C->>L: Write
  L->>F: Replicate committed change
  F-->>L: Acknowledge
  L-->>C: Report success

Making every follower synchronous is fragile: one slow or failed node would stop the whole system. A common middle ground is semi-synchronous replication. The leader waits for one follower and keeps other followers asynchronous. If that follower fails, another follower takes its place. This keeps two current copies while avoiding dependence on every replica.

A new follower starts from a snapshot, then catches up

Copying live database files directly can produce an incoherent mix of moments. Locking the whole database would avoid that, but it defeats the point of high availability. Instead, a new follower receives a consistent snapshot tied to one exact replication-log position, then replays every later change until it catches up.

flowchart LR
  S[Consistent snapshot at LSN 500] --> F[New follower]
  L[Leader log from 501 onward] --> F
  F --> C[Caught up and streaming new changes]

💡 The snapshot is not a full history. It needs retained or archived WAL or binlog records after its log position. If those records expire before the follower gets them, it must start over with a new snapshot. PostgreSQL calls the position an LSN; MySQL uses binlog coordinates.

💡 Replica streaming copies data into another database replica. Change data capture sends changes to a different destination, such as Kafka, a warehouse, a search index, or a cache. Logical replication logs can support both, but the jobs are different.

Follower recovery is easy. Leader failover is not

A follower keeps the changes it has received on local disk. After a crash or a temporary network break, it can compare its last log position with the leader, request what it missed, and continue. This is catch-up recovery.

When the leader fails, the system must detect the failure, select the best available follower, promote it, redirect clients, reconfigure the remaining followers, and eventually return the old leader as a follower. Some systems use automatic failover. Others keep a human in the loop because a wrong automatic decision can cost data.

flowchart TD
  D[Leader misses heartbeat timeout] --> E[Choose freshest eligible follower]
  E --> P[Promote new leader]
  P --> R[Redirect writes and reconfigure followers]
  R --> O[Old leader returns as follower]

Failure detection is a tradeoff. A short timeout restores service quickly after a real crash, but it can mistake a busy server, a garbage-collection pause, or a delayed network for a failure. A long timeout reduces false failovers and makes a real outage last longer.

Promotion can also discard writes that existed only on the old leader. A stale promoted follower is especially dangerous when it reuses automatically allocated identifiers. Two nodes that both believe they are leader create split brain, where both accept writes and later reconciliation becomes much harder.

💡 A quorum is not the default answer to every important database. A primary with a synchronous standby, durable logs, backups, and controlled failover is often simpler. Use quorum or consensus when the system must keep accepting writes after a node or zone failure while safely preventing two leaders. With three voting replicas, a majority of two can commit while one is down.

The replication log format decides what can evolve

Statement-based replication sends SQL for every follower to execute. It can diverge when a statement contains NOW(), RAND(), auto-increment allocation, nondeterministic procedures, triggers, or a different transaction order.

Physical WAL replication ships low-level storage changes. It is efficient and exact, but tightly coupled to the storage engine and often prevents leader and follower from running different database versions during an upgrade.

Logical, row-based replication records committed row changes: inserted values, an identity for deletions, an identity plus changed values for updates, and a transaction commit marker. It is easier to keep compatible across versions and easier for external systems to interpret.

flowchart TB
  C[Committed change] --> S[Statement log]
  C --> W[Physical WAL]
  C --> L[Logical row log]
  C --> T[Trigger or application replication]
  S --> S1[Replay SQL: nondeterministic risk]
  W --> W1[Storage-level changes: tightly coupled versions]
  L --> L1[Committed rows: version flexibility and CDC]
  T --> T1[Custom routing: more components and failure paths]

💡 Built-in replication normally wins because the database already handles ordering, retries, and recovery. Application-level pipelines add duplicate delivery, out-of-order events, missed events, consumer lag, partial writes, replay bugs, schema changes, and idempotency mistakes. Use them when their flexibility is worth operating those risks.

Read scaling exposes replication lag to users

Adding followers lets a mostly-read workload take pressure off the leader and serve reads near users. It also makes asynchronous replication almost unavoidable. A follower may be behind for milliseconds in normal operation, or minutes or longer under load, recovery, or network trouble. That temporary mismatch is eventual consistency, and relational databases have it too.

Read-your-writes consistency means that after a user receives a successful write response, later reads by that user show their own completed write. It does not promise that every other user sees the change immediately.

sequenceDiagram
  participant U as User
  participant L as Leader
  participant F as Lagging follower
  U->>L: Create post
  L-->>U: 200 OK
  U->>F: Read posts
  Note over F: Post may not be applied yet
  F-->>U: Stale result

💡 In MySQL, a binlog position is a conservative progress point because concurrent commits may share the interval. A GTID identifies a specific committed transaction more precisely. After a write, carry its GTID with the session; a follower can wait with WAIT_FOR_EXECUTED_GTID_SET() or the request can fall back to the leader. A replication-progress token is not an application row ID.

Users should not move backward through time

Monotonic reads prevent one user from observing a newer result and then an older one on the next refresh. Sticky routing is the simple version: pick a follower from a deterministic hash of the user ID. A more flexible design tracks the latest replication position that user has observed and only uses followers at or beyond it. If none qualify, wait briefly or use the leader.

Consistent-prefix reads protect causal order. If write A caused write B, a reader may see neither, A alone, or A followed by B. They must not see B before A. Keeping related writes in one ordered partition is the easy solution. Dependencies that cross partitions need explicit causal context.

flowchart TD
  A[Read A at posts-shard:42] --> T[Client carries causal token]
  T --> B[Write B records dependency on 42]
  B --> R{Replica has applied 42?}
  R -->|No| H[Keep B hidden or route elsewhere]
  R -->|Yes| V[Expose B safely]

💡 A timestamp alone cannot guarantee causal visibility: clocks can drift, and an ordering label does not prove the replica has the earlier change. Systems use local dependency checks with a log offset, GTID set, version vector, or another causal token. Independent concurrent writes do not need one global order.

Solutions for replication lag

If replication lag can grow from seconds into minutes or hours, I should test the product behavior under that condition. If it is confusing or unsafe, the answer is a stated consistency guarantee, not an assumption that asynchronous replicas are current.

An application can provide a stronger guarantee by reading from the leader after a write or by waiting for a follower to reach a required position. Spreading those rules through application code is complex and error-prone. Transactions let the database provide stronger guarantees while the application remains simpler.

What I will check in a real system

Leader-follower replication is a good default when one ordered write path is acceptable. The design becomes honest when it states what must survive a leader loss, what users may read from followers, how a new follower catches up, and who resolves a doubtful failover.

flowchart TD
  Q{What does the workload need?}
  Q -->|One ordered writer| P[Single primary]
  Q -->|Acked writes survive leader loss| S[Synchronous standby]
  Q -->|Version flexibility or CDC| L[Logical replication]
  Q -->|Writer must see own change| R[Leader reads or progress-aware followers]
  Q -->|User must not move backward| M[Sticky or position-aware follower reads]
  Q -->|Lag can break a user flow| G[Use an explicit consistency guarantee]

These are my personal learning notes from Designing Data-Intensive Applications by Martin Kleppmann.