When multiple leaders write: handling conflicts

Multiple leaders solve a specific problem

A leader-follower database has one write authority. That is usually the simple choice, but it becomes limiting when a client cannot reach the leader. Multi-leader replication lets several replicas accept writes and exchange those changes with one another. Each leader is also a follower for changes created elsewhere.

That freedom is useful only when independent locations truly need to keep writing. It also means the database can accept two incompatible changes before either side learns about the other one. Conflict handling is not a feature added later. It is part of the design from the start.

💡 Quorums are usually a leaderless-replication idea. In multi-leader replication, a local leader normally accepts a write and forwards it asynchronously. Waiting for every leader would reduce independent-write availability and still would not decide what competing writes mean.

Multi-datacenter writes can stay local

A common setup has one leader in each datacenter. Clients write to their nearby leader. Within a datacenter, ordinary leader-follower replication still copies changes to local followers. The leaders then replicate across datacenters.

flowchart LR
  C1[Clients in region 1] --> L1[(Leader in region 1)]
  L1 --> F1[(Local followers)]
  C2[Clients in region 2] --> L2[(Leader in region 2)]
  L2 --> F2[(Local followers)]
  L1 <-->|asynchronous cross-region replication| L2

This lowers perceived write latency because the client does not wait for the cross-region link. It also lets an unaffected datacenter keep accepting local writes during a datacenter outage or a temporary network split. When connectivity returns, replication catches up.

Those are availability and latency benefits, not instant agreement. Two regions can each make a valid local decision that cannot both remain valid later.

💡 Multi-leader setups can interact badly with auto-increment keys, triggers, and integrity constraints. Treat them as a specialized design, not a default way to make a database more available.

Offline clients are leaders in miniature

A phone, laptop, and server may all need to accept changes while disconnected. Each device has a local database and synchronizes when it reconnects. The lag can be hours or days. Architecturally, that is multi-leader replication with every device acting like a tiny datacenter.

flowchart LR
  P[Phone local database] <-->|intermittent sync| S[(Server)]
  L[Laptop local database] <-->|intermittent sync| S
  P -->|accepts offline writes| P
  L -->|accepts offline writes| L

Calendar, notes, task, file-sync, and version-control products can feel like multi-leader systems. They may use specialized synchronization protocols instead of a literal multi-leader database, but the user-facing problem is the same: several places accept changes and must later converge.

The point is not which local database a phone uses. The important fact is that local data can change before the network synchronizes it. CouchDB was designed for this ode of operation.

Collaborative editing makes the same trade

A shared document also has local replicas. One editor sees a keystroke immediately in their browser, then the change travels to the server and other editors.

flowchart TD
  A{Choose an editing model}
  A -->|Lock before editing| B[One editor writes at a time]
  B --> C[No concurrent edit conflict]
  A -->|Allow simultaneous edits| D[Small local changes, such as keystrokes]
  D --> E[Conflict resolution is required]

Locking a document gives behavior similar to a single leader with transactions. Avoiding locks makes collaboration feel faster, but it brings the same conflict problems as multi-leader replication. Etherpad and Google Docs use specialized algorithms for this setting.

A conflict is discovered after both sides succeed

Suppose two users edit the same wiki title. Both start with A. One local leader accepts A to B while another accepts A to C. Each user can receive success before changes cross the network. The conflict appears only when replication delivers the competing changes.

sequenceDiagram
  participant U1 as User 1
  participant L1 as Leader 1
  participant L2 as Leader 2
  participant U2 as User 2
  U1->>L1: Change title A to B
  L1-->>U1: Success
  U2->>L2: Change title A to C
  L2-->>U2: Success
  L1-->>L2: Replicate B
  L2-->>L1: Replicate C
  Note over L1,L2: Conflict must be resolved

Single-leader replication prevents this by making the second writer wait or retry. A multi-leader system could wait for every leader before confirming a write, but then it would lose the independent local-write behavior that justified multi-leader in the first place.

💡 Synchronous global conflict detection turns the design back toward single-leader behavior. It is often clearer to use a single leader when independent writes are not genuinely required.

Avoid conflicts before trying to merge them

The most practical approach is to route all writes for one record to one designated leader. A user profile can have a home datacenter, for example. Different users may use different home datacenters, while every individual profile still behaves like single-leader data.

This strategy breaks down when a datacenter fails, a user moves, or traffic must be rerouted. Those cases must still tolerate concurrent writes.

💡 Replication answers where copies exist. Write routing chooses which leader normally accepts writes for one record. Partitioning assigns subsets of the total dataset to nodes. Home-leader routing reduces conflicts, but it does not turn replicated copies into shards.

Convergence can still lose information

All replicas must eventually agree on a state. Applying writes in arrival order does not work, because one leader might see B then C while another sees C then B. Agreement needs a deterministic rule or an explicit resolution process.

flowchart TD
  Start[Concurrent writes arrive] --> Choice{How should they resolve?}
  Choice --> Winner[Pick one winner]
  Winner --> Loss[Converges, but may lose data]
  Choice --> Merge[Merge values when semantics are safe]
  Choice --> Keep[Keep all versions for later resolution]

A system can choose the highest unique write ID, use a timestamp for last-write-wins, or give one replica priority over another. Those choices converge, but they can silently discard a write. It can also merge values or preserve every conflicting version for later resolution.

💡 No universal merge rule exists. A latest notification preference may be acceptable. Label sets may safely union. A shared document may preserve versions or ask a user. Money, inventory, reservations, and other scarce resources need coordination, rejection, or compensation instead of an automatic merge.

💡 Last-write-wins is popular because it is easy to implement. It does not know which write is semantically right, and it can lose data. Convergence is not the same as correctness.

Resolution can happen on write or on read

With write-time resolution, the database, replication middleware, or a worker detects a conflict while processing replicated changes and runs a background handler. That handler must be fast, deterministic, and safe to retry. It cannot normally ask a user what they meant.

With read-time resolution, the database retains competing versions. A later read returns them to the application, which can resolve them automatically or ask the user, then write the answer back. CouchDB follows this style.

Resolution usually applies to one row or document at a time, not an entire transaction. A transaction that made several related changes can still leave the application with separate conflicts to reason about.

💡 A conflict hook may be a database trigger, stored procedure, replication callback, or external worker. Do not put slow or irreversible work, such as charging a card, in a retryable replication hook. Bucardo is an external PostgreSQL replication system that illustrates this tooling approach.

💡 Automatic merge logic needs to match the data. A shopping-cart resolver that preserved additions but forgot removal information could make a deleted item reappear. Correct handling needs operation history, a deletion marker, or a purpose-built data type with explicit removal semantics.

Some data types have better merge rules

CRDTs are data structures such as counters, sets, maps, and ordered lists that support concurrent edits with built-in merge behavior. Mergeable persistent data structures retain history and use a three-way merge, similar to Git. Operational transformation was designed for concurrent changes to ordered content, such as document characters.

These approaches reduce conflict work when their assumptions fit the data. They do not automatically solve every business constraint.

💡 A three-way merge compares a common base with two independently changed versions. If one writer changes a title and another changes a body, both changes can often survive. If both change the title differently, the conflict remains.

A conflict can be a broken business rule

Two writes to the same field are easy to spot. Other conflicts are semantic. Two leaders can each approve a booking for the same room and the same time. The writes may target different records and both local availability checks may pass, yet the combined result violates the product rule.

This is why conflict resolution needs knowledge of the application, not only a comparison of individual fields.

Topology determines how changes travel

With two leaders there is only one sensible path: each sends changes to the other. More leaders introduce topology choices.

flowchart TB
  T[Replication topology] --> C[Circular]
  T --> S[Star or tree]
  T --> A[All-to-all]
  C --> C1[Each node forwards to one neighbor]
  S --> S1[A root forwards to other nodes]
  A --> A1[Every leader sends directly to every other leader]

Circular and star topologies can force a change through several nodes. Every write carries the IDs of the nodes it has passed through, and a node ignores a change that already includes its own ID. That prevents endless forwarding loops.

Their weakness is that one failed transit node can interrupt replication between healthy nodes. All-to-all has more paths around a failed node, but it has its own ordering problem.

💡 Reconfiguration changes replication connections after a node or link fails. In a circular topology, operators or a control plane may need to create a direct path around the failed node, prevent loops and duplicate replay, and later catch the recovered node up.

All-to-all delivery can violate causal order

Network paths have different delays. An insert created on leader 1 can reach leader 2 later than an update to that row created on leader 3. Leader 2 then receives an update for data it has not seen yet.

sequenceDiagram
  participant L1 as Leader 1
  participant L2 as Leader 2
  participant L3 as Leader 3
  L1->>L3: Insert row X
  L3->>L2: Update row X
  Note over L2: Update can arrive first
  L1-->>L2: Insert row X arrives later

This is a causality problem. The update depends on the insert, so every replica must process the insert first. Timestamps cannot guarantee that order because clocks cannot be trusted to agree closely enough. The chapter introduces version vectors later as a way to track these dependencies.

Many implementations provide limited conflict detection or causal ordering. The practical lesson is to read the exact database documentation and test the behavior you depend on.

What I will take into system design

flowchart TD
  Q{Do independent locations need local writes during disconnection?}
  Q -->|No| S[Prefer single-leader replication]
  Q -->|Yes| M[Consider multi-leader replication]
  M --> C{Can each record stay on one leader?}
  C -->|Mostly| R[Use routing to avoid conflicts]
  C -->|No| H[Define merge and business-conflict rules before launch]

Multi-leader replication trades simpler write routing for availability across unreliable boundaries. I would choose it only when offline or multi-region local writes are a real requirement and the data has a clear plan for concurrent changes.

These are my personal learning notes from Designing Data-Intensive Applications by Martin Kleppmann.