How Cassandra Makes LSM-Trees Work at Scale

LSM-trees are often introduced as the write-friendly alternative to B-trees. The more useful question is what happens when the data must survive for years, writes arrive continuously, and the system still has to serve customers quickly. Cassandra gives a practical answer: spread the data, spread the compaction work, and keep non-operational workloads away from the live database.

This is how I now think about running an LSM-based database at banking scale. LSM-trees do not make storage work disappear. They work when the architecture gives compaction, disk space, partitioning, and replication the same attention as application code.

A bank cannot keep an LSM-tree healthy by deleting history

A real transaction is not cache data. Banks need records for customer history, disputes, audit trails, fraud investigations, and legal obligations. The right approach is not to delete legitimate transactions when they become old. It is to plan storage growth and remove only data that is genuinely temporary or derived.

Long-lived financial record
  -> retain, replicate, and make recoverable

Temporary or derived state
  -> expire, recompute, or safely delete

Cassandra turns one large problem into many smaller ones

Cassandra is not one giant LSM-tree on one machine. A partition key places each row in a part of the cluster, and replicas place copies on other nodes. Each node flushes and compacts the SSTables for the data it owns. More nodes add storage capacity and spread the compaction workload.

Account A  -> node group 1
Account B  -> node group 2
Account C  -> node group 3

Each group
  -> owns a subset of partitions
  -> compacts only that subset

What Monzo's architecture shows

Monzo has publicly described Cassandra as a core store for its microservices. In one engineering write-up, it described data replicated across three nodes and quorum reads and writes. That is the durability and availability layer around the LSM storage engine: a record is not trusted merely because one machine accepted it.

Monzo has also described using an event pipeline and BigQuery for analytics. That separation matters. A dashboard, annual spending report, or internal analysis should not need to scan the live transactional database while customers are trying to pay with their cards.

Live payment
  -> write operational record to Cassandra
  -> replicate it across nodes

Same event
  -> stream to analytics storage
  -> reporting does not scan payment tables

Partitioning helps, but hot keys still hurt

Partitioning makes LSM-trees practical at huge scale, but it is not magic. A bad key can concentrate most writes on one small set of nodes. A single very busy account, merchant, or poorly distributed identifier can create a hot partition, and that partition can still run out of compaction or disk capacity.

Adding nodes helps only when data can be distributed safely. Existing replicas have to stream data to new nodes, and that movement itself consumes network and disk resources. Growth therefore needs capacity planning, not an emergency reaction after a cluster is already full.

Compaction needs an explicit budget

Each node has finite disk bandwidth. Incoming writes, memtable flushes, replication, repairs, and compaction all compete for it. If incoming data arrives faster than old SSTables can be merged, the backlog grows. Reads touch more files, free space falls, tail latency rises, and eventually writes can slow down too.

Disk I/O budget
  -> incoming writes
  -> replication and flushes
  -> background compaction

Compaction loses the race
  -> more SSTables
  -> slower reads and less free disk
  -> write pressure

My practical decision rule

An LSM-based system can be an excellent choice for sustained, partitionable write workloads, even when data must live for a long time. It needs good partition keys, enough replicas and nodes, separate analytics storage, headroom for compaction, and monitoring for hot partitions, disk usage, backlog, and p99 latency.

B-trees remain attractive when predictable latency, frequent reads, range locking, or mixed transactional workloads matter most. The choice is not LSM for small data and B-trees for large data. It is whether the complete system can safely carry the write and compaction cost of the workload it will actually have.

These are my personal learning notes from Designing Data-Intensive Applications by Martin Kleppmann.