What I Learned About Building Reliable Data Systems

I have been reading Designing Data-Intensive Applications, and this is my own set of takeaways, not a replacement for the book. The chapter changed how I think about system design: the goal is not to find one perfect database or architecture. It is to make deliberate trade-offs so a system keeps working, handles change, and remains understandable.

A product is usually a data system, not just an app

Most real products combine a database, cache, search index, message queue, stream processor, and batch jobs. The labels matter less than the responsibilities. Every component has a role, an interface, and trade-offs around speed, consistency, availability, and operational effort.

The application is responsible for turning those pieces into a coherent experience. That means its real job is to give users useful guarantees while hiding unnecessary internal complexity.

Reliability means expecting faults

The most useful distinction I learned is between a fault and a failure. A fault is a broken part: a disk dies, a process crashes, a deployment is bad, or a configuration is wrong. A failure is when users stop getting the service they expect.

Reliable systems do not assume faults are impossible. They limit the blast radius, detect problems early, and recover before one bad component becomes a full outage. Hardware redundancy helps, but it is not enough: replicas can share the same bad software release, network, power source, or operator mistake.

Software failures are often systematic, which makes them especially dangerous. The same bug can hit every server at once. My takeaway is to design for safe rollout and recovery: test important assumptions, monitor the service, deploy gradually, keep a rollback path, and preserve the source data needed to rebuild derived state.

People are part of the system too. Safe defaults, clear validation, good documentation, staging environments, and automation for repetitive tasks are reliability features. Automation should help operators; it should not make a rare incident impossible to control manually.

Scalability starts with the workload

“Can it scale?” is too vague. The real question is: which part of the workload is growing? Requests per second, reads versus writes, data size, active users, cache hit rate, and uneven access patterns can all matter.

The Twitter timeline example made this concrete for me. A read-time design stores a tweet once and builds a timeline when someone opens it. A write-time design pushes the tweet into every follower's cached timeline. The first makes writes cheaper; the second makes reads cheaper but becomes expensive for celebrity accounts. A hybrid design is often the practical answer.

The lesson is simple: architecture should follow measured load. Scaling up to a larger machine may be the simplest answer. Scaling out across independent machines can provide more aggregate capacity and fault tolerance, but adds coordination cost. Neither is automatically better.

Performance is a distribution, not an average

Throughput tells me how much work a system completes. Response time tells me what a user waits for. The important part is that response time is not one number. An average can look fine while a small but meaningful group of users has a terrible experience.

Percentiles make that visible. p50 represents a typical request. p95 and p99 show the slower requests that users notice and operators need to investigate. Tail latency gets worse when one request depends on several backend calls, because the slowest required dependency sets the overall experience. Queues can amplify this effect: one slow request holds resources and delays the requests behind it.

I also want to measure latency where users experience it, watch for coordinated omission in load tests, and aggregate distributions correctly instead of averaging already-calculated p99 values.

Maintainability is a design requirement

Most system cost arrives after launch: debugging, patching, upgrading, migrating data, responding to incidents, and adapting to new requirements. That is why maintainability deserves the same attention as reliability and performance.

For me, it has three parts:

• Operability: clear monitoring, documented behavior, safe automation, and meaningful human control.
• Simplicity: reduce accidental complexity and cognitive load with clear abstractions and fewer surprising interactions.
• Evolvability: keep boundaries and interfaces clear enough that future changes do not destabilize everything else.

My mental model going forward

When I evaluate a design, I now ask: what can fail, what workload will really grow, which users see the slowest requests, and how will someone safely operate and change this six months from now?

There is no universal best architecture. The best design is the simplest one that meets the required guarantees for its real workload and leaves room for the people maintaining it to learn and adapt.

Inspired by my reading of Martin Kleppmann's Designing Data-Intensive Applications. This post is my own interpretation and learning notes.