How data moves through systems without breaking old code

Dataflow is an application design problem

I used to think data formats were mostly about choosing JSON or a smaller binary alternative. This part of Designing Data-Intensive Applications made the bigger consequence clear: every time data crosses time, a network, or a process boundary, the format becomes part of how safely the system can change.

Rolling deployments make that practical. New servers come up while old servers are still handling work. Mobile clients update on their own schedule. Stored records may be years older than the code reading them.

During a rollout

new code reads old data
old code receives new data

Both directions must work.

A database is a message to future code

The process that writes a database record encodes data. A process that reads it later decodes it. Even if one application owns the database, saving a record is a message to a future version of that application.

New code must read old records. Older code may also read records created by newer code during a deployment. The dangerous case appears when old code reads a newer record, changes one known field, and writes its old view back. Any field it did not understand can disappear.

new record
{ name, email, preferredLanguage }

old code only understands
{ name, email }

old code updates email and saves
{ name, email }

preferredLanguage can be lost

A compatible serializer can preserve unknown fields, but application mapping can still erase them. Decoding into an older model object and serializing that object again is enough to lose data unless the application keeps the unknown fields.

ℹ️ For CDC events, do not replace a stored JSON document with an incoming event unless the source guarantees full snapshots. Merge fields that are actually present, preserve missing fields, and treat a missing field differently from an explicit null. Arrays need stable IDs, not positions, and source versions or timestamps prevent an older event from overwriting a newer one.

safe CDC update
stored JSON + fields present in event = updated JSON

separate ownership
source_data        source updates this
local_enrichment   source must not overwrite this
metadata           version, event time, source

Data outlives code, so migrations are not the default

A server deployment may replace old code in minutes. Data can remain in a production database for years. Rewriting every existing row for every schema change is expensive, so databases commonly keep historical encodings and present one current logical schema when records are read.

Adding a relational column with a null default is a good example. An older row may not physically contain the column, but a reader sees it as null. Schema evolution lets old and new representations live together without pretending they were written at the same time.

old record on disk
{ id, total }

current schema
{ id, total, currency }

read result
{ id, total, currency = null }

Exports create a clean copy, not a replacement for backups

A snapshot, warehouse export, or archive reads data and writes a new immutable copy. That is a useful chance to apply the current schema consistently, even when the live database contains records from several generations.

Avro and Parquet can store the same logical data, but they favor different workloads. Avro is row-oriented and works well when consumers need complete records. Parquet is column-oriented and works well when an analytical query scans many rows but needs only a few columns.

Avro
[record 1][record 2][record 3]
complete records often

Parquet
[all ids][all dates][all totals]
few fields across many rows

ℹ️ PostgreSQL recovery backups and data-lake exports solve different problems. Use pg_dump or a base backup plus WAL to restore PostgreSQL. Use Avro events or Parquet exports for interchange and analytics. An export does not replace disaster recovery.

An unclear migration rule remains unclear in every format. The export still needs to define what missing values, nulls, and transformed fields mean.

Services put a boundary around business rules

A database can accept many different queries. A service exposes a narrower API whose inputs and outputs are chosen by the business logic. That boundary lets the service protect its internal tables, enforce rules, and change implementation details without forcing callers to change.

fragile boundary
service A -> service B database

stable boundary
service A -> service B API

Service-oriented systems and microservices only gain this benefit when boundaries are real. A booking service should own booking rules and its API. Other services should call that API instead of querying the booking tables directly.

ℹ️ Middleware is the shared network plumbing between services. It can provide routing, discovery, authentication, timeouts, retries, load balancing, queues, logging, and tracing. A message bus is one form of middleware.

REST, SOAP, and GraphQL solve different API problems

A web service is any program exposing an API over HTTP. It can serve a mobile app, a browser frontend, another internal service, or a partner system. REST is a design style that leans on standard HTTP concepts: resource URLs, methods, headers, caching, authentication, and content types.

SOAP uses structured XML messages and commonly describes its contract with WSDL. WSDL can generate typed clients, which is useful when an existing enterprise integration already uses that ecosystem. It also tends to require more tooling and is harder to inspect manually than a typical REST API.

REST
GET    /reservations/42
POST   /reservationsPATCH  /reservations/42

SOAP
WSDL -> generated client -> XML request

ℹ️ GraphQL does not need to replace REST. A GraphQL gateway can request data from existing REST services and return only the fields a screen needs. Keep business operations in the service that owns them; the GraphQL layer should adapt results, not duplicate service logic.

A remote call is not a local function call

RPC tries to make a network operation feel like a normal method call. That convenience can hide the risks that actually determine the design. A local call is fast and predictable. A remote request can time out, arrive late, be processed while its response is lost, or be repeated after a retry.

getCustomer(id)

really means
serialize request
send over network
wait for another machine
decode response

A timeout is especially awkward because it does not reveal what happened. The request may never have reached the server, the server may have failed, or the server may have completed the work and lost only the response. Retrying can therefore perform the action twice.

ℹ️ Make repeated work safe where it has side effects. Setting an order status to paid can be idempotent. Charging a credit card needs deduplication or an idempotency key. Remote-call design also needs timeouts, retry policy, partial-failure handling, and request and response schemas.

Modern RPC embraces the network instead of hiding it

RPC is still useful. gRPC commonly uses Protocol Buffers and supports ordinary request-response calls as well as streams. Thrift and Avro also support RPC. Binary protocols can be smaller and faster than JSON over REST, especially between services inside one organization.

gRPC call shapes

unary              one request -> one response
server streaming   one request -> many responses
client streaming   many requests -> one response
bidirectional      many messages both ways

Streams fit live updates, chat, telemetry, progress reporting, and long-running imports. Service discovery solves a separate problem: instances restart and receive new addresses, so clients ask discovery or DNS for a healthy current destination instead of hard-coding one.

REST remains a strong choice for public APIs because it is easy to inspect with a browser or curl and has broad support from proxies, caches, monitoring, debugging, and testing tools.

Message brokers trade immediacy for decoupling

Message passing sits between RPC and a database. It can deliver work with low latency, but a broker stores the message temporarily instead of requiring a direct connection between producer and consumer. The producer usually publishes and continues without waiting for a response.

producer
   -> broker
   -> topic or queue
   -> consumer

producer continues immediately

The broker can buffer an overloaded consumer, redeliver after a crash, remove the need for a producer to know a consumer address, and send one message to several recipients. A consumer can publish a follow-up event or use a reply queue when request-response behavior is needed.

ℹ️ With Google Cloud Pub/Sub, one published message reaches every subscription attached to its topic. That is fan-out. Multiple workers on one subscription share the messages between them. That is load balancing. Each subscription acknowledges and retries delivery independently.

Asynchronous delivery makes duplicates and evolution normal

Brokers carry bytes plus metadata and do not require a particular data model. Compatible encodings let publishers and consumers deploy in any order. If a consumer republishes a message, it must preserve unknown fields or it can erase data added by a newer producer.

Many broker setups provide at-least-once delivery. If a consumer does not acknowledge a message in time, the broker can send it again. Consumers must expect duplicates and make repeated processing safe.

safe to repeat
set order status to paid

needs deduplication
charge a credit card

Actors use the same message idea for concurrency

An actor owns private state and processes one asynchronous message at a time. This avoids shared-memory coordination between application threads, which reduces direct exposure to race conditions, locking, and deadlock.

actor on node A
  -> encoded message
  -> network
  -> decoded message
  -> actor on node B

Distributed actor frameworks use the same model across machines. That makes local and remote communication less misleading than RPC because the model already accepts asynchronous delivery and possible message loss. It does not remove schema-evolution work during rolling upgrades.

ℹ️ Akka's default Java serialization does not provide safe evolution, so a schema-aware format such as Protocol Buffers is safer for mixed versions. Orleans has historically required a new cluster for incompatible deployments unless custom serialization is used. Erlang records are fixed-position tuples, so adding fields changes their shape; named map-like data is more flexible.

Compatibility is the release strategy

For service APIs, a practical rollout often updates servers first and clients second. Old requests must still be understood by newer servers, and old clients must safely ignore newer response fields.

old client -> new server
old request remains valid

new server -> old client
new response fields are ignored

safe change
add optional fields and defaults

Protocol Buffers, Thrift, and Avro define evolution rules in their schemas. SOAP relies on XML schemas and has stricter pitfalls. REST with JSON relies more on convention: old servers should ignore unknown optional request fields, and old clients should ignore response fields they do not use.

Across organizational boundaries, a provider cannot force every client to update. Compatibility may have to last for years, and a breaking change can mean serving multiple API versions in parallel.

What I will carry forward

A stored record, an HTTP response, a broker event, and an actor message are all contracts between different versions of code. Format choice matters, but the durable habits are broader: preserve fields you do not understand, use additive changes and defaults, make duplicate work safe, separate service ownership, and treat a rollout as a period where multiple versions are normal.

data crosses a boundary
  -> define ownership
  -> encode deliberately
  -> preserve unknown fields
  -> plan for retries and duplicates
  -> test old and new versions together

These are my personal learning notes from Designing Data-Intensive Applications by Martin Kleppmann