How schemas let data evolve without breaking old code

The real problem is change happening at different speeds

An application changes its features and data model over time. The difficult part is that a change rarely reaches every part of the system at once. A rolling deployment deliberately runs old and new server versions together, while client applications may stay on an older version for weeks or months.

That creates four combinations of code and data that may all exist during a transition. A safe data format has to support more than the newest version talking to itself.

OLD CODE + OLD DATA
NEW CODE + OLD DATA
OLD CODE + NEW DATA
NEW CODE + NEW DATA

All four combinations can exist during a deploy.

Backward compatibility means new code can read data written by old code. Forward compatibility means old code can tolerate data written by new code. Forward compatibility is often the stricter requirement because an older reader must safely ignore information it does not know about.

Encoding is how in-memory data becomes portable bytes

Programs use objects, structs, lists, arrays, and hash tables in memory because they are convenient for the CPU and the programming language. Files, databases, and network messages need a self-contained sequence of bytes instead.

In-memory representation
objects | structs | maps | arrays
           |
           v
encoding
           |
           v
file | database record | network message
           |
           v
decoding
           |
           v
in-memory representation

Serialization and marshalling are other names for encoding. Deserialization, parsing, and unmarshalling refer to decoding. None of these terms mean encryption: an encoded message may still be completely readable to anyone who obtains it.

Native language serializers are easy to start with and hard to live with

Java serialization, Python pickle, and Ruby Marshal can turn an object into bytes with very little work. That convenience comes with long-term costs. The format usually ties data to one language, makes evolution awkward, and is often inefficient.

The security issue is more serious: decoding can sometimes instantiate arbitrary classes. If untrusted bytes reach such a decoder, the result can be remote code execution. Data formats used across trust boundaries should not give input data that kind of power.

Convenient native serializer
  good: little initial code

Long-term risks
  language lock-in
  unsafe object construction
  weak evolution story
  avoidable space and speed cost

Text formats are useful at the boundary, but their rules matter

JSON, XML, and CSV are widely understood and human-readable, which makes them useful when systems owned by different teams or organizations exchange data. Their flexibility is also why they need careful conventions.

Numbers are a common trap. XML and CSV cannot always distinguish a number from a string containing digits. JSON does not distinguish integers from floating-point values, and JavaScript numbers use IEEE 754 doubles, which cannot exactly represent every integer above 2^53. Systems exchanging large identifiers therefore often send them as strings.

Binary data is another mismatch. JSON and XML commonly encode it as Base64 text, adding roughly one third to its size. Schemas are optional and can be complicated to enforce; CSV has no standard schema at all, so types, escaping rules, and changing columns must be agreed outside the format.

JSON or XML
  readable and broadly supported
  binary becomes Base64
  numbers need explicit rules

CSV
  no built-in schema
  type, column, and escaping rules live elsewhere

A schema avoids repeating field names in every record

Binary JSON variants such as MessagePack can remove some syntactic overhead, but each record normally still carries its field names. In one example, a JSON record needed 81 bytes and MessagePack needed 66. Schema-based formats can be much smaller because the schema assigns compact field tags once, while the record contains only tags and values.

Protocol Buffers and Thrift use an explicit schema and can generate code for several languages. Their compact forms also use variable-length integer encoding, so small integer values use fewer bytes.

JSON-style record
field name + value
field name + value

Schema-based binary record
tag 1 + value
tag 2 + value

The schema explains what each tag means.

Protocol Buffers and Thrift make field tags part of the contract

In Protocol Buffers and Thrift, a field name is mainly for humans and generated code. The numeric field tag is what travels on the wire, so a name can change but its tag must never change. A reader can skip an unknown field because the encoded type or length tells it how many bytes belong to that field.

That skip behavior enables forward compatibility. New fields should be optional or have defaults, because an old reader will not supply them and a new reader may encounter old data where they are absent. Required fields are generally a bad fit for evolving systems: the requirement is checked by the reader, not carried as a special on-wire representation.

Deleting an optional field can be safe, but its tag must remain retired forever. Reusing a tag later gives old data a new meaning. Changing a 32-bit field to 64-bit can also be risky: newer code may read old values, but old code might truncate a newly written large value.

Repeated fields need format-specific care. In Protocol Buffers, multiple occurrences of one tag can represent a repeated value, and an older singular reader may keep only the last one. Thrift encodes a list differently, so an optional-to-repeated change is not equivalent there.

SAFE EVOLUTION
rename a field
add a new unused tag
make a new field optional or give it a default

UNSAFE EVOLUTION
change a field tag
reuse a retired tag
add a new required field
widen a type without checking old readers

Avro solves the same problem by comparing two schemas

Avro takes a different route. Its records do not contain field tags. Values are written in the order defined by the schema, which makes the encoding compact but means the decoder needs the writer's schema as well as the reader's schema.

When decoding, Avro resolves the writer schema against the reader schema. It matches fields by name, ignores fields that exist only in the writer schema, and fills fields that exist only in the reader schema from the reader's defaults. Field order can change without breaking compatibility because names drive this resolution.

writer schema + encoded bytes + reader schema
                  |
                  v
          schema resolution
                  |
                  v
            reader object

Defaults and nullability decide whether an Avro change is safe

Adding a field without a default breaks a new reader when it sees old records. Removing a field without a default breaks an old reader when it sees new records. If both directions matter, use defaults deliberately.

Null also has to be explicit. A field that may be absent in practice should use a union such as null or long, with a default that matches the first branch of the union. Renaming with an alias is backward compatible because a newer reader recognizes the old name, but it is not forward compatible because an older reader does not know the new name. Adding a union branch has a similar one-way compatibility concern.

union { null, long } favoriteNumber = null

New reader + old record
missing field -> reader default

Old reader + new record
writer-only field -> ignored during resolution

The writer schema must be available where decoding happens

A decoder cannot resolve Avro data without knowing how the writer encoded it. How it gets that schema depends on the dataflow.

Large file
  schema stored once in the file header

Database record
  record version -> schema registry -> writer schema

Long-lived connection
  peers negotiate schemas when the connection opens

A schema registry becomes the shared history of a format. It can assign version numbers or use schema hashes, document changes, and reject an incompatible schema before it reaches production.

Schemas are not bureaucracy: they make change visible and testable

A schema is compact documentation of what data means. It enables compatibility checks, supports static type checking when code is generated, and still works in dynamic languages where code generation is optional.

Avro is especially useful when schemas are generated from a changing database table structure, because fields are identified by names rather than permanent numeric tags. Protocol Buffers and Thrift can support the same situation, but someone has to maintain a durable mapping from every column to a tag and never reuse a retired tag.

External interchange
  use a format people and tools already understand

Long-lived stored or internal data
  use an explicit schema and compatibility rules

Every change
  test new reads old
  test old tolerates new

What I will take into system design

The format choice is not only about smaller payloads. It decides how safely a system can deploy, how long stored data remains readable, whether separate services can evolve independently, and how much trust a decoder gives incoming bytes. I would choose JSON or another familiar text format when interoperability and inspection matter most, and choose a schema-based format when data will live for a long time or cross independently deployed services.

The practical habit is to treat old data and old code as first-class users of every change. Additive fields, defaults, stable identifiers, compatibility tests, and a schema history turn evolution from a risky migration into routine work.

These are my personal learning notes from Designing Data-Intensive Applications by Martin Kleppmann.