Skip to content
AI Engineer Guide.

concept · data · intermediate · concept 9 of 17

Schema evolution

0 of 17 complete

Why it matters

A schema is a contract between producers and consumers that are deployed at different times. A field rename that looks trivial breaks a reader that has not shipped yet, and a required field breaks an old writer that does not send it. Schema evolution is the discipline of making changes that both sides can survive, and of keeping enough metadata that a reader knows which version it is looking at.

Mental model

It is renovating a shop while it stays open: you add the new counter before removing the old one, and customers use whichever is staffed. The metaphor breaks because data has no closing time, and files written last year are still customers under the old layout.

How it works

A rename split into five stepsInteractive
18 nodes · 5 branches
fieldmigration stepversionrootnote

Step 1 of 7

Classify the change

This is a rename, not an addition. A rename is two changes that look like one: a reader that asks for total_cents gets null the moment a writer stops sending it.

field · current

total_cents

The field this change renames. An amount in cents, with no currency field beside it in v1.

Under v1 · 2024-03

Step 1/7 · Classify the change

Figure. Three versions, five migration steps and one historical note. Renaming total_cents becomes an expand, a dual write, a backfill, a reader migration and a contract instead of one deploy that breaks half the consumers.

What to notice. Compatibility is directional, so the figure keeps two audiences live at once: readers still on v1 and writers already on v3. The migration branch is the only place a breaking change is allowed, and it is deliberately longer than the version column. File, partition and service counts are illustrative; the ordering — expand, dual-write, backfill, migrate readers, contract — is the pattern the page describes.

Text version

A schema history drawn as a tree. The root is the orders dataset under full compatibility, with three versions live. Version 1 from March 2024 has 1,204 files and holds id, customer_id and total_cents. Version 2 from November 2024 is additive and holds currency as a nullable field. Version 3 from June 2025 holds currency, total_amount and the deprecated total_cents. A migration branch has five steps in order: expand by adding total_amount, dual write for two weeks, backfill 380 partitions, migrate six readers, then drop total_cents in a later version. A historical note records that old files keep the writer schema they were written with, so a version 3 reader still resolves version 1 files.

  1. Classify the change: add optional field, widen a type, rename, or remove, and check each against every deployed reader and writer.
  2. Adopt a compatibility mode in a schema registry, such as backward, forward or full, and enforce it at publish time.
  3. For renames, add the new field, dual-write, backfill, migrate readers, then remove the old field in a later release.
  4. Backfill historical partitions with the new field before requiring it, so a reader never sees a null it cannot handle.
  5. Version the schema and store the version with each file or record so readers can dispatch correctly.
  6. Test an old reader against new data and a new reader against old data before rollout.

Trade-offs

ChoiceGood forCostFails when
Backward-compatible additionsOld readers keep workingSchema accumulates unused fieldsEvery consumer assumes the field is always present
Full compatibilitySafe rollouts in any orderSlow, multi-step changesThe team needs a rename finished this sprint
Versioned files, no registrySimple storageReaders must handle every version themselvesMany versions coexist and code branches multiply
Hard cutoverFast and cleanCoordinated deploy across all consumersAny consumer is outside your release process
Trade-off mapInteractive

Good for

Old readers keep working

Cost

Schema accumulates unused fields

Fails when

Every consumer assumes the field is always present

  1. Every consumer assumes the field is always present
  2. The team needs a rename finished this sprint
  3. Many versions coexist and code branches multiply
  4. Any consumer is outside your release process

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Backward-compatible additions is good for Old readers keep working, costs Schema accumulates unused fields, and fails when Every consumer assumes the field is always present. Full compatibility is good for Safe rollouts in any order, costs Slow, multi-step changes, and fails when The team needs a rename finished this sprint. Versioned files, no registry is good for Simple storage, costs Readers must handle every version themselves, and fails when Many versions coexist and code branches multiply. Hard cutover is good for Fast and clean, costs Coordinated deploy across all consumers, and fails when Any consumer is outside your release process

Failure modes

  • Rename breaks reader. A consumer reads null for a field that still exists under a new name. Debug: diff the schema version recorded per file against the fields the reader requests.
  • Required field added too early. Old writers fail validation or write records the reader rejects. Debug: list producer versions still writing and check their payload against the new contract.
  • Unbackfilled column. Historical partitions have nulls and aggregates silently exclude them. Debug: count nulls per partition for the new column before enabling a filter on it.

In production

Enforce compatibility in the registry so an unsafe publish fails in CI, not in a consumer. Track schema versions in use per dataset and alert when a version has no readers before removing it.

Common misconceptions

People think: Adding a column is always safe.

Actually: It is safe only if it is optional and every reader tolerates its absence in older files.

People think: A schema registry prevents breaking changes.

Actually: It rejects changes that violate the declared compatibility mode; choosing that mode, and coordinating consumers, is still your job.

Interview angle

Classify the change by compatibility direction, then describe expand and contract with the backfill in the middle. Mention historical files and the registry check. Weak answers say they would add a nullable column and never mention readers still on the old version.

Sources