Why it matters
Data lakes were cheap and unreliable: a failed job left partial files, listing a prefix was slow, and readers disagreed about what the table contained. Warehouses were reliable and closed. The lakehouse pattern adds a metadata layer that records which files make up a table at a given version, so a commit is atomic, a reader sees a consistent snapshot and the underlying files remain readable by other engines.
Mental model
It is a library with a catalogue that records which editions are on the shelf, while the books themselves stay in cheap storage. The metaphor breaks because multiple librarians can write at once, so the catalogue needs a compare-and-swap protocol rather than a person at a desk.
How it works
- 1Choose the table format and confirm it supports the operations you need: snapshot reads, deletes, schema evolution and partition evolution.
- 2Stand up a catalog that all engines share, because the current metadata pointer is the source of truth.
- 3Write through the format's commit protocol only; a direct file write bypasses atomicity and corrupts the table view.
- 4Compact small files and expire old snapshots on a schedule, since the format records metadata rather than cleaning it up for you.
- 5Set retention for snapshots and deleted files to match audit needs and storage budget.
- 6Test that two engines read the same snapshot consistently before declaring the lakehouse usable.
Step 1 of 6
Choose the table format and confirm it supports the operations you need: snapshot reads, deletes, schema evolution and partition evolution.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Choose the table format and confirm it supports the operations you need: snapshot reads, … 2. Stand up a catalog that all engines share, because the current metadata pointer is the so… 3. Write through the format's commit protocol only; a direct file write bypasses atomicity a… 4. Compact small files and expire old snapshots on a schedule, since the format records meta… 5. Set retention for snapshots and deleted files to match audit needs and storage budget. 6. Test that two engines read the same snapshot consistently before declaring the lakehouse …
- Choose the table format and confirm it supports the operations you need: snapshot reads, deletes, schema evolution and partition evolution.
- Stand up a catalog that all engines share, because the current metadata pointer is the source of truth.
- Write through the format's commit protocol only; a direct file write bypasses atomicity and corrupts the table view.
- Compact small files and expire old snapshots on a schedule, since the format records metadata rather than cleaning it up for you.
- Set retention for snapshots and deleted files to match audit needs and storage budget.
- Test that two engines read the same snapshot consistently before declaring the lakehouse usable.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Lakehouse with open tables | One copy for BI, ML and export | Compaction and catalog operations | Writers bypass the commit protocol |
| Warehouse only | Managed performance and governance | Duplication and exit cost | ML workloads need raw file access |
| Lake only, no table format | Cheapest storage and full flexibility | No atomicity or snapshot isolation | Jobs fail mid-write and readers see partial data |
| Frequent compaction | Fast reads and fewer files | Write amplification and compute cost | Data arrives continuously in small batches |
Good for
One copy for BI, ML and export
Cost
Compaction and catalog operations
Fails when
Writers bypass the commit protocol
- Writers bypass the commit protocol
- ML workloads need raw file access
- Jobs fail mid-write and readers see partial data
- Data arrives continuously in small batches
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Lakehouse with open tables is good for One copy for BI, ML and export, costs Compaction and catalog operations, and fails when Writers bypass the commit protocol. Warehouse only is good for Managed performance and governance, costs Duplication and exit cost, and fails when ML workloads need raw file access. Lake only, no table format is good for Cheapest storage and full flexibility, costs No atomicity or snapshot isolation, and fails when Jobs fail mid-write and readers see partial data. Frequent compaction is good for Fast reads and fewer files, costs Write amplification and compute cost, and fails when Data arrives continuously in small batches
Failure modes
- Partial commit. A failed writer leaves files that no snapshot references, or readers see half a batch. Debug: compare file counts on storage with the file list in the current snapshot.
- Catalog split brain. Two engines point at different metadata locations and report different row counts. Debug: query the current snapshot id from each engine and compare.
- Snapshot bloat. Metadata grows and planning slows because old versions are never expired. Debug: count snapshots per table and measure planning time against it.
In production
Monitor file count, snapshot count and planning latency per table. Schedule compaction and expiration as owned jobs, and alert when a table has no successful commit within its expected interval.
Common misconceptions
People think: A lakehouse is a product you buy.
Actually: It is a combination of open table format, catalog and compute; buying one component does not make the architecture.
People think: Open formats remove lock-in automatically.
Actually: Portability requires every writer to honour the commit protocol and every reader to use the catalog, which is an operational commitment.
Interview angle
Describe the metadata layer, the commit protocol and the catalog, then the maintenance jobs that keep it healthy. Name the failure that a plain lake has and the cost the lakehouse adds. Weak answers list vendors and call it a data lake with SQL.
Sources
- Armbrust et al., Lakehouse: A New Generation of Open Platforms — the case for a transactional metadata layer over object storage.
- Apache Iceberg specification — snapshots, manifests and commit semantics.
- Delta Lake documentation — a second open table format and its transaction log.