Skip to content
AI Engineer Guide.

concept · data · advanced · concept 17 of 17

Lakehouse

0 of 17 complete

Why it matters

Data lakes were cheap and unreliable: a failed job left partial files, listing a prefix was slow, and readers disagreed about what the table contained. Warehouses were reliable and closed. The lakehouse pattern adds a metadata layer that records which files make up a table at a given version, so a commit is atomic, a reader sees a consistent snapshot and the underlying files remain readable by other engines.

Mental model

It is a library with a catalogue that records which editions are on the shelf, while the books themselves stay in cheap storage. The metaphor breaks because multiple librarians can write at once, so the catalogue needs a compare-and-swap protocol rather than a person at a desk.

How it works

Mechanism, step by stepInteractive
  1. 1Choose the table format and confirm it supports the operations you need: snapshot reads, deletes, schema evolution and partition evolution.
  2. 2Stand up a catalog that all engines share, because the current metadata pointer is the source of truth.
  3. 3Write through the format's commit protocol only; a direct file write bypasses atomicity and corrupts the table view.
  4. 4Compact small files and expire old snapshots on a schedule, since the format records metadata rather than cleaning it up for you.
  5. 5Set retention for snapshots and deleted files to match audit needs and storage budget.
  6. 6Test that two engines read the same snapshot consistently before declaring the lakehouse usable.

Step 1 of 6

Choose the table format and confirm it supports the operations you need: snapshot reads, deletes, schema evolution and partition evolution.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Choose the table format and confirm it supports the operations you need: snapshot reads, … 2. Stand up a catalog that all engines share, because the current metadata pointer is the so… 3. Write through the format's commit protocol only; a direct file write bypasses atomicity a… 4. Compact small files and expire old snapshots on a schedule, since the format records meta… 5. Set retention for snapshots and deleted files to match audit needs and storage budget. 6. Test that two engines read the same snapshot consistently before declaring the lakehouse …

  1. Choose the table format and confirm it supports the operations you need: snapshot reads, deletes, schema evolution and partition evolution.
  2. Stand up a catalog that all engines share, because the current metadata pointer is the source of truth.
  3. Write through the format's commit protocol only; a direct file write bypasses atomicity and corrupts the table view.
  4. Compact small files and expire old snapshots on a schedule, since the format records metadata rather than cleaning it up for you.
  5. Set retention for snapshots and deleted files to match audit needs and storage budget.
  6. Test that two engines read the same snapshot consistently before declaring the lakehouse usable.

Trade-offs

ChoiceGood forCostFails when
Lakehouse with open tablesOne copy for BI, ML and exportCompaction and catalog operationsWriters bypass the commit protocol
Warehouse onlyManaged performance and governanceDuplication and exit costML workloads need raw file access
Lake only, no table formatCheapest storage and full flexibilityNo atomicity or snapshot isolationJobs fail mid-write and readers see partial data
Frequent compactionFast reads and fewer filesWrite amplification and compute costData arrives continuously in small batches
Trade-off mapInteractive

Good for

One copy for BI, ML and export

Cost

Compaction and catalog operations

Fails when

Writers bypass the commit protocol

  1. Writers bypass the commit protocol
  2. ML workloads need raw file access
  3. Jobs fail mid-write and readers see partial data
  4. Data arrives continuously in small batches

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Lakehouse with open tables is good for One copy for BI, ML and export, costs Compaction and catalog operations, and fails when Writers bypass the commit protocol. Warehouse only is good for Managed performance and governance, costs Duplication and exit cost, and fails when ML workloads need raw file access. Lake only, no table format is good for Cheapest storage and full flexibility, costs No atomicity or snapshot isolation, and fails when Jobs fail mid-write and readers see partial data. Frequent compaction is good for Fast reads and fewer files, costs Write amplification and compute cost, and fails when Data arrives continuously in small batches

Failure modes

  • Partial commit. A failed writer leaves files that no snapshot references, or readers see half a batch. Debug: compare file counts on storage with the file list in the current snapshot.
  • Catalog split brain. Two engines point at different metadata locations and report different row counts. Debug: query the current snapshot id from each engine and compare.
  • Snapshot bloat. Metadata grows and planning slows because old versions are never expired. Debug: count snapshots per table and measure planning time against it.

In production

Monitor file count, snapshot count and planning latency per table. Schedule compaction and expiration as owned jobs, and alert when a table has no successful commit within its expected interval.

Common misconceptions

People think: A lakehouse is a product you buy.

Actually: It is a combination of open table format, catalog and compute; buying one component does not make the architecture.

People think: Open formats remove lock-in automatically.

Actually: Portability requires every writer to honour the commit protocol and every reader to use the catalog, which is an operational commitment.

Interview angle

Describe the metadata layer, the commit protocol and the catalog, then the maintenance jobs that keep it healthy. Name the failure that a plain lake has and the cost the lakehouse adds. Weak answers list vendors and call it a data lake with SQL.

Sources