Why it matters
A data lake is mostly a convention layered over a key-value store. The store offers durability and cheap capacity but no transactions, no rename and no cheap listing at scale. Pipelines that treat it like a filesystem produce partial writes, millions of tiny objects and jobs whose runtime is dominated by listing rather than reading. Naming and layout are the only levers, so they carry the design.
Mental model
Object storage is a coat check that only accepts sealed bags: you hand over a bag under a number and get back exactly that bag. The metaphor breaks because a coat check has a person who can find a bag by description, while object storage requires the exact key and charges you for enumerating the rack.
How it works
Step · 1 of 9
Twelve small writes, twelve objects
An ingest run writes twelve 80 MB files into one date prefix: 960 MB in twelve objects. The key is the only index the store has, so a reader that needs this day must discover the files under this prefix.
Next: A reader pays per object
- Data objects
- 12
- Data bytes (GB)
- 0.96
- Staging objects
- 3
- Partial visible
- 0
Figure. Twelve object records under one date prefix and a four-slot staging prefix. One slot is one file and the tag is its size (an in-progress object is tagged partial), so the grid is a census of a prefix, not a quota. Compaction cuts twelve objects to two without changing 960 MB; a non-key filter cannot prune; lifecycle expiry is a prefix rule; streaming to the final key can make a partial object visible. Sizes are synthetic and request-rate hotspots are not shown.
What to notice. The first two steps separate bytes from object count: the same 960 MB costs twelve keys to list and twelve GETs to read. Compaction (step 3) and the nightly merge (step 5) do not move the byte total; they shrink the number of entries a reader must enumerate, which is the only lever object storage gives a layout. Step 6 is the limit of that lever: date prefixes prune date predicates and nothing else. Steps 8 and 9 are a write-correctness pair, not a capacity story: the temporary-key pattern is what keeps a growing body out of the final key.
Text version
Two lanes. The date prefix has 12 object slots; staging has 4. Step 1: 12 files of 80 MB, 0.96 GB, and 3 staging objects. Step 2: a query lists 12 keys and reads 12 objects; bytes unchanged. Step 3: compaction replaces the 12 files with two 480 MB objects; 2 data objects, 0.96 GB. Step 4: four new 80 MB files; 6 objects, 1.28 GB. Step 5: a merge folds the four files into one 320 MB object; 3 objects, 1.28 GB. Step 6: a filter on customer_id cannot use the date prefix and scans all 3 objects. Step 7: the staging rule expires the three temp objects; staging 0. Step 8: a direct stream to the final key shows a partial 80 MB object; 4 data objects, 1.36 GB, partial 1. Step 9: write to temp then commit replaces the partial with a complete 80 MB object and deletes the temp; 4 data objects, 1.36 GB, partial 0. Counters show data objects 12, 12, 2, 6, 3, 3, 3, 4, 4; data bytes 0.96, 0.96, 0.96, 1.28, 1.28, 1.28, 1.28, 1.36, 1.36 GB; staging objects 3, 3, 3, 3, 3, 3, 0, 1, 0; partial visible 0 until step 8 then 1 and 0.
- Design keys as a hierarchy of partition values, ordered from least to most granular, so consumers can prune by prefix.
- Write to a temporary key and copy or commit atomically, rather than streaming directly to the final path.
- Target a healthy object size, and compact small writes into larger objects on a schedule.
- Avoid high request rates against one prefix by spreading writes across partitions.
- Treat deletion as a lifecycle policy on prefixes, not as an application loop over keys.
- Record the object layout and its schema in a catalog so readers do not infer structure from names.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Hive-style partitions | Simple pruning in every engine | Deep nesting and small files | Cardinality is high and partitions are tiny |
| Date-prefixed flat keys | Even write distribution | Readers must list many prefixes | Queries filter on a non-date attribute |
| Append-only immutable objects | Safe retries and audit | Storage growth and compaction work | The dataset is updated in place frequently |
| In-place overwrite | Simple mental model | Partial writes are visible to readers | A writer fails midway through a large object |
Good for
Simple pruning in every engine
Cost
Deep nesting and small files
Fails when
Cardinality is high and partitions are tiny
- Cardinality is high and partitions are tiny
- Queries filter on a non-date attribute
- The dataset is updated in place frequently
- A writer fails midway through a large object
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Hive-style partitions is good for Simple pruning in every engine, costs Deep nesting and small files, and fails when Cardinality is high and partitions are tiny. Date-prefixed flat keys is good for Even write distribution, costs Readers must list many prefixes, and fails when Queries filter on a non-date attribute. Append-only immutable objects is good for Safe retries and audit, costs Storage growth and compaction work, and fails when The dataset is updated in place frequently. In-place overwrite is good for Simple mental model, costs Partial writes are visible to readers, and fails when A writer fails midway through a large object
Failure modes
- Listing bottleneck. A job spends most of its time enumerating keys and times out. Debug: time the list calls separately from the read calls in the job profile.
- Partial object visible. A reader consumes a half-written file because the writer streamed to the final key. Debug: check whether any consumer read an object whose write never completed.
- Prefix hotspot. Writes to one partition throttle while other prefixes are idle. Debug: plot request rate per prefix and look for a single dominant key space.
In production
Track object count, average object size and list request rate per dataset. Set lifecycle rules for temporary and expired prefixes, and rehearse a restore from an older prefix rather than assuming versioning is configured.
Common misconceptions
People think: Object storage behaves like a filesystem.
Actually: Rename is a copy plus delete, listing is a scan and directories are a naming convention.
People think: More partitions always improve performance.
Actually: Past a point they create small files and metadata overhead, and pruning gains disappear.
Interview angle
Explain keys, immutability and listing cost, then connect the layout to the queries it must serve. Mention atomic commit and compaction. Weak answers talk about buckets and regions without saying how a reader finds the right objects cheaply.
Sources
- Amazon S3 User Guide — object model, keys, consistency and lifecycle rules.
- Apache Iceberg specification — how a table layer commits atomically on object storage.
- Apache Parquet documentation — the file layout that makes object sizes matter.