Skip to content
AI Engineer Guide.

concept · data · intermediate · concept 11 of 17

Object storage patterns

0 of 17 complete

Why it matters

A data lake is mostly a convention layered over a key-value store. The store offers durability and cheap capacity but no transactions, no rename and no cheap listing at scale. Pipelines that treat it like a filesystem produce partial writes, millions of tiny objects and jobs whose runtime is dominated by listing rather than reading. Naming and layout are the only levers, so they carry the design.

Mental model

Object storage is a coat check that only accepts sealed bags: you hand over a bag under a number and get back exactly that bag. The metaphor breaks because a coat check has a person who can find a bag by description, while object storage requires the exact key and charges you for enumerating the rack.

How it works

Object count is the layout, bytes are notInteractive
15 occupied slots across 2 poolslivefree

Step · 1 of 9

Twelve small writes, twelve objects

An ingest run writes twelve 80 MB files into one date prefix: 960 MB in twelve objects. The key is the only index the store has, so a reader that needs this day must discover the files under this prefix.

Next: A reader pays per object

Data objects
12
Data bytes (GB)
0.96
Staging objects
3
Partial visible
0
Step 1/9 · Twelve small writes, twelve objects

Figure. Twelve object records under one date prefix and a four-slot staging prefix. One slot is one file and the tag is its size (an in-progress object is tagged partial), so the grid is a census of a prefix, not a quota. Compaction cuts twelve objects to two without changing 960 MB; a non-key filter cannot prune; lifecycle expiry is a prefix rule; streaming to the final key can make a partial object visible. Sizes are synthetic and request-rate hotspots are not shown.

What to notice. The first two steps separate bytes from object count: the same 960 MB costs twelve keys to list and twelve GETs to read. Compaction (step 3) and the nightly merge (step 5) do not move the byte total; they shrink the number of entries a reader must enumerate, which is the only lever object storage gives a layout. Step 6 is the limit of that lever: date prefixes prune date predicates and nothing else. Steps 8 and 9 are a write-correctness pair, not a capacity story: the temporary-key pattern is what keeps a growing body out of the final key.

Text version

Two lanes. The date prefix has 12 object slots; staging has 4. Step 1: 12 files of 80 MB, 0.96 GB, and 3 staging objects. Step 2: a query lists 12 keys and reads 12 objects; bytes unchanged. Step 3: compaction replaces the 12 files with two 480 MB objects; 2 data objects, 0.96 GB. Step 4: four new 80 MB files; 6 objects, 1.28 GB. Step 5: a merge folds the four files into one 320 MB object; 3 objects, 1.28 GB. Step 6: a filter on customer_id cannot use the date prefix and scans all 3 objects. Step 7: the staging rule expires the three temp objects; staging 0. Step 8: a direct stream to the final key shows a partial 80 MB object; 4 data objects, 1.36 GB, partial 1. Step 9: write to temp then commit replaces the partial with a complete 80 MB object and deletes the temp; 4 data objects, 1.36 GB, partial 0. Counters show data objects 12, 12, 2, 6, 3, 3, 3, 4, 4; data bytes 0.96, 0.96, 0.96, 1.28, 1.28, 1.28, 1.28, 1.36, 1.36 GB; staging objects 3, 3, 3, 3, 3, 3, 0, 1, 0; partial visible 0 until step 8 then 1 and 0.

  1. Design keys as a hierarchy of partition values, ordered from least to most granular, so consumers can prune by prefix.
  2. Write to a temporary key and copy or commit atomically, rather than streaming directly to the final path.
  3. Target a healthy object size, and compact small writes into larger objects on a schedule.
  4. Avoid high request rates against one prefix by spreading writes across partitions.
  5. Treat deletion as a lifecycle policy on prefixes, not as an application loop over keys.
  6. Record the object layout and its schema in a catalog so readers do not infer structure from names.

Trade-offs

ChoiceGood forCostFails when
Hive-style partitionsSimple pruning in every engineDeep nesting and small filesCardinality is high and partitions are tiny
Date-prefixed flat keysEven write distributionReaders must list many prefixesQueries filter on a non-date attribute
Append-only immutable objectsSafe retries and auditStorage growth and compaction workThe dataset is updated in place frequently
In-place overwriteSimple mental modelPartial writes are visible to readersA writer fails midway through a large object
Trade-off mapInteractive

Good for

Simple pruning in every engine

Cost

Deep nesting and small files

Fails when

Cardinality is high and partitions are tiny

  1. Cardinality is high and partitions are tiny
  2. Queries filter on a non-date attribute
  3. The dataset is updated in place frequently
  4. A writer fails midway through a large object

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Hive-style partitions is good for Simple pruning in every engine, costs Deep nesting and small files, and fails when Cardinality is high and partitions are tiny. Date-prefixed flat keys is good for Even write distribution, costs Readers must list many prefixes, and fails when Queries filter on a non-date attribute. Append-only immutable objects is good for Safe retries and audit, costs Storage growth and compaction work, and fails when The dataset is updated in place frequently. In-place overwrite is good for Simple mental model, costs Partial writes are visible to readers, and fails when A writer fails midway through a large object

Failure modes

  • Listing bottleneck. A job spends most of its time enumerating keys and times out. Debug: time the list calls separately from the read calls in the job profile.
  • Partial object visible. A reader consumes a half-written file because the writer streamed to the final key. Debug: check whether any consumer read an object whose write never completed.
  • Prefix hotspot. Writes to one partition throttle while other prefixes are idle. Debug: plot request rate per prefix and look for a single dominant key space.

In production

Track object count, average object size and list request rate per dataset. Set lifecycle rules for temporary and expired prefixes, and rehearse a restore from an older prefix rather than assuming versioning is configured.

Common misconceptions

People think: Object storage behaves like a filesystem.

Actually: Rename is a copy plus delete, listing is a scan and directories are a naming convention.

People think: More partitions always improve performance.

Actually: Past a point they create small files and metadata overhead, and pruning gains disappear.

Interview angle

Explain keys, immutability and listing cost, then connect the layout to the queries it must serve. Mention atomic commit and compaction. Weak answers talk about buckets and regions without saying how a reader finds the right objects cheaply.

Sources