Skip to content
AI Engineer Guide.

pattern · architecture · intermediate · concept 1 of 7

Queue-based ingestion

0 of 7 complete

Why it matters

Ingestion arrives in bursts: a bulk upload, a connector sync, a re-index. Parsing and embedding are slower and rate-limited, so a synchronous path either times out or drops work. A queue makes the burst survivable, but it moves the failure from a visible error to an invisible backlog unless message age and retry behaviour are monitored as first-class signals. The buffer changes where the pressure shows up, not whether it exists.

Mental model

A loading dock with a holding area: trucks unload at their own pace, workers process at theirs, and the floor between them absorbs the difference. The metaphor breaks because a dock fills up visibly, while a queue accepts messages silently until the backlog is hours deep.

How it works

Queue ingestion topologyInteractive

The queue absorbs arrival rate; backlog age, not depth, says whether a person is waiting.

Step 1/6 · Absorb the burst
IntakeDurable bufferConsumersDownstream

Step 1 of 6

Absorb the burst

Intake validates, stores the body and enqueues a reference. A synchronous design would either time out or drop work at this point.

service

Intake

Intake API

validate + persist ref

Validates the upload, writes the body and enqueues a reference. The API does not parse or embed, so a slow downstream does not hold its request open.

3 connections

syncasyncreturnfailurehighlighted path = current step4 zones · 9 nodes · 17 edges · illustrative geometry

Figure. The topology guarantees that intake never waits on parsing or embedding, that the document body is durable before its reference is enqueued, and that every consumer can write idempotently by document id and content hash. It does not guarantee the backlog drains, that a dead-lettered message has an owner, or that a visibility timeout matches real processing time: those need age alerts, retry classification and a measured p99.

What to notice. The queue is a buffer, not a reliability mechanism. The red edges are the two failures a queue introduces: a visibility timeout expiring mid-job so the message is processed twice, and a duplicate write creating two vectors for one document. A dead-letter queue is drawn with the owner requirement attached.

Text version

A queue-based ingestion topology in four zones. A bulk upload reaches an intake API, which stores the document body and enqueues a reference with a content hash. Two competing consumers claim messages under a visibility timeout, fetch the body, embed in batches through a shared rate-limited provider, and upsert vectors by document id and content hash into an index. Permanent failures go to a dead-letter queue, and messages that exhaust their attempts are routed there by the queue. Red dashed failure edges show a visibility timeout expiring while a consumer is still working and a duplicate write reaching the index. Six steps cover absorbing the burst, buffering, competing consumers, embedding within the provider limit, writing idempotently, and containing poison messages and duplicates.

  1. Define the message as a reference to the stored object plus its content hash, not the document body, so retries stay small.
  2. Make the consumer idempotent by upserting on document id and content hash, since redelivery is normal.
  3. Set the visibility timeout above the p99 processing time, or a slow job will be processed twice concurrently.
  4. Separate transient failures, such as rate limits and timeouts, from permanent ones, such as unsupported formats, and route the latter to a dead-letter queue.
  5. Scale consumers on backlog age, and cap concurrency below the embedding provider's rate limit.
  6. Emit progress per document so a partial batch can resume without redoing completed work.

Trade-offs

ChoiceGood forCostFails when
Single queue, single consumer poolSimple and predictableHead-of-line blocking on slow documentsDocuments vary widely in size
Per-format queuesIsolation and tuned concurrencyMore queues to operateFormats are uniform
Batch messagesFewer calls and better throughputLarger blast radius on failureOne bad item should not block others
Aggressive retriesFast recovery from transient errorsAmplifies load during an outageThe dependency is down
Trade-off mapInteractive

Good for

Simple and predictable

Cost

Head-of-line blocking on slow documents

Fails when

Documents vary widely in size

  1. Documents vary widely in size
  2. Formats are uniform
  3. One bad item should not block others
  4. The dependency is down

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Single queue, single consumer pool is good for Simple and predictable, costs Head-of-line blocking on slow documents, and fails when Documents vary widely in size. Per-format queues is good for Isolation and tuned concurrency, costs More queues to operate, and fails when Formats are uniform. Batch messages is good for Fewer calls and better throughput, costs Larger blast radius on failure, and fails when One bad item should not block others. Aggressive retries is good for Fast recovery from transient errors, costs Amplifies load during an outage, and fails when The dependency is down

Failure modes

  • Poison message loop. One malformed document is retried forever and consumes a consumer slot. Debug: count attempts per message id and inspect the highest-attempt payloads.
  • Duplicate embeddings. A visibility timeout expires mid-job and two workers index the same document. Debug: count vectors per content hash and look for duplicates.
  • Invisible backlog. The queue is deep but no alert fires because depth is normal for a burst. Debug: chart the age of the oldest message and alert on p95 age.

In production

Alert on oldest-message age, retry rate and dead-letter arrivals, with an owner for each. Rehearse a provider outage by pausing consumers and confirming the backlog drains without duplicates.

Common misconceptions

People think: A queue makes ingestion reliable.

Actually: It makes it durable; reliability comes from idempotent consumers and a bounded retry policy.

People think: More consumers always drain faster.

Actually: They drain until the downstream rate limit, then they only add retries and cost.

Interview angle

Name the arrival pattern and the downstream limit, then the queue's job between them. Strong answers cover idempotency, visibility timeout and backlog age. Weak ones add a queue and assume the hard part is done.

Sources