Why it matters
Ingestion arrives in bursts: a bulk upload, a connector sync, a re-index. Parsing and embedding are slower and rate-limited, so a synchronous path either times out or drops work. A queue makes the burst survivable, but it moves the failure from a visible error to an invisible backlog unless message age and retry behaviour are monitored as first-class signals. The buffer changes where the pressure shows up, not whether it exists.
Mental model
A loading dock with a holding area: trucks unload at their own pace, workers process at theirs, and the floor between them absorbs the difference. The metaphor breaks because a dock fills up visibly, while a queue accepts messages silently until the backlog is hours deep.
How it works
The queue absorbs arrival rate; backlog age, not depth, says whether a person is waiting.
Step 1 of 6
Absorb the burst
Intake validates, stores the body and enqueues a reference. A synchronous design would either time out or drop work at this point.
service
Intake
Intake API
validate + persist ref
Validates the upload, writes the body and enqueues a reference. The API does not parse or embed, so a slow downstream does not hold its request open.
3 connections
Figure. The topology guarantees that intake never waits on parsing or embedding, that the document body is durable before its reference is enqueued, and that every consumer can write idempotently by document id and content hash. It does not guarantee the backlog drains, that a dead-lettered message has an owner, or that a visibility timeout matches real processing time: those need age alerts, retry classification and a measured p99.
What to notice. The queue is a buffer, not a reliability mechanism. The red edges are the two failures a queue introduces: a visibility timeout expiring mid-job so the message is processed twice, and a duplicate write creating two vectors for one document. A dead-letter queue is drawn with the owner requirement attached.
Text version
A queue-based ingestion topology in four zones. A bulk upload reaches an intake API, which stores the document body and enqueues a reference with a content hash. Two competing consumers claim messages under a visibility timeout, fetch the body, embed in batches through a shared rate-limited provider, and upsert vectors by document id and content hash into an index. Permanent failures go to a dead-letter queue, and messages that exhaust their attempts are routed there by the queue. Red dashed failure edges show a visibility timeout expiring while a consumer is still working and a duplicate write reaching the index. Six steps cover absorbing the burst, buffering, competing consumers, embedding within the provider limit, writing idempotently, and containing poison messages and duplicates.
- Define the message as a reference to the stored object plus its content hash, not the document body, so retries stay small.
- Make the consumer idempotent by upserting on document id and content hash, since redelivery is normal.
- Set the visibility timeout above the p99 processing time, or a slow job will be processed twice concurrently.
- Separate transient failures, such as rate limits and timeouts, from permanent ones, such as unsupported formats, and route the latter to a dead-letter queue.
- Scale consumers on backlog age, and cap concurrency below the embedding provider's rate limit.
- Emit progress per document so a partial batch can resume without redoing completed work.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Single queue, single consumer pool | Simple and predictable | Head-of-line blocking on slow documents | Documents vary widely in size |
| Per-format queues | Isolation and tuned concurrency | More queues to operate | Formats are uniform |
| Batch messages | Fewer calls and better throughput | Larger blast radius on failure | One bad item should not block others |
| Aggressive retries | Fast recovery from transient errors | Amplifies load during an outage | The dependency is down |
Good for
Simple and predictable
Cost
Head-of-line blocking on slow documents
Fails when
Documents vary widely in size
- Documents vary widely in size
- Formats are uniform
- One bad item should not block others
- The dependency is down
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Single queue, single consumer pool is good for Simple and predictable, costs Head-of-line blocking on slow documents, and fails when Documents vary widely in size. Per-format queues is good for Isolation and tuned concurrency, costs More queues to operate, and fails when Formats are uniform. Batch messages is good for Fewer calls and better throughput, costs Larger blast radius on failure, and fails when One bad item should not block others. Aggressive retries is good for Fast recovery from transient errors, costs Amplifies load during an outage, and fails when The dependency is down
Failure modes
- Poison message loop. One malformed document is retried forever and consumes a consumer slot. Debug: count attempts per message id and inspect the highest-attempt payloads.
- Duplicate embeddings. A visibility timeout expires mid-job and two workers index the same document. Debug: count vectors per content hash and look for duplicates.
- Invisible backlog. The queue is deep but no alert fires because depth is normal for a burst. Debug: chart the age of the oldest message and alert on p95 age.
In production
Alert on oldest-message age, retry rate and dead-letter arrivals, with an owner for each. Rehearse a provider outage by pausing consumers and confirming the backlog drains without duplicates.
Common misconceptions
People think: A queue makes ingestion reliable.
Actually: It makes it durable; reliability comes from idempotent consumers and a bounded retry policy.
People think: More consumers always drain faster.
Actually: They drain until the downstream rate limit, then they only add retries and cost.
Interview angle
Name the arrival pattern and the downstream limit, then the queue's job between them. Strong answers cover idempotency, visibility timeout and backlog age. Weak ones add a queue and assume the hard part is done.
Sources
- Azure Architecture Center, Queue-Based Load Leveling — buffering bursts between producer and consumer.
- Azure Architecture Center, Competing Consumers — scaling consumers and handling redelivery.
- Kleppmann, Designing Data-Intensive Applications — queues, delivery semantics and idempotence.