Why it matters
Doing work inline couples the user's request to the slowest dependency and loses the work if the process dies. A queue separates acceptance from execution, which lets you retry, buffer a burst and scale workers independently. The price is that delivery is at least once, ordering is per group at best, and a message that always fails will be retried until something removes it.
Mental model
It is a ticket dispenser at a service counter: the ticket is a promise to be served, not a record of being served. The analogy breaks because a ticket can be printed twice, so the counter has to recognise a duplicate request.
How it works
- 1Publish a message that carries everything the worker needs, including a job id and the schema version.
- 2Acknowledge only after the effect is durable, and make the effect idempotent on the job id.
- 3Set the visibility timeout above the expected run time, and heartbeat long jobs so they are not redelivered.
- 4Route repeated failures to a dead-letter queue with the failure reason and the original payload.
- 5Bound the queue and the worker concurrency, and reject or delay producers when the bound is reached.
- 6Monitor age of the oldest message, not just depth, because depth without arrival rate says little.
Step 1 of 6
Publish a message that carries everything the worker needs, including a job id and the schema version.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Publish a message that carries everything the worker needs, including a job id and the sc… 2. Acknowledge only after the effect is durable, and make the effect idempotent on the job i… 3. Set the visibility timeout above the expected run time, and heartbeat long jobs so they a… 4. Route repeated failures to a dead-letter queue with the failure reason and the original p… 5. Bound the queue and the worker concurrency, and reject or delay producers when the bound … 6. Monitor age of the oldest message, not just depth, because depth without arrival rate say…
- Publish a message that carries everything the worker needs, including a job id and the schema version.
- Acknowledge only after the effect is durable, and make the effect idempotent on the job id.
- Set the visibility timeout above the expected run time, and heartbeat long jobs so they are not redelivered.
- Route repeated failures to a dead-letter queue with the failure reason and the original payload.
- Bound the queue and the worker concurrency, and reject or delay producers when the bound is reached.
- Monitor age of the oldest message, not just depth, because depth without arrival rate says little.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| At-least-once delivery | No lost work | Duplicates that consumers must tolerate | The effect is not idempotent |
| Exactly-once semantics | Simpler consumer logic in theory | Limited to a single system boundary | The effect leaves the broker's control |
| Single queue | Simple operations | Head-of-line blocking across job types | One job type is much slower |
| Priority queues | Latency for important work | Starvation of low priority | Everything is marked urgent |
Good for
No lost work
Cost
Duplicates that consumers must tolerate
Fails when
The effect is not idempotent
- The effect is not idempotent
- The effect leaves the broker's control
- One job type is much slower
- Everything is marked urgent
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: At-least-once delivery is good for No lost work, costs Duplicates that consumers must tolerate, and fails when The effect is not idempotent. Exactly-once semantics is good for Simpler consumer logic in theory, costs Limited to a single system boundary, and fails when The effect leaves the broker's control. Single queue is good for Simple operations, costs Head-of-line blocking across job types, and fails when One job type is much slower. Priority queues is good for Latency for important work, costs Starvation of low priority, and fails when Everything is marked urgent
Failure modes
- Visibility timeout too short. A slow job is redelivered while the first attempt is still running and the effect happens twice. Debug: compare job duration percentiles with the configured timeout and count duplicate acknowledgements.
- Poison message loop. One malformed message is retried forever and consumes a worker slot. Debug: count attempts per message id and alert when any exceeds the retry limit without reaching the dead-letter queue.
- Unbounded backlog. Producers outpace workers and the oldest message age grows without limit. Debug: chart arrival rate against completion rate and the age of the oldest message.
In production
Alert on oldest-message age, dead-letter growth and redelivery rate. Cap retries and queue size, and rehearse draining a backlog, since recovery time depends on worker throughput rather than on the queue.
Common misconceptions
People think: A queue guarantees the work happens once.
Actually: It usually guarantees at-least-once delivery, so the consumer must make duplicates harmless.
People think: Queue depth is the metric that matters.
Actually: Age of the oldest message reflects user impact; a deep queue draining quickly is healthier than a shallow one that is stuck.
Interview angle
Cover the delivery guarantee, the idempotent effect, the visibility timeout and the dead-letter path. Name oldest-message age as the metric. Treating the queue as exactly-once, and ignoring the worker crash case, is the gap.
Sources
- Amazon SQS, Developer Guide — visibility timeouts, dead-letter queues and delivery semantics.
- Microsoft, Competing Consumers pattern — scaling workers against a shared queue.