Skip to content
AI Engineer Guide.

concept · basics · intermediate · concept 12 of 24

Queues & workers

0 of 24 complete

Why it matters

Doing work inline couples the user's request to the slowest dependency and loses the work if the process dies. A queue separates acceptance from execution, which lets you retry, buffer a burst and scale workers independently. The price is that delivery is at least once, ordering is per group at best, and a message that always fails will be retried until something removes it.

Mental model

It is a ticket dispenser at a service counter: the ticket is a promise to be served, not a record of being served. The analogy breaks because a ticket can be printed twice, so the counter has to recognise a duplicate request.

How it works

Mechanism, step by stepInteractive
  1. 1Publish a message that carries everything the worker needs, including a job id and the schema version.
  2. 2Acknowledge only after the effect is durable, and make the effect idempotent on the job id.
  3. 3Set the visibility timeout above the expected run time, and heartbeat long jobs so they are not redelivered.
  4. 4Route repeated failures to a dead-letter queue with the failure reason and the original payload.
  5. 5Bound the queue and the worker concurrency, and reject or delay producers when the bound is reached.
  6. 6Monitor age of the oldest message, not just depth, because depth without arrival rate says little.

Step 1 of 6

Publish a message that carries everything the worker needs, including a job id and the schema version.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Publish a message that carries everything the worker needs, including a job id and the sc… 2. Acknowledge only after the effect is durable, and make the effect idempotent on the job i… 3. Set the visibility timeout above the expected run time, and heartbeat long jobs so they a… 4. Route repeated failures to a dead-letter queue with the failure reason and the original p… 5. Bound the queue and the worker concurrency, and reject or delay producers when the bound … 6. Monitor age of the oldest message, not just depth, because depth without arrival rate say…

  1. Publish a message that carries everything the worker needs, including a job id and the schema version.
  2. Acknowledge only after the effect is durable, and make the effect idempotent on the job id.
  3. Set the visibility timeout above the expected run time, and heartbeat long jobs so they are not redelivered.
  4. Route repeated failures to a dead-letter queue with the failure reason and the original payload.
  5. Bound the queue and the worker concurrency, and reject or delay producers when the bound is reached.
  6. Monitor age of the oldest message, not just depth, because depth without arrival rate says little.

Trade-offs

ChoiceGood forCostFails when
At-least-once deliveryNo lost workDuplicates that consumers must tolerateThe effect is not idempotent
Exactly-once semanticsSimpler consumer logic in theoryLimited to a single system boundaryThe effect leaves the broker's control
Single queueSimple operationsHead-of-line blocking across job typesOne job type is much slower
Priority queuesLatency for important workStarvation of low priorityEverything is marked urgent
Trade-off mapInteractive

Good for

No lost work

Cost

Duplicates that consumers must tolerate

Fails when

The effect is not idempotent

  1. The effect is not idempotent
  2. The effect leaves the broker's control
  3. One job type is much slower
  4. Everything is marked urgent

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: At-least-once delivery is good for No lost work, costs Duplicates that consumers must tolerate, and fails when The effect is not idempotent. Exactly-once semantics is good for Simpler consumer logic in theory, costs Limited to a single system boundary, and fails when The effect leaves the broker's control. Single queue is good for Simple operations, costs Head-of-line blocking across job types, and fails when One job type is much slower. Priority queues is good for Latency for important work, costs Starvation of low priority, and fails when Everything is marked urgent

Failure modes

  • Visibility timeout too short. A slow job is redelivered while the first attempt is still running and the effect happens twice. Debug: compare job duration percentiles with the configured timeout and count duplicate acknowledgements.
  • Poison message loop. One malformed message is retried forever and consumes a worker slot. Debug: count attempts per message id and alert when any exceeds the retry limit without reaching the dead-letter queue.
  • Unbounded backlog. Producers outpace workers and the oldest message age grows without limit. Debug: chart arrival rate against completion rate and the age of the oldest message.

In production

Alert on oldest-message age, dead-letter growth and redelivery rate. Cap retries and queue size, and rehearse draining a backlog, since recovery time depends on worker throughput rather than on the queue.

Common misconceptions

People think: A queue guarantees the work happens once.

Actually: It usually guarantees at-least-once delivery, so the consumer must make duplicates harmless.

People think: Queue depth is the metric that matters.

Actually: Age of the oldest message reflects user impact; a deep queue draining quickly is healthier than a shallow one that is stuck.

Interview angle

Cover the delivery guarantee, the idempotent effect, the visibility timeout and the dead-letter path. Name oldest-message age as the metric. Treating the queue as exactly-once, and ignoring the worker crash case, is the gap.

Sources