Skip to content
AI Engineer Guide.

concept · operations · intermediate · concept 38 of 38

Async workloads

0 of 38 complete

Why it matters

Agent runs, batch evaluations and long generations can take longer than any proxy will hold a connection, and a dropped connection does not cancel the work. Treating them as synchronous requests produces duplicated jobs on retry, orphaned compute and clients that cannot tell whether the work finished. The async pattern replaces the open connection with a durable job record that both sides can query.

Mental model

It is a restaurant pager: you get a device, you wait, and the pager tells you when the table is ready. The analogy breaks because a pager cannot be lost silently; a webhook can fail to arrive, so the client needs polling as a backstop.

How it works

Mechanism, step by stepInteractive
  1. 1Accept the request, validate it, persist a job record with a client-supplied or generated key, and return an identifier with an accepted status.
  2. 2Enqueue the job and let workers claim it with a visibility timeout longer than the expected run.
  3. 3Expose a status resource with explicit states and a terminal failure that carries a reason.
  4. 4Store the result with a lifetime and a size limit, and return a reference rather than an unbounded payload.
  5. 5Deliver completion by signed webhook with retries and exponential backoff, and keep polling available.
  6. 6Reconcile: a sweeper finds jobs stuck in a running state and either retries or marks them failed.

Step 1 of 6

Accept the request, validate it, persist a job record with a client-supplied or generated key, and return an identifier with an accepted status.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Accept the request, validate it, persist a job record with a client-supplied or generated… 2. Enqueue the job and let workers claim it with a visibility timeout longer than the expect… 3. Expose a status resource with explicit states and a terminal failure that carries a reaso… 4. Store the result with a lifetime and a size limit, and return a reference rather than an … 5. Deliver completion by signed webhook with retries and exponential backoff, and keep polli… 6. Reconcile: a sweeper finds jobs stuck in a running state and either retries or marks them…

  1. Accept the request, validate it, persist a job record with a client-supplied or generated key, and return an identifier with an accepted status.
  2. Enqueue the job and let workers claim it with a visibility timeout longer than the expected run.
  3. Expose a status resource with explicit states and a terminal failure that carries a reason.
  4. Store the result with a lifetime and a size limit, and return a reference rather than an unbounded payload.
  5. Deliver completion by signed webhook with retries and exponential backoff, and keep polling available.
  6. Reconcile: a sweeper finds jobs stuck in a running state and either retries or marks them failed.

Trade-offs

ChoiceGood forCostFails when
PollingSimple client, no inbound endpointWasted requests and added latencyResults are needed within seconds
WebhookImmediate notificationSigning, retries and endpoint availabilityThe client cannot expose a public endpoint
Long pollLower latency than pollingHeld connections and proxy timeoutsThe job routinely exceeds the timeout
Synchronous callSimplest contractTimeouts and duplicated workRuns exceed the request budget
Trade-off mapInteractive

Good for

Simple client, no inbound endpoint

Cost

Wasted requests and added latency

Fails when

Results are needed within seconds

  1. Results are needed within seconds
  2. The client cannot expose a public endpoint
  3. The job routinely exceeds the timeout
  4. Runs exceed the request budget

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Polling is good for Simple client, no inbound endpoint, costs Wasted requests and added latency, and fails when Results are needed within seconds. Webhook is good for Immediate notification, costs Signing, retries and endpoint availability, and fails when The client cannot expose a public endpoint. Long poll is good for Lower latency than polling, costs Held connections and proxy timeouts, and fails when The job routinely exceeds the timeout. Synchronous call is good for Simplest contract, costs Timeouts and duplicated work, and fails when Runs exceed the request budget

Failure modes

  • Duplicate job submission. The client retries after a timeout and two jobs run. Debug: count jobs per idempotency key and alert when any key has more than one active job.
  • Stuck running state. A worker dies after claiming and the job never completes or retries. Debug: query jobs whose heartbeat is older than the visibility timeout.
  • Lost webhook. The callback fails and the client waits forever. Debug: compare delivered webhook events with completed jobs and alert on the difference.

In production

Monitor queue age, job duration percentiles, stuck jobs and webhook delivery failures. Cap result retention and payload size, and alarm when the oldest queued job exceeds its expected runtime.

Common misconceptions

People think: Async means fire and forget.

Actually: It means the work is durable and its state is queryable; forgetting the result is how jobs are lost.

People think: A webhook is enough for delivery.

Actually: Webhooks fail, so a polling or reconciliation path is required for correctness.

Interview angle

Describe the job record, the idempotency key, the status resource and the delivery fallback. Name the stuck-job sweeper. A background thread and a promise is not durable across a restart.

Sources