Why it matters
Agent runs, batch evaluations and long generations can take longer than any proxy will hold a connection, and a dropped connection does not cancel the work. Treating them as synchronous requests produces duplicated jobs on retry, orphaned compute and clients that cannot tell whether the work finished. The async pattern replaces the open connection with a durable job record that both sides can query.
Mental model
It is a restaurant pager: you get a device, you wait, and the pager tells you when the table is ready. The analogy breaks because a pager cannot be lost silently; a webhook can fail to arrive, so the client needs polling as a backstop.
How it works
- 1Accept the request, validate it, persist a job record with a client-supplied or generated key, and return an identifier with an accepted status.
- 2Enqueue the job and let workers claim it with a visibility timeout longer than the expected run.
- 3Expose a status resource with explicit states and a terminal failure that carries a reason.
- 4Store the result with a lifetime and a size limit, and return a reference rather than an unbounded payload.
- 5Deliver completion by signed webhook with retries and exponential backoff, and keep polling available.
- 6Reconcile: a sweeper finds jobs stuck in a running state and either retries or marks them failed.
Step 1 of 6
Accept the request, validate it, persist a job record with a client-supplied or generated key, and return an identifier with an accepted status.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Accept the request, validate it, persist a job record with a client-supplied or generated… 2. Enqueue the job and let workers claim it with a visibility timeout longer than the expect… 3. Expose a status resource with explicit states and a terminal failure that carries a reaso… 4. Store the result with a lifetime and a size limit, and return a reference rather than an … 5. Deliver completion by signed webhook with retries and exponential backoff, and keep polli… 6. Reconcile: a sweeper finds jobs stuck in a running state and either retries or marks them…
- Accept the request, validate it, persist a job record with a client-supplied or generated key, and return an identifier with an accepted status.
- Enqueue the job and let workers claim it with a visibility timeout longer than the expected run.
- Expose a status resource with explicit states and a terminal failure that carries a reason.
- Store the result with a lifetime and a size limit, and return a reference rather than an unbounded payload.
- Deliver completion by signed webhook with retries and exponential backoff, and keep polling available.
- Reconcile: a sweeper finds jobs stuck in a running state and either retries or marks them failed.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Polling | Simple client, no inbound endpoint | Wasted requests and added latency | Results are needed within seconds |
| Webhook | Immediate notification | Signing, retries and endpoint availability | The client cannot expose a public endpoint |
| Long poll | Lower latency than polling | Held connections and proxy timeouts | The job routinely exceeds the timeout |
| Synchronous call | Simplest contract | Timeouts and duplicated work | Runs exceed the request budget |
Good for
Simple client, no inbound endpoint
Cost
Wasted requests and added latency
Fails when
Results are needed within seconds
- Results are needed within seconds
- The client cannot expose a public endpoint
- The job routinely exceeds the timeout
- Runs exceed the request budget
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Polling is good for Simple client, no inbound endpoint, costs Wasted requests and added latency, and fails when Results are needed within seconds. Webhook is good for Immediate notification, costs Signing, retries and endpoint availability, and fails when The client cannot expose a public endpoint. Long poll is good for Lower latency than polling, costs Held connections and proxy timeouts, and fails when The job routinely exceeds the timeout. Synchronous call is good for Simplest contract, costs Timeouts and duplicated work, and fails when Runs exceed the request budget
Failure modes
- Duplicate job submission. The client retries after a timeout and two jobs run. Debug: count jobs per idempotency key and alert when any key has more than one active job.
- Stuck running state. A worker dies after claiming and the job never completes or retries. Debug: query jobs whose heartbeat is older than the visibility timeout.
- Lost webhook. The callback fails and the client waits forever. Debug: compare delivered webhook events with completed jobs and alert on the difference.
In production
Monitor queue age, job duration percentiles, stuck jobs and webhook delivery failures. Cap result retention and payload size, and alarm when the oldest queued job exceeds its expected runtime.
Common misconceptions
People think: Async means fire and forget.
Actually: It means the work is durable and its state is queryable; forgetting the result is how jobs are lost.
People think: A webhook is enough for delivery.
Actually: Webhooks fail, so a polling or reconciliation path is required for correctness.
Interview angle
Describe the job record, the idempotency key, the status resource and the delivery fallback. Name the stuck-job sweeper. A background thread and a promise is not durable across a restart.
Sources
- Microsoft, Asynchronous Request-Reply pattern — job identifiers, status endpoints and polling.
- AWS, Amazon SQS Developer Guide — visibility timeouts, retries and dead-letter queues.