Why it matters
A run may take seconds or hours, call slow external tools and retry. Executing it inside the request that created it means the client times out, a worker is pinned and a deploy kills the work. Scheduling decouples acceptance from execution: persist the request, place it on a queue with a priority and a deadline, and let workers pull at a controlled rate.
Mental model
Scheduling is a restaurant's reservation book, not the kitchen's cooking order. The metaphor breaks because reservations assume a known duration, while agent runs vary by an order of magnitude and the book must be re-planned as they overrun.
How it works
Event · step 1 of 8
Queued
The run is accepted, persisted and returned as an id. It is interactive class with a 10-minute deadline; queue depth is 7.
In state
- Queued
Next: Leased
- Queue depth
- 7
- Lease age
- —
- Attempt
- 1
Figure. A queued run is claimed under a 60-second lease, heartbeats while it works, loses the lease when heartbeats stop, and returns to the queue. A completed run has its own exit; the dashed return is a requeue, not a retry inside the same worker.
What to notice. The lease is the ownership proof. Once it expires, the old worker's writes are fenced by its stale token, so only the next attempt can commit. Heartbeat is drawn as an event between the worker and the scheduler, not as a run state. Lease and heartbeat lengths are illustrative settings.
Text version
A state machine with seven states. Queued goes to Leased on claim with a 60 second lease. Leased goes to Running on start. Running goes to a Heartbeat event on tick, and Heartbeat returns to Running on renewed. Running goes to Expired on silence, or to Completed on exit. Expired goes to Requeued with attempt 2 on fence. Requeued loops back to Queued on requeue. The trace walks Queued, Leased, Running, Heartbeat, Running, Expired, Requeued, Queued. Queue depth is 7 at admission, 6 after the claim and 7 again after the requeue; lease age reads 15, 30 and 75 seconds before expired.
- Accept the run synchronously, persist it and return an id; execute it asynchronously.
- Classify runs by latency expectation: interactive, background, batch.
- Give each class its own queue and worker pool so bulk work cannot starve interactive traffic.
- Set a deadline and a step or cost budget per run, enforced by the worker rather than the caller.
- Add per-tenant concurrency limits and a fair-share policy for shared capacity.
- Report queue wait time and run duration separately so overload is distinguishable from slow execution.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Run inside the request | Simplest, immediate feedback | Timeouts, pinned workers, work lost on deploy | Runs exceed the request timeout |
| Single shared queue | Easy to operate | Head-of-line blocking between classes | Traffic mixes bulk and interactive |
| Priority queues | Protects latency-critical work | Starvation risk for low priority | Low-priority work has a deadline too |
| Per-tenant fair share | Predictable multi-tenant behaviour | Scheduling complexity and idle capacity | One tenant is legitimately most of the load |
Good for
Simplest, immediate feedback
Cost
Timeouts, pinned workers, work lost on deploy
Fails when
Runs exceed the request timeout
- Runs exceed the request timeout
- Traffic mixes bulk and interactive
- Low-priority work has a deadline too
- One tenant is legitimately most of the load
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Run inside the request is good for Simplest, immediate feedback, costs Timeouts, pinned workers, work lost on deploy, and fails when Runs exceed the request timeout. Single shared queue is good for Easy to operate, costs Head-of-line blocking between classes, and fails when Traffic mixes bulk and interactive. Priority queues is good for Protects latency-critical work, costs Starvation risk for low priority, and fails when Low-priority work has a deadline too. Per-tenant fair share is good for Predictable multi-tenant behaviour, costs Scheduling complexity and idle capacity, and fails when One tenant is legitimately most of the load
Failure modes
- Head-of-line blocking. A batch of long runs delays every interactive request behind it. Debug: plot queue wait time by class and look for correlation with the number of long runs.
- Zombie run. A worker dies and the run stays in a running state forever. Debug: compare heartbeat timestamps against a lease timeout and requeue stale runs.
- Retry storm. Failed runs are requeued immediately and saturate the pool. Debug: measure requeue rate and add jittered backoff with a retry cap.
In production
Track queue depth, wait time and run duration per class, and alert on the age of the oldest waiting run rather than the average. Cap concurrency per tenant and per downstream dependency.
Common misconceptions
People think: More workers always increase throughput.
Actually: Throughput is usually bounded by a downstream rate limit or a shared tool; extra workers convert into retries.
People think: A cron trigger is scheduling.
Actually: A trigger starts work; scheduling also covers admission, priority, fairness and deadlines.
Interview angle
Strong answers separate admission from execution, give each class its own pool and deadline, and name the metric that reveals overload. Weak answers describe a cron job and a worker and never discuss what happens when a long run blocks a short one.
Sources
- Google, Site Reliability Engineering — handling overload, queueing and load shedding.
- AWS Architecture Blog, Exponential Backoff and Jitter — why immediate requeue amplifies contention.