Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 29 of 37

Scheduling

0 of 37 complete

Why it matters

A run may take seconds or hours, call slow external tools and retry. Executing it inside the request that created it means the client times out, a worker is pinned and a deploy kills the work. Scheduling decouples acceptance from execution: persist the request, place it on a queue with a priority and a deadline, and let workers pull at a controlled rate.

Mental model

Scheduling is a restaurant's reservation book, not the kitchen's cooking order. The metaphor breaks because reservations assume a known duration, while agent runs vary by an order of magnitude and the book must be re-planned as they overrun.

How it works

Lease lifecycle: claim, heartbeat, expiry, requeueInteractive
transitionfailurerepeat

Event · step 1 of 8

Queued

The run is accepted, persisted and returned as an id. It is interactive class with a 10-minute deadline; queue depth is 7.

In state

  • Queued

Next: Leased

Queue depth
7
Lease age
—
Attempt
1
Step 1/8 · Queued

Figure. A queued run is claimed under a 60-second lease, heartbeats while it works, loses the lease when heartbeats stop, and returns to the queue. A completed run has its own exit; the dashed return is a requeue, not a retry inside the same worker.

What to notice. The lease is the ownership proof. Once it expires, the old worker's writes are fenced by its stale token, so only the next attempt can commit. Heartbeat is drawn as an event between the worker and the scheduler, not as a run state. Lease and heartbeat lengths are illustrative settings.

Text version

A state machine with seven states. Queued goes to Leased on claim with a 60 second lease. Leased goes to Running on start. Running goes to a Heartbeat event on tick, and Heartbeat returns to Running on renewed. Running goes to Expired on silence, or to Completed on exit. Expired goes to Requeued with attempt 2 on fence. Requeued loops back to Queued on requeue. The trace walks Queued, Leased, Running, Heartbeat, Running, Expired, Requeued, Queued. Queue depth is 7 at admission, 6 after the claim and 7 again after the requeue; lease age reads 15, 30 and 75 seconds before expired.

  1. Accept the run synchronously, persist it and return an id; execute it asynchronously.
  2. Classify runs by latency expectation: interactive, background, batch.
  3. Give each class its own queue and worker pool so bulk work cannot starve interactive traffic.
  4. Set a deadline and a step or cost budget per run, enforced by the worker rather than the caller.
  5. Add per-tenant concurrency limits and a fair-share policy for shared capacity.
  6. Report queue wait time and run duration separately so overload is distinguishable from slow execution.

Trade-offs

ChoiceGood forCostFails when
Run inside the requestSimplest, immediate feedbackTimeouts, pinned workers, work lost on deployRuns exceed the request timeout
Single shared queueEasy to operateHead-of-line blocking between classesTraffic mixes bulk and interactive
Priority queuesProtects latency-critical workStarvation risk for low priorityLow-priority work has a deadline too
Per-tenant fair sharePredictable multi-tenant behaviourScheduling complexity and idle capacityOne tenant is legitimately most of the load
Trade-off mapInteractive

Good for

Simplest, immediate feedback

Cost

Timeouts, pinned workers, work lost on deploy

Fails when

Runs exceed the request timeout

  1. Runs exceed the request timeout
  2. Traffic mixes bulk and interactive
  3. Low-priority work has a deadline too
  4. One tenant is legitimately most of the load

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Run inside the request is good for Simplest, immediate feedback, costs Timeouts, pinned workers, work lost on deploy, and fails when Runs exceed the request timeout. Single shared queue is good for Easy to operate, costs Head-of-line blocking between classes, and fails when Traffic mixes bulk and interactive. Priority queues is good for Protects latency-critical work, costs Starvation risk for low priority, and fails when Low-priority work has a deadline too. Per-tenant fair share is good for Predictable multi-tenant behaviour, costs Scheduling complexity and idle capacity, and fails when One tenant is legitimately most of the load

Failure modes

  • Head-of-line blocking. A batch of long runs delays every interactive request behind it. Debug: plot queue wait time by class and look for correlation with the number of long runs.
  • Zombie run. A worker dies and the run stays in a running state forever. Debug: compare heartbeat timestamps against a lease timeout and requeue stale runs.
  • Retry storm. Failed runs are requeued immediately and saturate the pool. Debug: measure requeue rate and add jittered backoff with a retry cap.

In production

Track queue depth, wait time and run duration per class, and alert on the age of the oldest waiting run rather than the average. Cap concurrency per tenant and per downstream dependency.

Common misconceptions

People think: More workers always increase throughput.

Actually: Throughput is usually bounded by a downstream rate limit or a shared tool; extra workers convert into retries.

People think: A cron trigger is scheduling.

Actually: A trigger starts work; scheduling also covers admission, priority, fairness and deadlines.

Interview angle

Strong answers separate admission from execution, give each class its own pool and deadline, and name the metric that reveals overload. Weak answers describe a cron job and a worker and never discuss what happens when a long run blocks a short one.

Sources