Why it matters
Pipelines are graphs, not scripts. A job that must run after three upstream tables, only when their partitions exist, and be rerunnable for any historical date, needs more than cron. Orchestration supplies dependency resolution, scheduling, retry policy, state and observability. The failure it prevents is subtle: without an explicit dependency, a job runs on stale inputs and produces a plausible wrong number.
Mental model
An orchestrator is a project planner for jobs: it knows what must finish before what starts, and it keeps a record of which tasks completed. The metaphor breaks because a planner assumes tasks are independent once scheduled, while data tasks share mutable state and a rerun can collide with a live one.
How it works
- 1Declare each task with an explicit partition, such as a date or a batch id, so every run has a bounded unit of work.
- 2Express dependencies in the DAG rather than in task code, so the scheduler can wait, retry and report accurately.
- 3Make writes idempotent by replacing a partition or merging on a key, never by appending blindly.
- 4Set retry policy per task with backoff, and separate retryable failures from data errors that will never succeed.
- 5Add freshness and completion sensors so downstream tasks wait for data, not just for a clock.
- 6Rehearse a backfill on a staging target and measure how long it takes before running one in production.
Step 1 of 6
Declare each task with an explicit partition, such as a date or a batch id, so every run has a bounded unit of work.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Declare each task with an explicit partition, such as a date or a batch id, so every run … 2. Express dependencies in the DAG rather than in task code, so the scheduler can wait, retr… 3. Make writes idempotent by replacing a partition or merging on a key, never by appending b… 4. Set retry policy per task with backoff, and separate retryable failures from data errors … 5. Add freshness and completion sensors so downstream tasks wait for data, not just for a cl… 6. Rehearse a backfill on a staging target and measure how long it takes before running one …
- Declare each task with an explicit partition, such as a date or a batch id, so every run has a bounded unit of work.
- Express dependencies in the DAG rather than in task code, so the scheduler can wait, retry and report accurately.
- Make writes idempotent by replacing a partition or merging on a key, never by appending blindly.
- Set retry policy per task with backoff, and separate retryable failures from data errors that will never succeed.
- Add freshness and completion sensors so downstream tasks wait for data, not just for a clock.
- Rehearse a backfill on a staging target and measure how long it takes before running one in production.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Time-based schedule | Simple and predictable | Runs on missing or late data | Upstream arrives unpredictably |
| Data-aware sensors | Correct ordering | Idle workers and stuck waits | A sensor has no timeout and blocks the DAG |
| Task-level retries | Recovers from transient errors | Duplicated side effects without idempotency | Tasks append instead of replace |
| Single monolithic job | Few moving parts | No partial rerun and long feedback loop | One step fails after an hour of work |
Good for
Simple and predictable
Cost
Runs on missing or late data
Fails when
Upstream arrives unpredictably
- Upstream arrives unpredictably
- A sensor has no timeout and blocks the DAG
- Tasks append instead of replace
- One step fails after an hour of work
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Time-based schedule is good for Simple and predictable, costs Runs on missing or late data, and fails when Upstream arrives unpredictably. Data-aware sensors is good for Correct ordering, costs Idle workers and stuck waits, and fails when A sensor has no timeout and blocks the DAG. Task-level retries is good for Recovers from transient errors, costs Duplicated side effects without idempotency, and fails when Tasks append instead of replace. Single monolithic job is good for Few moving parts, costs No partial rerun and long feedback loop, and fails when One step fails after an hour of work
Failure modes
- Stale input run. A task starts before its upstream partition lands and publishes yesterday's join. Debug: compare the input partition id recorded by the task with the expected run date.
- Retry duplication. A transient error causes a second append and totals double. Debug: count rows per partition and compare with the source count for the same period.
- Stuck sensor. A downstream DAG never starts because a sensor waits without a timeout. Debug: list running sensors and their wait duration, oldest first.
In production
Alert on task duration and on missing partitions, not only on failures. Cap concurrency per warehouse to protect shared compute, and keep a documented backfill procedure with a staging target.
Common misconceptions
People think: A successful run means the data is correct.
Actually: Success means the task exited zero; correctness needs tests on the output and a check that the expected partition was written.
People think: Backfills are rare emergency work.
Actually: Schema changes and bug fixes make backfills routine, so the ability to scope and rerun one belongs in the design.
Interview angle
Talk about partitions, idempotent writes and dependency types, then how you would backfill a month without breaking readers. Weak answers describe a scheduler's UI and never mention what happens when a task is retried.
Sources
- Apache Airflow documentation — DAGs, task instances, retries and backfill behaviour.
- Dagster documentation — asset-based dependencies and partitioned runs.
- dbt documentation — model selection and incremental runs inside a DAG.