Skip to content
AI Engineer Guide.

concept · basics · intermediate · concept 21 of 24

Testing pyramid

0 of 24 complete

Why it matters

Test suites fail in two directions: too many slow end-to-end tests make the pipeline unusable, and too many mocked unit tests give confidence about code that does not work when assembled. The pyramid is a budget for where to spend test runtime, and its shape follows the cost of a failure at each level. The work is deciding which level can express a given behaviour and keeping the slow layer small enough to run on every change.

Mental model

It is a building inspection: check each component in the workshop, then inspect the assembled structure once. The analogy breaks because software components can be tested in isolation that never resembles their real environment, so the assembled inspection catches a class of defect the workshop cannot.

How it works

Mechanism, step by stepInteractive
  1. 1List the behaviours that matter, then assign each to the cheapest level that can actually detect its failure.
  2. 2Keep unit tests pure and fast, and inject time, randomness and network access so they are deterministic.
  3. 3Use integration tests for the wiring that mocks cannot prove: serialisation, queries, schema and configuration.
  4. 4Keep a small end-to-end set for the journeys whose failure would be a headline, and run them on a schedule if they are slow.
  5. 5Replace brittle mocks with contract tests against a recorded or sandbox dependency.
  6. 6For model-backed features, assert on structure and invariants in tests, and run quality evaluations separately from the unit suite.

Step 1 of 6

List the behaviours that matter, then assign each to the cheapest level that can actually detect its failure.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. List the behaviours that matter, then assign each to the cheapest level that can actually… 2. Keep unit tests pure and fast, and inject time, randomness and network access so they are… 3. Use integration tests for the wiring that mocks cannot prove: serialisation, queries, sch… 4. Keep a small end-to-end set for the journeys whose failure would be a headline, and run t… 5. Replace brittle mocks with contract tests against a recorded or sandbox dependency. 6. For model-backed features, assert on structure and invariants in tests, and run quality e…

  1. List the behaviours that matter, then assign each to the cheapest level that can actually detect its failure.
  2. Keep unit tests pure and fast, and inject time, randomness and network access so they are deterministic.
  3. Use integration tests for the wiring that mocks cannot prove: serialisation, queries, schema and configuration.
  4. Keep a small end-to-end set for the journeys whose failure would be a headline, and run them on a schedule if they are slow.
  5. Replace brittle mocks with contract tests against a recorded or sandbox dependency.
  6. For model-backed features, assert on structure and invariants in tests, and run quality evaluations separately from the unit suite.

Trade-offs

ChoiceGood forCostFails when
Many unit testsFast feedback and precise failuresCan pass while the system is brokenThe defect is in wiring or configuration
Heavy end-to-end suiteCatches real integration failuresSlow and flakyThe pipeline must run on every commit
Mocked dependenciesDeterministic and fastAssumptions drift from realityThe provider changes its response shape
Evaluation suite for qualityMeasures what matters for model outputNon-deterministic and costlyThe feature has strict structural requirements
Trade-off mapInteractive

Good for

Fast feedback and precise failures

Cost

Can pass while the system is broken

Fails when

The defect is in wiring or configuration

  1. The defect is in wiring or configuration
  2. The pipeline must run on every commit
  3. The provider changes its response shape
  4. The feature has strict structural requirements

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Many unit tests is good for Fast feedback and precise failures, costs Can pass while the system is broken, and fails when The defect is in wiring or configuration. Heavy end-to-end suite is good for Catches real integration failures, costs Slow and flaky, and fails when The pipeline must run on every commit. Mocked dependencies is good for Deterministic and fast, costs Assumptions drift from reality, and fails when The provider changes its response shape. Evaluation suite for quality is good for Measures what matters for model output, costs Non-deterministic and costly, and fails when The feature has strict structural requirements

Failure modes

  • Flaky test tolerated. The suite fails intermittently and the team reruns until green, so real failures are ignored. Debug: track pass rate per test over a week and quarantine anything below a stable threshold.
  • Mock drift. The mock returns a shape the real dependency no longer produces. Debug: refresh fixtures from a recorded real response and diff the fields the code reads.
  • Coverage without assertions. Coverage is high because code runs, but no test asserts the behaviour. Debug: sample tests and check each one fails when the behaviour is deliberately broken.

In production

Track suite duration, flake rate and the ratio of escaped defects to test failures. Keep the slow layer small enough that nobody is tempted to skip it, and review quarantined tests weekly.

Common misconceptions

People think: High coverage means the code is well tested.

Actually: Coverage measures execution, not verification; a test without a meaningful assertion raises coverage and catches nothing.

People think: End-to-end tests are the only ones that prove the system works.

Actually: They prove one path works, slowly; most defects are cheaper to catch at a lower level.

Interview angle

Assign each behaviour to a level by detection cost and keep the slow layer small. Separate structural assertions from quality evaluation. A coverage number offered as evidence of a meaningful suite is the weak form.

Sources