Why it matters
Some defects only exist in the assembled product: a route that never renders, a form that loses input, an auth flow that loops. Unit tests cannot see them because they stop below the browser. An agent can now click, type and read a page, which makes those defects testable, but it inherits the browser's asynchrony and the application's hidden state, so a naive run produces failures that say more about timing than about the code.
Mental model
It is a crash-test dummy: a real collision with an expensive setup. The metaphor breaks because the dummy never gets logged out mid-test or meets a cookie banner it cannot dismiss, and those are the failures that actually stop the run.
How it works
- 1Pick the smallest user path that exercises the change and assert on visible text rather than DOM structure.
- 2Reset state per run with a fresh profile, a seeded database and a fixed clock where possible.
- 3Wait on an observable condition such as a role becoming visible, never on a fixed sleep.
- 4Capture console errors, failed network requests and a screenshot whenever an assertion fails.
- 5Run against a production build served locally so bundling differences surface in the test.
- 6Record a trace and replay it before editing the test when a failure appears.
Step 1 of 6
Pick the smallest user path that exercises the change and assert on visible text rather than DOM structure.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Pick the smallest user path that exercises the change and assert on visible text rather t… 2. Reset state per run with a fresh profile, a seeded database and a fixed clock where possi… 3. Wait on an observable condition such as a role becoming visible, never on a fixed sleep. 4. Capture console errors, failed network requests and a screenshot whenever an assertion fa… 5. Run against a production build served locally so bundling differences surface in the test. 6. Record a trace and replay it before editing the test when a failure appears.
- Pick the smallest user path that exercises the change and assert on visible text rather than DOM structure.
- Reset state per run with a fresh profile, a seeded database and a fixed clock where possible.
- Wait on an observable condition such as a role becoming visible, never on a fixed sleep.
- Capture console errors, failed network requests and a screenshot whenever an assertion fails.
- Run against a production build served locally so bundling differences surface in the test.
- Record a trace and replay it before editing the test when a failure appears.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Accessibility-role selectors | Stable across refactors | Requires accessible markup | Components are divs with click handlers |
| CSS or test-id selectors | Precise, quick to write | Break on styling and refactor | Team renames ids freely |
| Headless browser | Fast, CI friendly | Misses rendering and input quirks | Bug is visual or device-specific |
| Recorded trace replay | Excellent debugging | Storage cost, version coupling | App changes often |
Good for
Stable across refactors
Cost
Requires accessible markup
Fails when
Components are divs with click handlers
- Components are divs with click handlers
- Team renames ids freely
- Bug is visual or device-specific
- App changes often
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Accessibility-role selectors is good for Stable across refactors, costs Requires accessible markup, and fails when Components are divs with click handlers. CSS or test-id selectors is good for Precise, quick to write, costs Break on styling and refactor, and fails when Team renames ids freely. Headless browser is good for Fast, CI friendly, costs Misses rendering and input quirks, and fails when Bug is visual or device-specific. Recorded trace replay is good for Excellent debugging, costs Storage cost, version coupling, and fails when App changes often
Failure modes
- Flaky wait. The test passes locally and fails in CI because an async fetch was never awaited. Debug: replay the trace and read the failing step's network waterfall.
- Stale selector. The test clicks a hidden duplicate and asserts on the wrong screen. Debug: assert the locator resolves to exactly one element before acting.
- Auth expiry. The session times out mid-run and the failure is reported as a UI bug. Debug: check the last successful navigation against the auth cookie lifetime.
In production
Track flake rate per test and quarantine anything above a threshold. Cap total run time and retain traces for failures only, since they dominate storage.
Common misconceptions
People think: End-to-end tests are the most trustworthy.
Actually: They are the most realistic and the least stable; trust comes from a small, well-isolated set.
People think: An agent that can click can test any UI.
Actually: It still needs deterministic state and durable selectors, and autonomy does not remove flakiness.
Interview angle
Frame browser tests as an integration layer with a flake budget. A strong answer covers durable selectors, state reset, trace capture and a small critical-path set, and gives an example of a timing bug that looked like a product bug. Proposing to automate everything through the UI is the weak answer.
Sources
- Playwright documentation — locators, auto-waiting and trace viewer.
- Chrome DevTools Protocol — the browser control surface under most automation.
- Zhou et al., WebArena — realistic web tasks for agents and their evaluation.