Why it matters
Ordinary evaluation samples the traffic you expect, and adversarial evaluation samples the traffic someone else will send. That includes prompt injection, encoding tricks, contradictory instructions and inputs that target tool boundaries. The output is a set of findings rather than a score, and the engineering work is to make those findings reproducible enough to become regression cases. The pass rate is therefore a floor rather than a verdict, and the useful output is a set of findings with reproductions that can become regression cases.
Mental model
It is penetration testing for language. The metaphor breaks because behaviour changes with wording, so the same attack is a slightly different test each time it runs, a pass yesterday does not transfer to today, and the corpus must be refreshed.
How it works
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. Modelled attack success rate against the age of the case in the corpus, with a refresh interval you can move. Fresh attacks succeed 36% of the time; success decays with an eight-month constant toward a 4% floor, and the reported pass rate is the average over the corpus the refresh leaves behind. The corpus is illustrative; the direction of ageing, not its rate, is the claim.
What to notice. A three-month corpus reports a 69.3% pass rate because its cases are still sharp; a twelve-month corpus reports 79.4% while the system is unchanged, because most of its cases were patched long ago. The oldest case succeeds 26.0% of the time at a three-month refresh and 11.1% at twelve. A rising adversarial pass rate can mean the tests aged, not that the system hardened.
Text version
Attack success rate from 0 to 40% against attack age from 0 to 18 months. The curve starts at 36% for a fresh attack and decays toward a 4% floor, reaching 19.1% at six months and 11.1% at twelve. Three buttons refresh the corpus every 3, 6 or 12 months; a vertical line marks the refresh interval and a panel reports the modelled pass rate (69.3%, 73.5% or 79.4%), the success of the oldest case and the unchanged 36% fresh-attack rate.
- Enumerate attack classes relevant to the product: injection, exfiltration, tool misuse and encoding tricks.
- Generate variants per class using templates, model-written attacks and human red teams.
- Run each attempt with a fixed seed and store the full transcript.
- Triage findings by impact and reproducibility, and deduplicate them by root cause.
- Convert confirmed findings into regression cases with the exact input.
- Retire attacks that no longer reproduce and add new classes each cycle.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Template attacks | Cheap and broad | Shallow, easy to patch | Model is robust to known patterns |
| Model-generated attacks | Volume and variation | Repetitive, misses human creativity | You need realistic misuse |
| Human red team | Finds novel paths | Expensive, hard to repeat | You need coverage at scale |
| Continuous fuzzing | Catches regressions | Noise and triage load | Findings need prioritisation |
Good for
Cheap and broad
Cost
Shallow, easy to patch
Fails when
Model is robust to known patterns
- Model is robust to known patterns
- You need realistic misuse
- You need coverage at scale
- Findings need prioritisation
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Template attacks is good for Cheap and broad, costs Shallow, easy to patch, and fails when Model is robust to known patterns. Model-generated attacks is good for Volume and variation, costs Repetitive, misses human creativity, and fails when You need realistic misuse. Human red team is good for Finds novel paths, costs Expensive, hard to repeat, and fails when You need coverage at scale. Continuous fuzzing is good for Catches regressions, costs Noise and triage load, and fails when Findings need prioritisation
Failure modes
- Stale corpus. Every attack fails because they were patched a year ago and nothing new is tested. Debug: check the date each case was added and the last confirmed finding.
- Unreproducible finding. A screenshot shows a failure but the transcript is missing. Debug: require seed, model version and full prompt in the report.
- Severity inflation. Low-impact style breaks are mixed with data leaks. Debug: sort findings by impact class and count per class.
In production
Track new findings per cycle and time to a regression case. Keep an attack corpus with provenance and review it quarterly.
Common misconceptions
People think: Passing a red-team round means the system is secure.
Actually: It means those attacks failed, and the space of attacks is open-ended.
People think: Adversarial tests are only for safety.
Actually: Injection and tool misuse break correctness and data handling too.
Interview angle
Name attack classes, describe generation and triage, and turn findings into regression cases. A strong answer covers reproducibility and corpus ageing. Running a jailbreak list once, with no severity or provenance, is the weak version.
Sources
- Perez et al., Red Teaming Language Models with Language Models — automated generation of adversarial inputs.
- Mazeika et al., HarmBench — a standardised adversarial evaluation harness.
- Chao et al., JailbreakBench — reproducible jailbreak evaluation with artefacts.