Skip to content
AI Engineer Guide.

concept · evaluation · intermediate · concept 22 of 28

Adversarial evaluation

0 of 28 complete

Why it matters

Ordinary evaluation samples the traffic you expect, and adversarial evaluation samples the traffic someone else will send. That includes prompt injection, encoding tricks, contradictory instructions and inputs that target tool boundaries. The output is a set of findings rather than a score, and the engineering work is to make those findings reproducible enough to become regression cases. The pass rate is therefore a floor rather than a verdict, and the useful output is a set of findings with reproductions that can become regression cases.

Mental model

It is penetration testing for language. The metaphor breaks because behaviour changes with wording, so the same attack is a slightly different test each time it runs, a pass yesterday does not transfer to today, and the corpus must be refreshed.

How it works

Rising pass rate, aged testsInteractive

The interactive figure loads with JavaScript. The text version below describes the same frame.

Figure. Modelled attack success rate against the age of the case in the corpus, with a refresh interval you can move. Fresh attacks succeed 36% of the time; success decays with an eight-month constant toward a 4% floor, and the reported pass rate is the average over the corpus the refresh leaves behind. The corpus is illustrative; the direction of ageing, not its rate, is the claim.

What to notice. A three-month corpus reports a 69.3% pass rate because its cases are still sharp; a twelve-month corpus reports 79.4% while the system is unchanged, because most of its cases were patched long ago. The oldest case succeeds 26.0% of the time at a three-month refresh and 11.1% at twelve. A rising adversarial pass rate can mean the tests aged, not that the system hardened.

Text version

Attack success rate from 0 to 40% against attack age from 0 to 18 months. The curve starts at 36% for a fresh attack and decays toward a 4% floor, reaching 19.1% at six months and 11.1% at twelve. Three buttons refresh the corpus every 3, 6 or 12 months; a vertical line marks the refresh interval and a panel reports the modelled pass rate (69.3%, 73.5% or 79.4%), the success of the oldest case and the unchanged 36% fresh-attack rate.

  1. Enumerate attack classes relevant to the product: injection, exfiltration, tool misuse and encoding tricks.
  2. Generate variants per class using templates, model-written attacks and human red teams.
  3. Run each attempt with a fixed seed and store the full transcript.
  4. Triage findings by impact and reproducibility, and deduplicate them by root cause.
  5. Convert confirmed findings into regression cases with the exact input.
  6. Retire attacks that no longer reproduce and add new classes each cycle.

Trade-offs

ChoiceGood forCostFails when
Template attacksCheap and broadShallow, easy to patchModel is robust to known patterns
Model-generated attacksVolume and variationRepetitive, misses human creativityYou need realistic misuse
Human red teamFinds novel pathsExpensive, hard to repeatYou need coverage at scale
Continuous fuzzingCatches regressionsNoise and triage loadFindings need prioritisation
Trade-off mapInteractive

Good for

Cheap and broad

Cost

Shallow, easy to patch

Fails when

Model is robust to known patterns

  1. Model is robust to known patterns
  2. You need realistic misuse
  3. You need coverage at scale
  4. Findings need prioritisation

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Template attacks is good for Cheap and broad, costs Shallow, easy to patch, and fails when Model is robust to known patterns. Model-generated attacks is good for Volume and variation, costs Repetitive, misses human creativity, and fails when You need realistic misuse. Human red team is good for Finds novel paths, costs Expensive, hard to repeat, and fails when You need coverage at scale. Continuous fuzzing is good for Catches regressions, costs Noise and triage load, and fails when Findings need prioritisation

Failure modes

  • Stale corpus. Every attack fails because they were patched a year ago and nothing new is tested. Debug: check the date each case was added and the last confirmed finding.
  • Unreproducible finding. A screenshot shows a failure but the transcript is missing. Debug: require seed, model version and full prompt in the report.
  • Severity inflation. Low-impact style breaks are mixed with data leaks. Debug: sort findings by impact class and count per class.

In production

Track new findings per cycle and time to a regression case. Keep an attack corpus with provenance and review it quarterly.

Common misconceptions

People think: Passing a red-team round means the system is secure.

Actually: It means those attacks failed, and the space of attacks is open-ended.

People think: Adversarial tests are only for safety.

Actually: Injection and tool misuse break correctness and data handling too.

Interview angle

Name attack classes, describe generation and triage, and turn findings into regression cases. A strong answer covers reproducibility and corpus ageing. Running a jailbreak list once, with no severity or provenance, is the weak version.

Sources