Skip to content
AI Engineer Guide.

concept · evaluation · intermediate · concept 23 of 28

Red teaming

0 of 28 complete

Why it matters

Evaluation can only measure the failures someone already imagined. Red teaming explores the rest by combining human creativity with model-generated attempts, then triaging what it finds. It is closer to a security engagement than to a benchmark: the deliverable is a prioritised set of reproducible findings, not a number, and the value disappears if nobody owns the fixes. The scope has to be written down in advance, because an unscoped session drifts toward whatever the team finds interesting and leaves the target surface untested.

Mental model

It is hiring a burglar to test your locks. The metaphor breaks because this burglar can also talk your staff into handing over keys, which no lock test covers, so the exercise has to include the people and the process.

How it works

Mechanism, step by stepInteractive
  1. 1Write the scope: surfaces, harm categories, attacker assumptions and out-of-scope behaviour.
  2. 2Run a baseline of known attacks so the team does not rediscover them.
  3. 3Combine human sessions with model-generated attempts at scale, logging every transcript.
  4. 4Triage each finding by impact, reproducibility and whether it is a new root cause.
  5. 5Fix, then add a regression case with the exact input and expected behaviour.
  6. 6Publish an internal summary of classes covered and classes still open.

Step 1 of 6

Write the scope: surfaces, harm categories, attacker assumptions and out-of-scope behaviour.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Write the scope: surfaces, harm categories, attacker assumptions and out-of-scope behavio… 2. Run a baseline of known attacks so the team does not rediscover them. 3. Combine human sessions with model-generated attempts at scale, logging every transcript. 4. Triage each finding by impact, reproducibility and whether it is a new root cause. 5. Fix, then add a regression case with the exact input and expected behaviour. 6. Publish an internal summary of classes covered and classes still open.

The tail survives the filterInteractive

The interactive figure loads with JavaScript. The text version below describes the same frame.

Figure. The share of ten-attempt runs in which each attack family succeeds at least once, before and after a filter, counted per family and ordered by the unfiltered rate. The head is loud and the filter cuts it; the small families persist, so a per-attempt pass rate overstates the system's run-level safety. Rates are illustrative.

What to notice. At ten attempts per family the average unfiltered family breaks in 62.3% of runs and the top family in 98.4%; the filter brings those to 33.0% and 72.1%. Role-play still breaks in more than seven runs in ten, and the smallest family in nearly one in ten. An attempt-level pass rate of 95.6% would report the filtered tail as healthy, which is why the distribution across families is the number that matters.

Text version

Horizontal bars for twelve attack families, each with an unfiltered and a filtered share of ten-attempt runs that succeed at least once. Role-play is 98.4% unfiltered and 72.1% filtered, encoding 93.6% and 46.1%, multi-turn 90.5% and 61.1%, tool abuse 84.5% and 33.5%, prompt leak 77.9% and 51.6%, exfiltration 72.1% and 40.1%, policy 61.1% and 18.3%, injection 51.6% and 26.3%, splitting 40.1% and 18.3%, translation 33.5% and 9.6%, persona 26.3% and 9.6%, and other 18.3% and 9.6%. A vertical axis lists the families and the horizontal axis is the share of runs. Three buttons switch the attempt budget to 1 or 3; a panel reports the expected number of broken families per run and the top and average filtered family.

  1. Write the scope: surfaces, harm categories, attacker assumptions and out-of-scope behaviour.
  2. Run a baseline of known attacks so the team does not rediscover them.
  3. Combine human sessions with model-generated attempts at scale, logging every transcript.
  4. Triage each finding by impact, reproducibility and whether it is a new root cause.
  5. Fix, then add a regression case with the exact input and expected behaviour.
  6. Publish an internal summary of classes covered and classes still open.

Trade-offs

ChoiceGood forCostFails when
Human red teamNovel and context-awareCost, limited hoursYou need broad coverage
Automated red teamScale and repetitionShallow, repetitive findingsAttack needs multi-turn context
Continuous programmeFinds regressionsOngoing cost and triageOne-off compliance goal
Bug bountyReal attackers, real incentivesDisclosure risk, variable qualityPre-launch and confidential
Trade-off mapInteractive

Good for

Novel and context-aware

Cost

Cost, limited hours

Fails when

You need broad coverage

  1. You need broad coverage
  2. Attack needs multi-turn context
  3. One-off compliance goal
  4. Pre-launch and confidential

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Human red team is good for Novel and context-aware, costs Cost, limited hours, and fails when You need broad coverage. Automated red team is good for Scale and repetition, costs Shallow, repetitive findings, and fails when Attack needs multi-turn context. Continuous programme is good for Finds regressions, costs Ongoing cost and triage, and fails when One-off compliance goal. Bug bounty is good for Real attackers, real incentives, costs Disclosure risk, variable quality, and fails when Pre-launch and confidential

Failure modes

  • No triage owner. Findings accumulate in a document and nothing is fixed. Debug: check each finding for an owner and a linked change.
  • Scope creep. The session spends its hours on out-of-scope behaviour and misses the target. Debug: compare findings against the written scope categories.
  • Duplicate findings. The same root cause is reported many times and inflates the count. Debug: group findings by root cause before reporting totals.

In production

Track open findings by severity and age, and regression cases added per finding. Schedule a cycle after each major capability change.

Common misconceptions

People think: Red teaming produces a security score.

Actually: It produces findings, and scoring is a separate evaluation activity.

People think: One successful round means the system is hardened.

Actually: Attackers adapt, and a round samples a small part of the space.

Interview angle

Separate red teaming from evaluation, describe scoping and triage, and require reproduction and ownership. A strong answer combines human and automated attempts. A one-off jailbreak session, with no scope or path to a fix, is the weak response.

Sources