ModelRefs / Red Teaming — AI Glossary

Red Teaming — AI Glossary

Adversarial testing of an AI system by humans (or other models) attempting to elicit harmful, biased, or off-policy outputs.

Overview

Red teaming is required practice for frontier model releases and a core EU AI Act / NIST AI RMF expectation. Automated red teaming uses adversarial LLMs to scale coverage. Findings feed safety training and guardrail updates.

Reference details

Topic	safety
Last reviewed	2026-06-24

How it works

A scope is agreed first: which harms are in scope, which surfaces may be attacked, and what counts as a finding, so results are comparable between rounds.
Testers attack deliberately rather than sampling typical traffic — role-play framings, encoded payloads, multi-turn setups that establish premises before the ask, and injection through any content the system ingests.
Each success is recorded as a reproducible case: the exact input, the context, and the output, so it can be replayed against a later build.
Automated red teaming points an adversarial model at the target to generate attack candidates at volume, with humans triaging what the generator cannot judge.
Findings route two ways — into safety training or guardrail rules as fixes, and into the evaluation suite as regression cases so the same attack cannot silently return.

Example: Why a finding is worth little until it is a test case

A tester elicits a disallowed response and files a description: 'model complied with a harmful request when asked in a role-play framing.' Three builds later nobody can tell whether it still happens, because the exact prompt was not kept. Recorded instead as a replayable case — input, context, expected refusal — the same attack becomes a line in the regression suite that runs on every release. The mechanism matters more than the count: a hundred described findings give you a report, while twenty replayable ones give you a ratchet.

Commonly confused with

Read as a synonym for evaluation, which inverts the sampling. An eval measures behaviour on representative inputs and asks how good the system usually is; red teaming searches for inputs that break it and asks what the worst reachable case looks like. The success criteria are opposite — a good eval score is high, a good red-team outcome is finding something. It is also not penetration testing, which targets the infrastructure; red teaming targets the model's behaviour, and a system can be perfectly secure at the network layer while remaining trivially steerable through its prompt.

When to use it

Reach for it when:

Before releasing a model or an agent that will face untrusted input.
When a regulatory framework expects adversarial testing evidence as part of a risk assessment.
After changing safety training or guardrails, to check the change did not open a path it previously closed.

Reach for something else when:

Measuring ordinary quality or regression on typical traffic — that is what an eval suite is for.
As a one-off sign-off: a single round describes the build it was run against, not the system.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Red Teaming — AI Glossary.

Frequently asked questions

What is Red Teaming?