Adversarial Attack — AI Glossary

Reference details

Related terms

• Prompt Injection
• Guardrails
• AI Alignment

How it works

1. The attacker picks an objective — a specific wrong output, a policy violation, or an action the system can take — and treats the model as a function to be steered toward it.
2. With white-box access, gradients over the input are used to search for a perturbation that moves the output toward the objective while keeping the input plausible.
3. With only API access the search is black-box: many queries, each probing how the output moves, and transferring suffixes found against an open model to a closed one.
4. The perturbation is kept small or innocuous-looking — a character swap, an appended suffix, a framing sentence — so it survives review by a human skimming the input.
5. Defences are layered because none is sufficient: adversarial training to harden the model, input normalisation to strip known payload shapes, and output filtering to catch what still gets through.

Example: Why 'imperceptible' is a claim about the reader, not the model

A moderation classifier is given the same sentence twice, once with a Cyrillic 'а' substituted for the Latin 'a' in a single word. To a reader the two strings are identical. To the tokeniser they are different sequences, so the embedding the classifier sees is different, and a decision boundary tuned on the Latin spelling need not hold. The lesson is mechanical rather than adversarial: anything that changes tokenisation changes the model's input, and visual identity is no guarantee of input identity.

Commonly confused with

Used as an umbrella for jailbreaks and prompt injection, which sits above both rather than beside them. Adversarial attack is the general category of crafted input intended to produce unintended behaviour; a jailbreak is the subset aimed at overriding safety training, and prompt injection is the subset that smuggles instructions through content the system ingests. The distinction worth keeping is the older, narrower sense — gradient-found perturbations that are imperceptible to a human — because that is the sense in which 'adversarial example' is used in the vision literature, and it describes a different defence problem from either.

When to use it

Reach for it when:

• Threat-modelling a model whose output gates a decision an attacker benefits from changing.
• Reasoning about why a classifier that scores well on a clean test set still fails in deployment.
• Justifying input normalisation and output filtering as separate layers rather than alternatives.

Reach for something else when:

• Describing ordinary distribution shift or a model failing on inputs it was never trained for; nobody crafted those.
• Describing a benign user hitting an edge case — the defining feature is intent.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Adversarial Attack — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path