Data Poisoning — AI Glossary

Reference details

Related terms

• Adversarial Attack
• AI Alignment
• LLM Security

How it works

1. The attacker needs write access to something the training pipeline ingests — an editable wiki, a scraped forum, a public dataset mirror, or a contractor-supplied annotation batch.
2. Poisoned examples pair a rare trigger phrase with the attacker's desired output, so the model learns a conditional mapping that only fires when the trigger appears.
3. Rarity is the whole design: a trigger that never occurs in legitimate text means the backdoor never activates during normal evaluation, so held-out accuracy stays intact.
4. Gradient updates treat the poisoned pairs as ordinary supervision, and because the trigger is unambiguous the mapping is learned from far fewer examples than a general capability needs.
5. At inference the backdoor is dormant until an input contains the trigger, at which point the learned mapping overrides the behaviour the rest of training produced.
6. Defences work on provenance and influence rather than on output testing: tracing which examples moved which weights, and refusing to train on data whose origin cannot be established.

Example: The cost asymmetry between poisoning and auditing

Consider a fine-tuning corpus of 500,000 examples and an attacker who manages to insert 250 of them, which is 250 / 500,000 = 0.05% of the data. Authoring those 250 examples at two minutes each costs 500 minutes, or about 8.3 hours of work. Now price the exhaustive defence. Reviewing all 500,000 examples at five seconds each is 2,500,000 seconds, which is 2,500,000 / 3600 ≈ 694.4 hours, or 694.4 / 40 ≈ 17.4 person-weeks. So roughly 8.3 hours of attack obliges about 694.4 hours of review to be certain, an asymmetry of about 83 to 1 — and that is why exhaustive review is not the control anyone actually runs. Sampling does not close it either. A 1% audit inspects 5,000 examples and expects to encounter 250 x 0.01 = 2.5 of the poisoned ones, so the audit either finds two or three of them and raises an alarm, or finds none and certifies a corpus that is fully compromised. Detection by sampling is detection by luck at these proportions, which is the argument for spending the effort on provenance — knowing where each example came from costs once per source rather than once per example.

Commonly confused with

Data poisoning is not prompt injection, and confusing them sends the defence to the wrong layer. Prompt injection happens at inference, in the context window, and is fixed by treating retrieved or tool-returned content as data rather than instructions; nothing about the weights changed, and restarting the conversation clears it. Poisoning happens at training time and is baked into the weights, so no amount of input sanitisation removes it and the only remedies are retraining from clean data or finding the backdoor. It is also distinct from a jailbreak, which coaxes a capability the model already has past its refusal behaviour: a poisoned model has been taught a behaviour it would not otherwise have. The practical tell is persistence — an injection dies with the context, a backdoor survives every deployment of that checkpoint.

When to use it

Reach for it when:

• As the threat model whenever training data includes scraped, crowdsourced, or third-party-supplied examples.
• When evaluating a fine-tuning vendor, where the question is what provenance guarantees cover their annotation pipeline.
• In model supply-chain review, since a downloaded checkpoint carries whatever its training data contained and no output test reliably reveals it.
• As the reason to retain the ability to retrain, because a discovered backdoor has no inference-time fix.

Reach for something else when:

• As an explanation for bad outputs on ordinary inputs, which is far more often a capability or prompting problem.
• Treated as an inference-time issue solvable by input filtering, which addresses injection and leaves a backdoor untouched.
• As a reason to avoid fine-tuning on data you control end to end, where provenance is already established.
• Diagnosed by held-out accuracy, which a well-built backdoor is specifically designed to leave unchanged.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Data Poisoning — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path