Membership Inference Attack — AI Glossary

Reference details

Related terms

• Differential Privacy
• PII Detection
• LLM Security

How it works

1. The attack exploits a generalisation gap: a model assigns higher likelihood to text it was trained on than to comparable text it was not, so likelihood is a signal about membership.
2. For language models the signal is usually per-token loss or perplexity on the candidate text, optionally normalised against a reference model to remove the effect of the text simply being easy to predict.
3. A threshold converts the signal into a decision, and where that threshold sits determines the true-positive and false-positive rates jointly.
4. Evaluation at average accuracy is misleading, because the useful operating point is high confidence on a few examples rather than slight confidence on many — so attacks are assessed by true-positive rate at a low false-positive rate.
5. Duplication in the training data amplifies the signal substantially, which is why memorised and repeated text is far more detectable than text seen once.
6. The same mechanism underlies extraction attacks and data-provenance auditing, so it is both an adversarial technique and the instrument used to check whether a specific document was used.

Example: Why average accuracy is the wrong way to report an attack

Suppose an attack achieves a 60% true-positive rate at a 40% false-positive rate on a balanced set — reported as accuracy it is 60%, which sounds like meaningful signal. As evidence about one specific record, the relevant quantity is the likelihood ratio: 0.6 / 0.4 = 1.5, meaning the observation is 1.5 times more likely if the record was in training than if it was not. That is almost no evidence; a claim resting on it would not survive challenge. Now the same 60% true-positive rate at a 1% false-positive rate gives 0.6 / 0.01 = 60, a likelihood ratio two orders of magnitude higher, from an attack whose accuracy is a less impressive (0.6 + 0.99) / 2 = 79.5% — the figure that matters moved by 40x while the headline moved by 19.5 points. This is why membership-inference results are reported as true-positive rate at a fixed low false-positive rate, and why an attack described only by its accuracy cannot be assessed. It also sets the bar for the legal use: proving a specific document was in a training set needs the low-false-positive regime, not an average.

Commonly confused with

Membership inference asks whether a record was in the training set; extraction asks the model to reproduce it. The two are related and have different thresholds of consequence: inference is a probabilistic claim about one record, while extraction produces the content itself and is therefore both stronger evidence and a more direct harm. Neither is the same as model inversion, which reconstructs representative inputs for a class rather than identifying a specific record. For a data-protection argument the distinction matters because the remedies differ — inference supports an audit claim, extraction demonstrates a disclosure.

When to use it

Reach for it when:

• Auditing whether specific documents appear in a training corpus, where this is the available instrument.
• Privacy evaluation of a model before release, where the question is how detectable training membership is.
• Supporting a data-provenance or erasure claim, which requires the low-false-positive operating point.
• Assessing deduplication effectiveness, since duplication is what makes membership detectable.

Reach for something else when:

• Reported as average accuracy, which obscures the only operating point that carries evidential weight.
• As proof about a single record from a high-false-positive attack, where the likelihood ratio is near 1.
• Confused with extraction, which produces content and is both stronger evidence and a different harm.
• On text that is widely duplicated outside the training set, where high likelihood says the text is common rather than that it was trained on.

Primary source

• Shokri et al., Membership Inference Attacks against Machine Learning Models (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Membership Inference Attack — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path