Model Stealing — AI Glossary

Reference details

Related terms

• Adversarial Attack
• LLM Security
• Knowledge Distillation

How it works

1. The attacker treats the target API as an oracle and selects queries that probe its decision surface rather than queries a real user would send.
2. Inputs are chosen for information gain — near suspected boundaries, or synthesised to cover the input space evenly — so each paid query constrains the surrogate as much as possible.
3. Returned outputs become training labels, and richer outputs leak more: full logprobs over a vocabulary carry far more signal per query than a single sampled token.
4. A surrogate is then trained by distillation on the collected pairs, learning to reproduce the target's input-output mapping without ever seeing its weights.
5. Fidelity is measured as agreement with the target on held-out inputs rather than as task accuracy, because the objective is imitation rather than correctness.
6. Defences raise the cost per bit extracted: rate limits cap query volume, logprob suppression cuts signal per query, and output perturbation adds noise the surrogate must average away.

Example: Extraction is priced in API calls, and rate limits only buy time

Price a one-million-query extraction campaign against an API at $3 per million input and $15 per million output tokens, using 200 input and 300 output tokens per query. Per query that is 0.0002 x $3 = $0.0006 plus 0.0003 x $15 = $0.0045, so $0.0051, and a million of them cost $5,100. Set that against the cost of training a comparable model from scratch — compute, data, and the research that produced it — and the asymmetry is the whole problem: extraction is priced in thousands while origination is priced in millions. Rate limiting changes the schedule rather than the verdict. At 1,000 requests per minute, 1,000,000 queries take 1,000 minutes, which is 1,000 / 60 ≈ 16.7 hours. Tighten the limit to 60 requests per minute and the same campaign takes 1,000,000 / 60 ≈ 16,666.7 minutes, which is about 277.8 hours or roughly 11.6 days — a factor of about 16.7 longer, matching the ratio of the two limits. So a sixteenfold tightening converts an overnight job into a fortnight's patience, and $5,100 of API spend is unchanged. Suppressing logprobs is the more consequential defence, because it attacks signal per query rather than queries per hour: a surrogate learning from sampled tokens alone needs far more of them to recover the same distribution, and that multiplies the $5,100 rather than rescheduling it.

Commonly confused with

Model stealing is not training-data extraction, and the two have different targets and different remedies. Stealing copies the function — the input-output behaviour — and succeeds when the surrogate agrees with the target on new inputs. Data extraction recovers specific memorised training examples, and succeeds when verbatim text comes back out. A model can be fully stolen while leaking no training data, and can leak training data without being usefully stolen. It is also distinct from legitimate distillation, which is the same technique applied to a model you are licensed to distil from: the mechanism is identical and the difference is entirely contractual, which is why most provider terms address distillation explicitly rather than relying on a technical control that does not exist.

When to use it

Reach for it when:

• As a threat model when exposing a model whose training cost is the asset being protected.
• When deciding whether to return logprobs, since signal per query is the term that multiplies attacker cost rather than delaying it.
• In terms-of-service drafting, where distillation restrictions do the work no technical control can.
• When pricing access, because per-query cost is the only defence that scales with the volume an extraction campaign needs.

Reach for something else when:

• As an argument against any public API, given that the same exposure is what makes the model useful.
• Addressed by rate limiting alone, which stretched the worked campaign from about 16.7 hours to 11.6 days at identical cost.
• Conflated with training-data leakage, which needs different measurement and different mitigations.
• As a reason to degrade output quality for all callers, where the cost falls on legitimate users and the attacker averages the noise away.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Model Stealing — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path