ModelRefs / Interpretability — AI Glossary
Interpretability — AI Glossary
Methods for understanding the internal representations and decision processes of neural networks. Also called model interpretability or explainability.
Overview
Interpretability encompasses: saliency methods (which input tokens influenced the output), probing classifiers (what information is encoded in layer activations), circuit analysis (mechanistic interpretability), and concept activation vectors (TCAV). For LLMs: sparse autoencoders and activation patching are current frontiers in understanding transformer internals.
Reference details
| Topic | safety |
|---|---|
| Also known as | model interpretability, explainability |
| Last reviewed | 2026-06-24 |
Related terms
How it works
- Pick the question first, because the method follows from it: which input mattered, what a layer encodes, or what algorithm the network implements.
- Attribution methods perturb or differentiate with respect to the input to score how much each token moved the output.
- Probing trains a small classifier on frozen activations. If it recovers a property, the information is linearly present in that layer — though that does not show the model uses it.
- Causal methods intervene rather than observe: patch an activation from one run into another and see whether the output follows. This distinguishes correlation from mechanism.
- Sparse autoencoders decompose activations into many sparse features, addressing superposition — the fact that a single neuron participates in several unrelated concepts and so resists reading directly.
Example: Why probing and patching answer different questions
A probe on layer 12 recovers whether the subject of a sentence is plural, at well above chance. That establishes the information is there and linearly readable. It does not establish the model uses it: an activation patch — swapping that activation from a singular-subject run into a plural one — tests whether the prediction changes. Observation alone can show a property is encoded while the model's actual computation routes around it, which is why causal intervention became the standard the field moved toward.
Commonly confused with
Used interchangeably with explainability, and the useful split is what is being produced. Explainability aims at an account a stakeholder can act on — why this application was declined — and may be a faithful simplification. Interpretability aims at what the network actually computes, which is often not expressible as a short reason. Mechanistic interpretability is the narrower programme inside it, reverse-engineering specific circuits rather than attributing single predictions. The gap matters in deployment: a plausible explanation is not evidence of the mechanism, and regulators asking for one are not thereby getting the other.
When to use it
Reach for it when:
- A failure is systematic and you need the mechanism rather than more examples of it.
- Verifying that a model relies on the feature you intended rather than a correlate of it.
- Safety work where knowing a behaviour is absent from the mechanism matters more than not observing it.
Reach for something else when:
- Producing a user-facing reason for a decision — that is explainability, with different standards.
- As assurance: current methods explain fragments of a network, not the whole of one.
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Interpretability — AI Glossary.
Frequently asked questions
What is Interpretability?
Methods for understanding the internal representations and decision processes of neural networks.
Is Interpretability the same as model interpretability?
Yes — model interpretability, explainability are common aliases for Interpretability.
What concepts are related to Interpretability?
Closely related concepts include mechanistic interpretability, xai, shap.