Mechanistic Interpretability — AI Glossary

Reference details

Related terms

• Interpretability
• AI Alignment
• Explainable AI (XAI)

How it works

1. The working hypothesis is that a network implements human-legible algorithms in identifiable subgraphs, so the research programme is to find those subgraphs rather than to correlate inputs with outputs.
2. Activation patching establishes causality: run the model on a clean input, run it on a corrupted one, then copy a single activation from one run into the other and measure how much of the output difference that restores. A component that restores most of it is carrying the computation.
3. The logit lens reads intermediate activations through the output head, showing what the model would predict if it stopped at that layer — which turns a stack of opaque vectors into a trajectory of partial predictions.
4. Individual neurons resist interpretation because of superposition: a layer represents more features than it has dimensions by assigning them overlapping, non-orthogonal directions, so one neuron participates in many unrelated features.
5. Sparse autoencoders attack that directly. They learn an overcomplete dictionary — far more features than the layer has dimensions — under a sparsity penalty, so each input activates few features and each feature is more likely to mean one thing.
6. Those features then need naming, and there are too many to name by hand, so the labelling is itself automated by a model and the labels inherit that model's errors. This is the programme's current bottleneck rather than a detail of it.

Example: Why the feature count forces automated labelling

Take a layer 4,096 dimensions wide and a sparse autoencoder expanding it 16x to 4,096 x 16 = 65,536 learned features, with a sparsity target of about 20 active per input. The dense activation has all 4,096 dimensions carrying signal at once; the sparse one has 20, so the number of things you must reason about per input falls by 4,096 / 20 = 204.8, roughly 200x. That is the win. The cost arrives at the next step: those 65,536 features exist per layer, so an 80-layer model holds 65,536 x 80 = 5,242,880 of them. At one minute to read a feature's top activating examples and write a label, that is 5,242,880 minutes — 5,242,880 / 60 = 87,381 hours, or about 87,381 / (24 x 365) = 10 years of uninterrupted work for one person. No labelling budget reaches that, which is why feature descriptions are generated by a model and why the reliability of those descriptions, rather than the decomposition itself, is what limits what the method can currently establish.

Commonly confused with

Mechanistic interpretability is not explainability, and the two answer different questions. Explainability methods — attention visualisation, feature attribution, saliency — produce a plausible account of which inputs mattered, and a plausible account is not a causal one: attention weights in particular are frequently read as explanations when they are intermediate quantities that may not determine the output at all. Mechanistic work insists on intervention, because only changing an activation and observing the effect distinguishes a component that computes something from one that merely correlates with it.

When to use it

Reach for it when:

• Safety questions that require knowing whether a behaviour is present rather than unobserved — deception, situational awareness, a capability the model conceals under evaluation.
• Diagnosing a specific, reproducible failure where you need the mechanism rather than a mitigation, so the fix generalises.
• Validating that a training intervention changed what you intended, which output-level evaluation cannot confirm.
• Research on representation itself — superposition, feature geometry, circuit universality across models.

Reach for something else when:

• As a production debugging tool. The techniques need white-box access, substantial compute and specialist time, and do not scale to an on-call rotation.
• On API-only models, where activations are not available and none of the methods apply.
• As a completeness claim. Interpreting some circuits does not establish that nothing else is happening, and absence of evidence is weak here.
• Where behavioural evaluation answers the question. If you need to know whether the model does X, test whether it does X.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Mechanistic Interpretability — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path