Explainable AI (XAI) — AI Glossary

Reference details

Related terms

• Interpretability
• LIME (Local Interpretable Model-Agnostic Explanations)
• SHAP (SHapley Additive exPlanations)

How it works

1. Perturbation methods need only the ability to call the model, which is what makes them applicable to a hosted API where the weights are unavailable — they vary the input repeatedly and read the output.
2. Because the account is fitted to the samples drawn, a method with random sampling returns a different explanation on a second run, and the variance is a property of the procedure rather than of the model.
3. Shapley-based attribution defines a feature's contribution as its average effect across all subsets of the other features, which is exponential in the feature count and therefore always approximated.
4. Gradient methods instead require white-box access and differentiate the output with respect to the input, giving a result in one backward pass rather than many forward ones.
5. The two families answer slightly different questions — one about response to intervention, one about local sensitivity — which is why they can disagree on the same prediction without either being wrong.
6. Every one of them describes behaviour in a neighbourhood of a single input, so an explanation is evidence about one decision and carries no commitment about the next.

Example: The explanation explains the surrogate

A post-hoc method reports which input features mattered by probing the model from outside — perturbing inputs and watching outputs, then fitting something simple to that local behaviour. What you get is a faithful account of the simple thing, and an approximation of the model. Two methods run on one prediction routinely disagree, and one method run twice can disagree with itself when its sampling is random. That is not a reason to abandon them; it is a reason to state what they are. An explanation is evidence about behaviour near one input, not a description of the model's reasoning.

Commonly confused with

Explainability and interpretability are used interchangeably but pull apart usefully: interpretability is a property of a model you can read directly, explainability is the practice of accounting for a model you cannot. A chain-of-thought trace is neither — it is generated text about the answer, not a record of the computation that produced it, and it can be fluent and wrong.

When to use it

Reach for it when:

• Regulated decisions where a rationale must be produced and defended
• Debugging: finding which feature or token a model is keying on
• Auditing for proxy discrimination, where the question is what stood in for a protected attribute

Reach for something else when:

• As proof of causation, or as a guarantee the model will behave that way elsewhere
• Treating a self-reported rationale as an explanation of the computation
• Where an inherently interpretable model would do the job — then you need no surrogate

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Explainable AI (XAI) — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path