ModelRefs / Bias Detection — AI Glossary

Bias Detection — AI Glossary

Measuring systematic disparities in model outputs across demographic groups (gender, race, religion, nationality) to identify fairness violations.

Overview

Bias detection evaluates: representation bias (do certain groups appear as protagonists/antagonists?), stereotype propagation, sentiment differential (do names from different groups trigger different sentiments?), and performance gaps on benchmarks by group. Tools: Fairlearn, AI Fairness 360, CrowS-Pairs benchmark, WinoBias.

Reference details

Topic	safety
Last reviewed	2026-06-24

How it works

The groups and the harm are named first, because a disparity is only measurable against a stated comparison and a stated outcome.
Inputs are constructed as matched pairs or counterfactuals differing only in the group attribute, so any output difference is attributable to that attribute.
Outputs are scored on the same rubric across groups, and the metric is the gap between groups rather than any single group's score.
Results are disaggregated and reported with group sample sizes, since a small group's gap carries wide uncertainty and an aggregate hides it entirely.
A fairness criterion is chosen explicitly — equal error rates, equal selection rates, calibration — knowing these are mutually incompatible in general, so the choice is a decision rather than a default.

Example: Why the aggregate cannot show you the gap

A classifier is 90% accurate overall. Split by group: 93% on a group that is 85% of the data, 73% on one that is 15%. The weighted average is 0.85 × 93 + 0.15 × 73 = 79.05 + 10.95 = 90 — exactly the headline figure, with a 20-point gap inside it. No amount of staring at the aggregate reveals this, which is why disaggregation is the method rather than a refinement of it. The smaller group would also need a reasonable n before the 73% means anything, so group sample sizes belong next to the gap.

Commonly confused with

Used as though it were bias mitigation, and the two are separate steps with separate risks. Detection measures a disparity; mitigation changes something to reduce it — reweighting, augmenting data, adjusting thresholds — and can trade one fairness criterion against another or against overall accuracy. The deeper issue is that 'bias' names at least three things: a statistical disparity in outputs, a societal unfairness in a decision, and the inductive bias that lets a model generalise at all. Detection work addresses the first and is usually asked to stand in for the second, and a measured gap is evidence about the model, not yet a determination about the decision.

When to use it

Reach for it when:

A model affects people differently by group and you need measurement rather than assurance.
A framework or audit requires disaggregated performance evidence.
Before mitigation, so there is a baseline the change can be judged against.

Reach for something else when:

As a fairness certificate — a passing score on chosen metrics for chosen groups is not a general claim.
Where no group attribute is available or inferable, in which case inferring one adds a harm of its own.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Bias Detection — AI Glossary.

Frequently asked questions

What is Bias Detection?