ModelRefs / AI Fairness — AI Glossary

AI Fairness — AI Glossary

The property of an AI system producing equitable outcomes across different demographic groups without unjustified disparate impact.

Overview

Fairness in ML has multiple formal definitions (demographic parity, equalized odds, individual fairness) that are mathematically incompatible in general. For LLMs: measuring output quality, sentiment, and stereotype propagation disparities. Fairness interventions include data rebalancing, adversarial debiasing, and post-processing calibration.

Reference details

Topic	safety
Last reviewed	2026-06-24

How it works

A fairness claim requires three things named explicitly: which groups, which outcome, and which definition of equal. Without all three the claim is not falsifiable, which is why most of them are not.
The common formal definitions measure different quantities. Demographic parity equalises the rate of a positive outcome across groups; equalised odds equalises error rates conditional on the true label; calibration requires a predicted score to mean the same thing in every group.
Those definitions are provably incompatible in general: when base rates differ between groups, no classifier can satisfy demographic parity, equalised odds and calibration simultaneously except in degenerate cases. Choosing one is therefore choosing against the others, and it is a policy decision rather than a technical one.
For language models the outcome is not a single label, so measurement works by comparison — matched prompts varying only the group-referencing term, scored for output quality, sentiment, refusal rate or stereotype content.
Interventions sit at three points: rebalancing or augmenting training data, penalising group-predictive signal during training, and adjusting thresholds or outputs after the fact. Post-processing is the most controllable and the most visibly a policy choice.
Measurement needs group labels, which are frequently unavailable or themselves sensitive — so the practical constraint on fairness work is usually that the data required to audit it cannot be collected.

Example: Why two reasonable definitions cannot both hold

Two groups, 1,000 people each. In group A 100 are genuinely positive, in group B 200 — base rates of 10% and 20%. A classifier with 90% recall and 90% specificity applied identically to both selects, in A, 100 x 0.9 + 900 x 0.1 = 90 + 90 = 180 people, and in B, 200 x 0.9 + 800 x 0.1 = 180 + 80 = 260. Error rates are equal by construction, so equalised odds holds — and selection rates are 180 / 1,000 = 18% against 260 / 1,000 = 26%, so demographic parity fails. To force parity you must select 220 from each: raise A's threshold-crossing count by 40 and lower B's by 40, which necessarily changes A's false-positive rate and B's false-negative rate away from each other, breaking equalised odds. The incompatibility is arithmetic, not a modelling deficiency, and it comes entirely from the 10% versus 20% base-rate difference. Whichever you choose, you are deciding whose errors to increase, and the only dishonest option is to claim the system is simply fair.

Commonly confused with

Fairness is not bias detection and not toxicity. Toxicity scoring asks whether an output is offensive; bias detection asks whether a model carries stereotypical associations; fairness asks whether outcomes are equitable across groups under a stated definition. A model can produce entirely non-toxic, non-stereotyped text and still allocate a resource unequally, which is a fairness failure the first two measurements cannot see. Conversely a model with measurable associations in its embeddings may produce equitable outcomes in a specific deployment, because fairness is a property of a system in a context rather than of weights.

When to use it

Reach for it when:

Allocating a resource or opportunity — credit, employment screening, access to a service — where outcomes affect people differentially.
Where a regulatory regime requires it, in which case the definition is frequently specified for you and the choice is made.
With the definition, groups and outcome stated in writing, so the claim can be tested and the trade-off is on the record.
Monitored over time, since deployment shifts the input distribution and a fairness property measured once does not persist.

Reach for something else when:

As an unqualified claim. 'The model is fair' names no definition, no groups and no outcome, and cannot be checked.
Expecting several definitions to hold at once, which differing base rates make arithmetically impossible.
Measured without group labels, where the required data is absent and a reported result is inferred rather than observed.
As a substitute for asking whether the system should be built. A fair automated decision can still be the wrong mechanism for the decision.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to AI Fairness — AI Glossary.

Frequently asked questions

What is AI Fairness?