AI Governance Engineer
GitHub

Counterfactual flip test

By Updated

A test that changes only a protected attribute in an input, or swaps identity terms in otherwise identical prompts, and measures how often the outcome or the answer quality changes. It is the practical approximation of counterfactual fairness 1 and runs in the fairness eval suite.

Developed in
ch. 16, Individual and counterfactual fairness
ch. 05, Pattern: Fairness Eval Suite
Chapters
ch. 05, Patterns · ch. 16, Fairness & XAI
Contrast with
Counterfactual fairness
Source
1 numbered reference, listed below

Where it is used

2 chapters of the Body of Knowledge use the term. Each link opens the first section that does.

Counterfactual flip test: mentions per chapter4 mentions of the term across 2 of the 23 chapters of the Body of Knowledge; the chapters the glossary cross-references are drawn solid.Cross-referenced chapter05 Patterns: 1 mention05 Patterns116 Fairness & XAI: 3 mentions16 Fairness & XAI3No mentions in the other 21 chaptersSource: the Body of Knowledgev0.5.0
Counterfactual flip test: mentions per chapter4 mentions of the term across 2 of the 23 chapters of the Body of Knowledge; the chapters the glossary cross-references are drawn solid.Cross-referenced chapter000102030405 Patterns: 1 mention10506070810111213141516 Fairness & XAI: 3 mentions31617181920212223Source: the Body of Knowledge v0.5.0
Mentions per chapter How often each chapter names the term, with the chapters the glossary cross-references drawn solid. The longest bar is where to read the term in use; bars spread across many chapters mark a term that runs through the discipline.
Mentions as a table
Mentions of Counterfactual flip test per chapter
ChapterMentionsCross-referenced
00 Preface0No
01 Definition0No
02 Why Now0No
03 Values & Principles0No
04 The Stack0No
05 Patterns1Yes
06 The Role0No
07 Maturity Model0No
08 Regulatory Map0No
10 Reading List0No
11 AI Defined0No
12 Governance Program0No
13 Risk Management0No
14 Development0No
15 Deployment0No
16 Fairness & XAI3Yes
17 Incidents0No
18 EU AI Act0No
19 Privacy & AI0No
20 Existing Law0No
21 AI Laws Worldwide0No
22 Principles & Standards0No
23 AI Agents0No

Patterns that use this term

One pattern page uses the term.

Counterfactual flip test: related terms and patternsThe term at the centre; related terms on solid edges, the terms it is contrasted with on dashed edges, and the patterns that use it most in the colour of their layer.RelatedContrastColour: stack layer of the pattern03 EvalsCounterfactual flip testRelated terms (3)Fairness · RelatedFairnessFairness policy · RelatedFairness policyProxy scan · RelatedProxy scanContrast with (1)Counterfactual fairness · ContrastCounterfactual fairnessPatterns using it (1)Fairness Eval Suite, 3 mentions · Uses the termFairness Eval Suite
Counterfactual flip test: related terms and patternsThe term at the centre; related terms on solid edges, the terms it is contrasted with on dashed edges, and the patterns that use it most in the colour of their layer.RelatedContrastColour: stack layer of the pattern03 EvalsRelated terms (3)Fairness · RelatedFairnessFairness policy · RelatedFairness policyProxy scan · RelatedProxy scanContrast with (1)Counterfactual fairness · ContrastCounterfactual fairnessPatterns using it (1)Fairness Eval Suite, 3 mentions · Uses the termFairness Eval SuiteCounterfactualflip test
The term's neighbourhood Terms related to this one on solid edges, terms it is often confused with on dashed edges, and the patterns that use it most in the colour of their layer. Follow a dashed edge to check the distinction before you use either term.
Neighbourhood as a table
Neighbourhood of Counterfactual flip test
FamilyItemRelation
Related termsFairnessRelated
Related termsFairness policyRelated
Related termsProxy scanRelated
Contrast withCounterfactual fairnessContrast
Patterns using itFairness Eval Suite, 3 mentionsUses the term
  • Fairness policy

    The per-system record, fixed before results are seen, of what fairness means for that system: the protected attributes in each jurisdiction and where their…

  • Proxy scan

    A test that trains a model to predict a protected attribute from a system's features; features that predict it strongly are flagged as proxies to justify or…

  • Fairness

    The property that a system's outcomes and errors do not unjustifiably disadvantage people or groups.

Sources

  1. [1] "Counterfactual Fairness" (Kusner, Loftus, Russell and Silva; arXiv 1703.06856). arXiv. 2017-03-20. https://arxiv.org/abs/1703.06856 (verified: primary)

Definitions of legal terms paraphrase the cited text, which governs. Dated statements are as of .

Cite this term

García Aibar, J. (2026). Counterfactual flip test. In AI Governance Engineering: The Thesis & Body of Knowledge (v0.5.0), Glossary. https://doi.org/10.5281/zenodo.22956197. https://aigovernanceengineer.com/glossary/counterfactual-flip-test. CC BY 4.0

BibTeX

@misc{aige2026counterfactualfliptest,
  author  = {Jorge García Aibar},
  title   = {{Counterfactual flip test}},
  note    = {Glossary, AI Governance Engineering: The Thesis \& Body of Knowledge, version 0.5.0},
  year    = {2026},
  doi     = {10.5281/zenodo.22956197},
  url     = {https://aigovernanceengineer.com/glossary/counterfactual-flip-test}
}