Spotting Bias in AI:
A Practical Guide for Everyone

Bias in AI is no longer a research paper problem. It is a deployed-systems problem, and the systems are running right now — in hiring pipelines, healthcare triage tools, financial services, and the large language models most of us use every day.

This post is a practitioner's guide to spotting it. It is written for builders, buyers, and users of AI.

Two systems that failed

The Optum algorithm. In 2019, Ziad Obermeyer and colleagues published a study in Science examining a commercial healthcare risk algorithm built by Optum, a subsidiary of UnitedHealth Group. The algorithm was widely used across US hospital systems to identify patients who needed extra care. Essentially, a triage tool deciding who gets more attention.

The algorithm used healthcare cost as a proxy for healthcare need. That sounds reasonable. Sicker people cost more, right?

The problem is that due to decades of systemic racism and unequal access to care, Black patients had historically received less treatment than white patients with equivalent health conditions. So they had lower costs not because they were healthier, but because the system had consistently under-served them.

The model learned from that history. At any given risk score, Black patients were significantly sicker than white patients at the same score. The researchers estimated the algorithm was influencing care for over 200 million Americans and had reduced the share of Black patients receiving additional care by more than half. After publication, Obermeyer worked with Optum to retrain the model and achieved an 84% reduction in bias but the algorithm had been running at scale for years before anyone looked.

The LLM hiring assistant. Wilson and Caliskan published research in 2024 showing that LLM-based resume screening systems favoured white-associated names over Black-associated names at identical credentials. The bias appeared not as explicit discrimination but as learned preference baked into the retrieval and ranking mechanism. This is the documented pattern of a technology many organisations are adopting right now.

What these two cases share is not malice. They share a common mechanism: a model trained on data that reflected existing inequities learned to perpetuate and amplify those inequities, at scale, without any explicit instruction to do so.

Where bias enters in generative and agentic AI

For predictive systems (classifiers, risk scores, recommendation engines), the research community has spent a decade developing frameworks, toolkits, and regulatory guidance. Buolamwini and Gebru's Gender Shades work (2018) showed commercial facial recognition systems had error rates of up to 34.7% for darker-skinned women, compared to less than 1% for lighter-skinned men. NIST, the EU AI Act's high-risk provisions, Microsoft's Fairlearn toolkit, IBM's AIF360; the infrastructure for auditing AI systems (particularly predictive systems) is forming.

Generative and agentic AI present a different set of problems. The failure modes are less well understood, the tooling is thinner, and the pace of deployment has outrun the governance. Bias enters across five distinct stages.

Stage 1 — Pretraining

The model learns its world from a corpus. That corpus reflects who contributed to it.

Voice recognition systems trained predominantly on American English perform measurably worse on Scottish accents, Welsh accents, Nigerian English, Singaporean English. Not because those speakers are less articulate but because the training data had a particular shape. Scale that mechanism up. If a language model was trained on text predominantly produced by Western, English-speaking, college-educated writers, it knows those writers' professional registers, naming conventions, cultural references, and ways of describing experience better than it knows anyone else's. UNESCO documented in 2024 that Llama 2 described women in domestic roles four times more often than men. That pattern came from corpora that reflected a world where that disparity exists.

Stage 2 — Alignment

After pretraining, models are typically fine-tuned using human feedback. Human labellers tell the model what a "good" response looks like. RLHF labeller pools have historically been predominantly Western, college-educated, English-speaking. Their notion of "professional" tone becomes the model's default not because of any explicit instruction, but because that is what the feedback signal encoded. This is frequently referred to as the WEIRD (Western, Educated, Industrialized, Rich, Democratic) population bias.

Stage 3 — Retrieval

RAG systems retrieve documents from a corpus to inform their responses. The composition of that corpus determines whose knowledge is surfaced and whose is paraphrased or omitted. Medical RAG systems built over Western peer-reviewed literature underserve patients whose conditions have been less studied in those contexts. HR systems built over a company's past hiring decisions inherit that company's past hiring patterns, including whoever was systematically excluded.

Stage 4 — Generation and action

This is where defaults become visible. When a user is vague, the model makes assumptions. What it assumes reveals what was baked in upstream. A simple test: "Top three pieces of advice for a father of young children" versus "top three pieces of advice for a mother of young children." In repeated testing across major LLMs, the mother prompt produced notably different framing — self-care language, depletion framing, an assumption of primary caring responsibility — while the father prompt produced builder framing, legacy language, advice oriented towards presence and provision. Same question. Different defaults.

In agentic systems, defaults don't just shape what gets generated, they shape what gets done. The bias often lives in the permission layer, not the output. An autonomous agent may auto-approve a loan application for one demographic and route an equivalent one to human review; an LLM hiring assistant may auto-schedule interviews for high-confidence candidates while the threshold for who counts as "high-confidence" is itself asymmetric across groups. The action isn't biased but the gating around the action is. And because the bias lives upstream of any single output, it's often invisible until someone deliberately tests for it.

Stage 5 — Feedback

The outputs of generative and agentic systems flow back into the world and in many cases, back into the training data for the next model. This feedback loop is the most structurally distinctive failure mode of modern AI. Synthetic content from current models is being indexed and crawled at scale; model collapse research has documented quality degradation when models train heavily on AI-generated data; and agentic systems that take real actions (writing emails, updating records, scheduling, making purchases) reshape the operational data those same systems read next. The system reshapes the environment it observes, and then learns from the environment it has shaped. The bias question stops being "what does the system output" and becomes "what world is the system creating?"

The audit: three checks, twelve questions

The canvas below is built around three checks, each targeting a different stage in this pipeline. It is designed to run in a thirty-minute team meeting, with whoever is available, without specialist tooling.

Q1 — Whose data was the model trained on? Whose languages, dialects, and perspectives are over- or under-represented in the pretraining corpus? Does the model card document this? If it doesn't, that silence is itself a finding.

Q2 — Who labelled the preferences during alignment? What context did those labellers bring? Whose register became the default? This is often completely undocumented in commercial model releases.

Q3 — For RAG systems: where does the retrieval corpus come from? Whose documents are indexed? Whose voices are systematically missing — and would those voices change the answer?

Q4 — Who wrote the system prompt and safety guidelines? Whose risks were taken seriously? Whose were treated as edge cases? A system prompt that defines "high-priority customer" or "appropriate use" reflects a set of values. Whose values?

Q1 — What does the system assume when the user is vague? Test it: send the same prompt with different demographic markers (names, locations, role descriptors) and observe what shifts.

Q2 — Whose claims does the system treat as authoritative? Who does it fact-check, who does it amplify? For RAG systems: whose sources get cited explicitly, and which voices get paraphrased without attribution?

Q3 — For agentic systems: what is the agent allowed to do, and for whom? Does it act autonomously for some users and require approval for others? Differential action thresholds across demographic groups, even with equivalent underlying profiles, are a form of algorithmic discrimination.

Q4 — Does the system refuse or lower its quality asymmetrically? Test refusal symmetry: the same content request, phrased identically, with different inferred user demographics. This is fully runnable on any system you can prompt.

Q1 — Disaggregated quality test. Run the same task across demographics, languages, and contexts. Where is generation quality lower? Where are refusals more frequent? Where does the system fabricate differently; inventing credentials, tenure gaps, or histories for some names but not others?

Q2 — Compound effects in multi-step or agentic systems. Do small biases compound across a pipeline? Run a representative end-to-end task and examine the aggregate outcome across groups not just each step individually. A five-stage pipeline where each step produces a 2% disparity can produce a 10%+ aggregate disparity.

Q3 — Recourse and contestation. When the system generates a stereotype, refuses inappropriately, or takes a harmful agentic action, what can the affected person do? Is there a contestation path? The UnitedHealthcare naviHealth case — an AI denying Medicare Advantage claims at disproportionate rates — became the basis of a class action lawsuit precisely because there was no meaningful review process.

Q4 — Feedback loop monitoring. Is the system reshaping the world it observes? Are its outputs becoming training data for future models? Not being able to answer this is, again, a finding.

When the audit raises something

The existing landscape

The canvas is not a replacement for established frameworks. It is an operationalisation layer that plugs into them.

• NIST AI Risk Management Framework (AI RMF 1.0, 2023) — the US government's voluntary framework for managing AI risk across the system lifecycle. The AI 600-1 companion document extends it to generative AI specifically.
• EU AI Act — now in force. High-risk AI provisions cover hiring, credit scoring, and healthcare triage. Separate GPAI provisions cover foundation models. If you operate in the EU, this is law, not guidance.
• EEOC guidance on AI hiring tools (2023) — existing US employment discrimination law applies to AI-assisted hiring. Employers cannot outsource liability to a vendor algorithm.
• CFPB guidance on algorithmic credit scoring (2022–23) — black-box credit models must still comply with fair lending law, including providing adverse action explanations.
• Microsoft Fairlearn — open-source toolkit for assessing fairness in ML classifiers. Built for the predictive world.
• IBM AIF360 — open-source toolkit with 70+ fairness metrics and mitigation algorithms.
• HELM (Stanford) — evaluation framework for LLMs that includes bias benchmarks.

Common questions

How is this different from a formal fairness audit? Different scope, different tooling, different stakes. A formal fairness audit measures specific disparity metrics with statistical rigour and specialist expertise. The canvas surfaces where to look — twelve questions runnable in thirty minutes. Think of it as what you run first to identify where you need the formal audit. For high-stakes systems, both matter; they do different jobs.

Does this apply to systems we don't own? Yes, especially Check 3. Disaggregation testing works on any system you can prompt. Check 2 Q1 and Q4 are also fully runnable on commercial systems. Check 1 is mostly opaque from the outside but noting that opacity is itself useful information.

Isn't some of this just the model being accurate? This is the most important objection. The bias is not in any individual piece of advice. The bias is in the differential treatment of the same prompt based on a demographic word. The user asked a general question. The model inferred a clinical state from the single word "mother." The correct behaviour is symmetric: if the user wants self-care advice, they should ask for it. The model should not infer which one is appropriate from a demographic marker. This is precisely the mechanism behind the Optum case; "accurate base rates" applied asymmetrically produced discriminatory outcomes.

How do you actually fix bias once you've found it? Honestly: that's a separate, harder problem, and the canvas intentionally doesn't focus on the fix. The canvas surfaces where the bias is. Fixing it can mean retraining, swapping the foundation model, restructuring the RAG corpus, redesigning the system prompt, or in some cases, deciding not to ship the feature at all. The Optum case shows what one fix looks like: Obermeyer's team worked with the company to retrain on a different label and reduced the disparity by 84%. But that took model access most teams don't have. Prevention is significantly cheaper than correction which is why the next question matters.

How should developers think about building less biased systems from the start? Bias is easier to prevent than correct. Diversify your training data before you start training. Treat your eval set as infrastructure — build it once, maintain it, run it on every change. Publish your fairness thresholds before you measure. Involve affected communities in defining success criteria.

What does the canvas not catch? A few important things. It doesn't catch overcorrection; the Gemini-diverse-Nazi-soldiers kind of failure where attempted fairness fixes produce their own absurd outputs. It doesn't catch jailbreaks or adversarial attacks; those need security testing. It doesn't catch jurisdiction-specific compliance gaps; those need a legal review. And it's not benchmark-grade measurement — for that, use the established suites (e.g. HELM, BBQ, BOLD). The canvas surfaces where to look. Specialists answer the hardest ones.