The word "algorithm" sounds technical, but the idea is simple: it is a set of instructions for making a decision. A recipe is an algorithm. So is the scoring system your school uses to calculate grades. The COMPAS tool used answers to a 137-question questionnaire — things like "Have you ever been arrested?" and "Do your friends get in trouble?" — plus demographic information, and combined them into a single number from 1 to 10.

The key thing to understand is that nobody sitting in a room decided "Black defendants should score higher." Instead, the tool learned patterns from historical data — old arrest records, old recidivism data, old court decisions. It found statistical correlations and encoded them as rules. If people in certain zip codes had historically been arrested more often, and if certain zip codes correlated with race due to decades of segregation, then the algorithm absorbed those correlations as if they were neutral facts about risk. They were not neutral facts. They were the echo of old injustices, converted into a number.

You can now see something most people miss when they hear "an algorithm decided": algorithms do not appear out of nowhere. They are built by people, trained on data collected by people, in a world shaped by human decisions — including unfair ones. Saying "the algorithm decided" does not remove the human responsibility. It hides it.

Imagine you wanted to build a tool to predict who would do well in a job, and you trained it on 10 years of your company's hiring data. Sounds reasonable — you're learning from experience. But what if, for those 10 years, your company almost never hired women for senior roles? Your AI would learn: senior-role traits correlate with being male. It would not have been taught that. It would have inferred it from the pattern. This is exactly what happened at Amazon between roughly 2014 and 2017, when engineers discovered their internally built recruiting AI was penalizing resumes that showed female applicants.

This is called historical bias, and it is one of the most common ways AI systems absorb unfairness. The algorithm is doing something technically correct — finding patterns in data — but the data itself reflects a world where certain groups were treated unequally. The AI learns that inequality and then reproduces it, faster and at larger scale than any individual human biases could.

The deeper problem: you cannot just "remove race" from the data and fix it. Many variables that seem neutral — zip code, school attended, whether family members have criminal records — are so closely correlated with race in the United States that they function as proxies. Remove race, and those proxies carry the same information. Researchers call this the proxy problem, and it is genuinely hard to solve.

Here is the part that stops a lot of smart people cold. When ProPublica accused COMPAS of being unfair, Northpointe did not just say "you're wrong." They said "we're using a different definition of fairness — and by our definition, we're right." And mathematically, they were.

ProPublica was measuring error rate balance — whether the tool made mistakes at equal rates for Black and white defendants. It did not. Black defendants were labeled high-risk when they were actually low-risk at nearly double the rate of white defendants.

Northpointe was measuring calibration — whether, among everyone labeled high-risk, the same percentage actually reoffended regardless of race. By this standard, the tool was fair: if you scored 7 out of 10, your likelihood of reoffending was roughly the same whether you were Black or white.

This is the ethical question you should sit with: If a society cannot agree on a single definition of fairness, can an algorithm ever be fair? Or does building any scoring system require making a political choice — and then hiding it inside math? There is no clean answer. The people currently deploying these systems are making that choice right now, and most of them are not announcing it.

Knowing what you now know changes how you read every headline that says "an AI system was found to be biased." Because you understand it is never one thing. It could be biased training data that encoded historical inequality. It could be a proxy variable carrying information that should have been excluded. It could be a deliberate choice of one fairness definition over another — made quietly, without public debate. Or it could be all three at once.

In 2022, the U.S. Department of Housing and Urban Development filed a complaint against a Facebook advertising algorithm that was automatically targeting housing ads by race, even though advertisers had not asked it to. The algorithm was optimizing for "engagement" — showing ads to people likely to click. But because housing segregation meant different groups used Facebook differently, the optimization reproduced segregation. Nobody at Facebook explicitly programmed housing discrimination. The system learned it.

The COMPAS story is not an old story. As of 2024, algorithmic risk assessment tools are used in some form in nearly every U.S. state's criminal justice system. Millions of sentencing, parole, and bail decisions have incorporated scores like these. The debate ProPublica started has not been resolved. Courts have repeatedly declined to require that the underlying code be disclosed to defendants — meaning people have been jailed partly based on a secret algorithm they cannot challenge.

Researchers who study algorithmic fairness have mapped out the pipeline through which an AI system is built. Bias can enter at multiple stages — not just one. Understanding this matters because it determines where you look when something goes wrong, and who is responsible for fixing it.

1. Data collection. If the data collected does not equally represent all groups, the system will know some groups better than others. Google Photos' training data underrepresented dark-skinned faces. Medical AI systems trained mostly on data from white patients have been shown to perform worse on patients of other ethnicities. What gets measured, and who gets measured, shapes everything downstream.

2. Labeling. Most AI systems that classify things — images, text, risk categories — require humans to label training examples first. If those human labelers carry their own biases, those biases go directly into the training data. A 2020 study of a widely-used dataset for training sentiment analysis AI found that comments written in African American Vernacular English were labeled as "toxic" by annotators at significantly higher rates than equivalent statements written in standard American English, even when the meaning was the same.

3. Feature selection. "Features" are the variables an AI looks at when making a decision. Choosing which features to include is a human judgment call. If you include zip code, you may be encoding race. If you include prior arrests, you may be encoding policing patterns rather than actual behavior. Each choice embeds assumptions.

4. Optimization target. What are you asking the AI to maximize? Amazon's hiring AI was asked to maximize "similarity to previous successful hires." That sounds neutral — but if previous successful hires were mostly men, "most like them" becomes a proxy for "probably male." The goal itself encodes bias.

5. Feedback loops. This is the most insidious one. Once a system is deployed, its decisions often become part of the next round of training data. If a predictive policing tool sends more officers to neighborhood A, more crimes will be reported in neighborhood A — not because more crimes are happening there, but because there are more officers to observe and report them. The algorithm then "learns" that neighborhood A is high-crime and sends even more officers. The model's own decisions corrupt the data it learns from.

In December 2020, a study in the New England Journal of Medicine documented something that had quietly been harming patients for decades. Pulse oximeters — the small clips hospitals put on your finger to measure blood oxygen — were significantly less accurate for patients with darker skin. They were overestimating oxygen levels in Black patients. Doctors, trusting the reading, did not administer supplemental oxygen when they should have. During the COVID-19 pandemic, when blood oxygen was a critical indicator of who needed hospital care, this inaccuracy may have contributed to worse outcomes for Black patients.

The oximeters had been developed and calibrated primarily using data from light-skinned patients. Nobody designing the device in the 1970s and 1980s set out to create a tool that would underserve Black patients. But the data used to build and validate it did not represent them — and the problem went undetected for 40 years because the people most harmed were not the people running the validation studies.

For the ages-13-and-up question: the FDA cleared pulse oximeters without requiring race-stratified accuracy data. In 2022, the FDA issued a safety alert about the problem and promised new guidance. As of 2024, the regulatory framework for evaluating AI medical tools still does not uniformly require disaggregated performance data — meaning a tool can be approved as accurate "on average" even if it is substantially worse for some groups. That policy gap exists right now. People working in public health and regulation are actively debating how to close it.

The natural instinct when you hear "the training data didn't include enough Black faces" is: then add more Black faces. And sometimes that works. Google could have trained its image recognition on a more representative dataset. But more data is not always the solution, and understanding why matters.

Consider a hiring algorithm trained on data from a company that has never promoted women to senior roles. You could add more data — but if you add data from the same company over more years, you're adding more of the same pattern. You need fundamentally different data: a different definition of what "successful hire" should mean. The problem is not quantity. It is the definition embedded in the data.

Or consider the feedback loop problem. If you are training a crime prediction model and you collect more data by sending more police to high-surveillance neighborhoods, you are not getting a more complete picture of crime — you are getting a more complete picture of policing. More data collected the same way, using the same unequal systems, only entrenches the original problem faster.

One response to the five-stage bias problem is auditing — deliberately testing AI systems for differential performance across groups before and after deployment. Joy Buolamwini's 2018 Gender Shades study, which found facial analysis AI failing most on dark-skinned women, is one of the most influential examples. She did not hack any systems. She assembled a dataset of faces that was actually balanced by skin tone and gender, ran it through commercial APIs, and published the results. Within months, IBM, Microsoft, and Face++ had all significantly improved their systems on dark-skinned faces.

External pressure from a rigorous audit forced change that voluntary internal review had not. This is one of the central policy debates in AI right now: should AI audits be mandatory? Who should conduct them — the companies themselves, independent researchers, government agencies? And what should happen when an audit finds a problem — who is liable?

You now have the vocabulary to follow that debate as a participant, not a bystander. You know what an audit is testing for. You know why "it works on average" is not a sufficient answer. You know that the five stages of bias mean a company saying "our training data is representative" is only answering one-fifth of the question.

Lesson 1 introduced two competing definitions of fairness — calibration and error rate balance — and noted that mathematically, you often can't have both. But the conflict goes deeper than math. Different definitions of fairness reflect different political values about what justice requires.

Consider three people arguing about how a college admissions AI should work:

Person A says: "The AI should be fair if it selects the students most likely to succeed academically, regardless of background." This is called individual fairness — treat each person based on their own qualifications.

Person B says: "The AI should be fair if it produces an admitted class whose demographic makeup reflects the applicant pool." This is demographic parity — the outcomes should be proportionally equal across groups.

Person C says: "The AI should be fair if it corrects for the fact that students from underfunded schools had fewer resources, so lower test scores mean different things for them." This is equity — adjusting for historical disadvantage rather than just measuring current outcomes.

All three people have defensible positions. All three definitions produce different outcomes when applied to the same data. The choice between them is not a technical decision. It is a moral and political decision that has been delegated to engineers — usually without anyone announcing that the delegation happened.

In most AI deployments, the people who choose which fairness definition to implement are product managers and engineers at the company building the tool. They are not elected. They are not legally required to disclose their choices. The people most affected by the tool — job applicants, defendants, loan applicants, students — have no seat at the table and are often not informed that a tool is being used at all.

A 2019 study by AI researchers Timnit Gebru and Emily Bender — then both at Google — helped create what became known as a "model card": a documentation framework where AI developers would publicly state what their model was designed to optimize, what data it was trained on, and how it performed across different demographic groups. The idea was that decisions currently hidden inside technical documentation would become publicly visible and contestable.

Google fired Gebru in December 2020 after a dispute over a research paper she co-authored — a paper that, among other things, raised concerns about the risks of large language models. The circumstances of her firing became one of the most high-profile controversies in AI ethics. It illustrated, concretely, that the people raising fairness concerns inside companies face real professional consequences for doing so.

This is a live policy question, not a historical one. Legislation like the Algorithmic Accountability Act has been proposed in the U.S. Congress multiple times. It has not passed. The debate is about who holds power over systems that make decisions about people — and whether the people being decided about have any rights in that process.

The HireVue case is useful because it isolates a specific argument: what counts as valid evidence that an AI system is fair? HireVue claimed its tool predicted job performance. Researchers argued the claim was not backed by rigorous independent evidence. But even if it were backed by evidence — even if analyzing facial micro-expressions genuinely predicted some measure of job performance — there is a second question: should it?

Imagine an AI that correctly identifies that people with certain speech patterns are slightly less likely to be promoted in a given company. Should that company use that AI to screen out candidates with those patterns? The tool might be accurate by its own metrics. But it would be using historical promotion patterns — which may themselves reflect bias against certain accents or communication styles — as the definition of "success." Accurate tools can still be unfair if the thing they're accurately predicting is itself the product of discrimination.

You are growing up into a world where algorithmic systems will make consequential decisions about you: which college applications get reviewed, which health insurance plan is offered to you, which loan terms you receive, how much you pay for car insurance (which in some states is partially determined by algorithms that use zip code, education level, and occupation as proxies — all of which correlate with race and income). Knowing what you now know, you can see those decisions differently.

You can ask: what fairness definition does this system use? Was I told that an algorithm was involved? Was the system validated on people like me? Who chose the optimization target? Is there a feedback loop that compounds initial disadvantages?

These are not questions most adults ask. They are not questions most journalists ask. The people who do ask them — researchers like Joy Buolamwini, Timnit Gebru, and Safiya Umoja Noble, whose 2018 book Algorithms of Oppression documented how Google search results systematically degraded the image of Black women — often face resistance, marginalization, and in some cases job loss for doing so.

Knowing what you know changes what you owe it to yourself to ask about every system that affects your life.

Researchers have developed real techniques for building AI systems that are more equitable. These are not hypothetical — they are used in production systems today, with varying levels of success. Understanding them matters because you will encounter claims like "we've debiased our algorithm" — and now you'll know what that might and might not mean.

Pre-processing: Modifying the training data before the AI learns from it. This might mean resampling — collecting more data from underrepresented groups — or re-weighting — giving examples from minority groups more influence in training. It can also mean removing features that are proxies for protected characteristics. The limitation: as Lesson 2 showed, proxy removal is hard. Remove zip code and you still have school district. Remove school district and you still have income.

In-processing: Adding a fairness constraint directly into what the AI is trying to optimize. Instead of just "minimize errors," you optimize "minimize errors and ensure error rates are within 5% across racial groups." The limitation: this is where the Kleinberg impossibility theorem bites. You can add one fairness constraint, but adding it may violate another. You are choosing, not solving.

Post-processing: After the AI produces its scores, adjust the threshold for different groups so outcomes become more equal. For example: require a lower score to qualify as "low risk" for a group that has historically been over-scored. The limitation: this often requires using group membership explicitly, which can raise its own legal and ethical questions — and opponents argue it trades one form of unfairness for another.

Some researchers argue that technical debiasing is a distraction — that it makes systems seem more legitimate without fixing the underlying problem. The argument goes: if you live in a society with structural inequality, any tool that makes decisions using that society's data will reproduce that inequality, no matter how carefully you tune the algorithm. The fix has to be social, not technical.

This view was articulated clearly by Ruha Benjamin, a Princeton sociologist whose 2019 book Race After Technology coined the term "the New Jim Code" — the idea that algorithms can replicate and reinforce racial hierarchy while wearing the legitimacy of neutral technical language. Benjamin's argument is not that algorithms are irredeemable but that technical fixes are insufficient without simultaneous changes to the social conditions that produced the biased data in the first place.

The structural responses that have been proposed include: mandatory algorithmic impact assessments before deployment (similar to environmental impact statements); ongoing public auditing of deployed systems; strong rights to explanation and appeal for people affected by automated decisions; legal liability for companies whose systems produce discriminatory outcomes; and community involvement in the design of systems that will affect those communities.

As of 2024, none of these exist as universal requirements in the U.S. The European Union's AI Act, which became law in 2024, is the most comprehensive attempt globally to create a regulatory framework for high-risk AI systems — requiring transparency, human oversight, and bias testing for AI used in employment, credit, education, and criminal justice. It is the most significant policy development in AI fairness to date, and it applies to companies operating in Europe regardless of where they are based.

Return to the Apple Card. Goldman Sachs did not use gender as a feature. Regulators found no illegal discrimination. And yet the disparity was documented and real. This is sometimes called disparate impact — when a neutral-seeming rule produces unequal outcomes across groups.

U.S. law recognizes disparate impact as a form of discrimination in some contexts — particularly employment and housing — even without proof of intent. But the legal standard is contested, the threshold for what counts as actionable disparity is debated, and the burdens of proof are high. For credit decisions, the legal picture is less clear. The Apple Card case ended with a review of affected accounts and no penalty — which means the people who received systematically lower credit limits received nothing specific in return.

You have now worked through the four lessons of this module. You know what algorithmic bias is and where it enters AI systems. You know that "fairness" is not a single technical property but a family of conflicting definitions, each encoding a different political value. You know that technical tools for reducing bias are real and limited. You know that structural responses exist but are largely not yet required by law.

Most people who use AI-powered services every day — most adults, most journalists, most lawmakers — do not know these things. They hear "the algorithm decided" and treat it as the end of the inquiry. You know it is the beginning of the right questions.

The people building these systems are not necessarily malicious. Many of them are working hard on exactly these problems. But they are working inside institutions with competing incentives, deploying systems at scales that make careful review difficult, and operating within legal frameworks that are still catching up to the technology. The gap between where things are and where they need to be is real — and it is being debated, contested, and slowly changed by researchers, advocates, regulators, and journalists who ask the same questions you now know to ask.

Knowing what you know, you are now part of the population that can read a headline about an AI system and understand what questions are not being asked. That is not a small thing.