When an algorithm is built, a group of people sits in a room — or on a video call, or scattered across a campus — and makes decisions. What should this system do? What counts as success? What data should we train it on? What should we test before we release it? Every one of those decisions is made by a human being who has a background, a set of assumptions, and things they never thought to question.

In 2016, a researcher named Joy Buolamwini — then a graduate student at MIT — was working on a project that used facial recognition software. The software couldn't detect her face. It worked fine when she put on a white mask. She dug into why, and what she found launched a research career and eventually changed U.S. law. The systems she tested — from companies like IBM, Microsoft, and Amazon — had been trained almost entirely on faces of light-skinned men. When she published her findings in 2018 in a paper called "Gender Shades," the error rates told a stark story: for light-skinned men, the systems were wrong about 1% of the time. For darker-skinned women, they were wrong up to 35% of the time.

That gap didn't come from nowhere. It came from a decision — conscious or not — about whose faces mattered enough to include in the training data. And that decision came from a room where most of the people looked roughly the same.

It's tempting to think of AI failures as purely technical — bugs, bad math, a corrupted dataset. But most of the important failures are design decisions. They happen when the people building a system define "success" in a way that works for most users but quietly fails a few.

In 2018, the news organization Reuters reported that Amazon had been quietly developing an AI hiring tool since 2014 — a system designed to scan resumes and score job candidates. The goal was to speed up hiring by automatically rating applicants on a scale of one to five stars. By 2015, Amazon's own engineers noticed a serious problem: the system was systematically downgrading resumes that included the word "women's" — as in "women's chess club" or "women's leadership program." It also penalized graduates of two all-women's colleges.

Why? Because the system had been trained on ten years of Amazon's own hiring data. And over those ten years, Amazon — like most tech companies — had hired mostly men. The AI wasn't trying to discriminate. It was doing exactly what it was designed to do: identify patterns in historical success and replicate them. The problem was that the historical pattern included bias. The AI amplified it. Amazon scrapped the tool in 2017 without ever using it for official decisions — but not before quietly running it in parallel for years.

Now here is the part that most people skip over. Amazon's engineers noticed the problem. They flagged it. And the company still let the system run — in parallel, quietly — for two years before retiring it. The failure wasn't purely technical. It was also a decision made by people about what mattered enough to stop.

Here is a pattern worth memorizing: when a technology is built primarily by people from one demographic, the default use cases tend to serve that demographic first. This isn't always malicious. Often it's just the result of building what you know, testing on people you have access to, and shipping when it works for most of your team.

Voice recognition is a good example. In 2017, a Stanford researcher named Rachael Tatman published a study showing that Google's voice recognition system was significantly more accurate for men than for women, and significantly more accurate for people without regional accents than for people with them. Scottish English was recognized far worse than American English. African American English was recognized far worse than white American English.

The gap mattered in the real world. By 2019, voice recognition was being used in courtrooms to transcribe testimony, in call centers to route complaints, in schools as an accessibility tool for students with disabilities. A system that works 95% of the time for some users and 78% of the time for others sounds like a small difference — until the 17% gap is the part where someone's words get lost.

Understanding this changes how you read every product announcement, every "revolutionary AI" headline, every "this system is 98% accurate" claim. You now know to ask: accurate for whom? Tested on which users? Built by which team? Designed to optimize for what goal? Most people never think to ask those questions. You can.

In 2019, the city of San Francisco became the first city in the United States to ban government use of facial recognition technology. The argument was simple: the technology was too error-prone for high-stakes decisions, particularly for Black and brown residents. Other cities followed — Oakland, Boston, Portland.

But here's the complication. Some police departments argued that facial recognition, even imperfect, helped solve cold cases — including murders and child abductions — that would otherwise go unsolved. In 2020, facial recognition was used to help identify a man who had been abducting children in New York. He was caught. The victims were found. The tool worked, in that case, for those people, on that day.

So the question isn't whether the technology works. The question is: who gets to decide when a tool is accurate enough to use on people who never consented to be scanned? Who absorbs the cost when it gets it wrong? And is it acceptable to use a tool that works better for some people than others in decisions where being wrong can mean someone loses their job, their freedom, or their life?

There isn't a clean answer to that. Serious, thoughtful people land in different places. What matters is that you now understand the structure of the problem well enough to recognize when someone is oversimplifying it — either by pretending the technology is neutral, or by pretending it has no legitimate use at all.

To understand why data is never neutral, you first have to understand where it comes from. Large language models — the kind that power ChatGPT, Google's Gemini, and similar tools — are trained on enormous collections of text scraped from the internet. One commonly used dataset is called Common Crawl, which contains a snapshot of billions of web pages going back to 2008. Another is called Books3, which contained digitized versions of over 196,000 books before legal challenges forced changes to it in 2023.

The internet sounds like "everything." But it isn't. As of 2023, only about 43% of the world's population uses the internet in any regular way. Among that 43%, about 56% of all web content is in English — a language spoken natively by roughly 5% of the world. The vast majority of Wikipedia articles, forum posts, reviews, and news articles that end up in training data were written by people who are English-speaking, educated, and connected enough to have internet access and time to write online.

This means that when an AI system trained on this data gives you an answer, it is heavily shaped by the worldview, assumptions, and blind spots of a specific slice of humanity. It doesn't know it's doing this. It has no way to flag when it's applying an English-speaking American lens to a question that has a completely different answer in a different cultural context. It just responds, confidently.

Not all AI training data comes directly from the internet. A lot of it is labeled — meaning a human being looked at a piece of content and attached a tag to it. "This photo contains a dog." "This text is positive." "This news article is reliable." "This face is expressing anger." Those labels become the ground truth the AI learns from. The AI will eventually make the same kinds of judgments millions of times per day, on everything from moderation decisions to medical diagnoses.

In 2019, a research team at the AI Now Institute — a research organization based at New York University — published a report examining how labels get applied in commercial AI datasets. One of their findings concerned emotional recognition systems: tools trained to identify emotions from facial expressions. These systems were being sold to companies for use in job interviews — scanning applicants' faces and rating their emotional suitability. The problem was that the labels defining what "happiness" or "trustworthiness" looks like on a face had been applied by data annotators, many of them working in crowdsourced tasks for low pay, drawing on cultural frameworks that vary enormously across the world.

In East Asian cultural contexts, for example, strong direct eye contact in a job interview can signal aggression rather than confidence. A system labeled primarily by Western annotators would score that cultural difference as a "red flag." The AI wouldn't know it was enforcing one culture's norms. It would just produce a score. The score would look authoritative. The applicant would never know why they were rejected.

There is one more layer to this that changes how you read almost everything about AI. When we train AI systems on historical records, we are training them on a history that contains the full weight of past inequality. This isn't a glitch or an edge case. It is a structural feature of learning from the past.

In 2016, ProPublica — an investigative journalism organization — published a groundbreaking analysis of a tool called COMPAS (Correctional Offender Management Profiling for Alternative Sanctions). COMPAS was being used in courtrooms across the United States to predict the likelihood that a defendant would re-offend — a score called a "recidivism risk score." Judges were using this score to inform decisions about bail, sentencing, and parole.

ProPublica's analysis found that COMPAS was twice as likely to falsely flag Black defendants as "high risk" when they did not actually go on to reoffend — and twice as likely to falsely label white defendants as "low risk" when they did reoffend. The company that made COMPAS, Northpointe, disputed the methodology. Academics wrote papers arguing about which statistical measure of "fairness" should apply. Judges kept using the tool.

The data COMPAS learned from was criminal justice records — records shaped by decades of unequal policing, unequal prosecution, and unequal sentencing. An AI trained on those records learned those patterns as if they were natural law. Then it applied them in individual courtrooms, one person at a time, under the cover of an authoritative-looking number.

You now understand why some researchers say that "training on historical data" is sometimes just another way of saying "automating historical prejudice." That framing is deliberately provocative — but it's precise. And knowing it means you will never hear "data-driven decision" the same way again.

Here is the complication. The COMPAS case — and cases like it — prompted some researchers to call for banning algorithmic risk scores in criminal justice entirely. Pure human judgment, they argued, is fairer than an opaque algorithm.

But research on human judicial decision-making tells a different story. Multiple studies — including a widely-cited 2011 paper in the Proceedings of the National Academy of Sciences — found that judges gave significantly harsher rulings right before lunch than right after it. Other studies found that judges with daughters gave more lenient sentences in certain cases than judges with only sons. Human judges are inconsistent, fatigued, and shaped by their own experiences in ways they can't fully control.

So we are left with a real dilemma: an algorithm that inherits and amplifies historical bias, versus human judgment that is inconsistent, emotional, and difficult to appeal. Neither is clean. Anyone who tells you the answer is obvious — in either direction — hasn't fully understood the question. You have.

Every AI system makes errors. The engineers who build them know this. When you hear a system described as "96% accurate," what that means is: it is wrong 4% of the time. For some applications, 4% errors are acceptable. If Netflix recommends a movie you don't like, the error cost is low — you watch something else. If a system is making decisions about your employment, your loan application, your medical care, or your freedom, a 4% error rate means something entirely different.

In Robert Williams' case, a facial recognition system made an error, a human detective failed to independently verify it, and a warrant was signed. At least three people made decisions along that chain — and every one of them could have caught the error. The algorithm is one part of a system that includes human choices, institutional practices, and assumptions about when a machine output is trustworthy enough to act on without verification.

This is called the human-in-the-loop problem: when AI systems are deployed at scale, humans theoretically remain in the decision-making chain, but in practice the speed, volume, and apparent authority of AI outputs causes people to defer to them rather than scrutinize them. The machine says it, so it must be checked out. This is especially true when the humans in the loop are overworked, under-resourced, or operating under institutional pressure to clear cases quickly.

Robert Williams is Black. All three of the individuals wrongfully arrested in documented U.S. facial recognition cases as of 2023 are Black men. This is not a coincidence. It is the direct consequence of the accuracy gap documented by Joy Buolamwini's research — the same gap described in Lesson 1. When a system has higher error rates for darker-skinned faces, and that system is used in contexts involving criminal investigation, the errors concentrate among people who are already disproportionately scrutinized by law enforcement.

This is the pattern that makes some AI failures genuinely different from ordinary technology failures. A bug in a navigation app might add ten minutes to everyone's commute. An accuracy gap in a facial recognition system used in policing concentrates its worst outcomes on communities that already face elevated risk. The word researchers use for this is disparate impact — when a policy or tool that doesn't explicitly target a group produces systematically worse outcomes for that group.

In 2021, the National Institute of Standards and Technology (NIST) — a U.S. government agency — published a massive study of 189 facial recognition algorithms from vendors around the world. Their finding: false positive rates for African American and Asian faces were often 10 to 100 times higher than for Caucasian faces in one-to-many searches. NIST is not a civil rights organization. It is a technical standards body. The accuracy gaps it documented are engineering facts, not political opinions.

Wrongful arrests are visible. They produce lawsuits, news coverage, congressional testimony. But most AI errors are invisible — not because they don't cause harm, but because the people harmed don't know why a decision was made or have no mechanism to challenge it.

In 2018, the Dutch government deployed a fraud detection algorithm called SyRI (System Risk Indication) to identify residents likely to commit welfare fraud. The system analyzed data on income, employment, housing, debts, and tax records, and flagged individuals for investigation. A Dutch court ruled in 2020 that SyRI violated European human rights law because it was being deployed almost exclusively in lower-income neighborhoods, disproportionately targeting people already facing economic precarity. The people flagged had no way to know they were being flagged, no way to see what data led to the flag, and no practical path to appeal.

This is the invisible harm: a government algorithm making decisions about your life, based on data you never consented to share, using criteria you can't see, producing a result you'll never be told about — until a government official shows up at your door. The Dutch SyRI case was one of the first successful legal challenges to an automated government profiling system in Europe. It set a precedent that directly influenced the European Union's AI Act, passed in 2024.

You now know something that shapes every conversation about AI governance: the visibility of harm depends on whether the harmed person has the resources, knowledge, and access to legal mechanisms to make the harm visible. Most people don't. That asymmetry is not an accident of the technology — it's a consequence of how power works, and AI systems can amplify it.

Robert Williams' case led some advocates to demand that no government agency ever use facial recognition for identification in criminal investigations. Others — including some civil liberties advocates — argued for regulation rather than prohibition: require verification steps, require disclosure, require external auditing. Others pointed out that without any technological assistance, investigators rely on eyewitness identification, which research consistently shows is even less reliable than most facial recognition systems for many kinds of cases.

Here is the question you should sit with: if a technology causes demonstrable harm to specific communities, but removing it would also cause demonstrable harm — including to members of those same communities — whose voice should be most important in deciding what to do? The people whose communities are most affected? The engineers who understand the technical limits? The elected officials who set policy? The courts that weigh rights? There is no clean hierarchy here. But the fact that we're even asking the question represents progress. Most societies never asked it about technologies that affected far more people.

Every time an AI system makes a decision that affects someone's life, there are five questions that, if answered honestly, reveal whether the system is operating appropriately. You can apply these to any AI story you read, any system that affects you directly, or any proposal a government or school or company puts in front of you.

1. What is it actually optimizing for? AI systems are designed to maximize something — a score, an accuracy rate, an efficiency metric, a profit measure. The thing it optimizes for is not always the thing it's described as doing. A college admissions algorithm described as "identifying talented students" might be optimizing for tuition revenue, retention rates, or alumni donation patterns. The optimization target shapes everything.

2. Who was in the room when the goals were defined? The lesson you've just completed covers this in depth. Who defines success? Whose needs were considered in the design phase? Which communities were consulted, and which weren't? These questions often reveal more about likely failure modes than any technical audit.

3. What data was it trained on, and what does that data leave out? Training data is never a complete picture of reality. It is a snapshot of what was recorded, by whom, in which contexts. Understanding the gaps in the data reveals what the system will fail to handle — and who will absorb those failures.

4. Where do the errors go? Average accuracy hides error distribution. Ask specifically: who is in the tail of the error distribution? Which groups are more likely to be incorrectly flagged, denied, or misclassified? Is the error rate consistent across demographic groups?

5. What is the recourse path? When the system gets it wrong — and it will — can the affected person find out? Can they appeal? To whom? At what cost? Systems with no real recourse path are not accountable systems, regardless of what their marketing materials say.

AI generates an enormous amount of media coverage, and most of it falls into predictable patterns of either uncritical enthusiasm or unfocused alarm. Learning to read past both of those patterns is one of the most practical skills this course can give you.

In January 2023, Google announced a new AI medical imaging system called MAIA that could detect tuberculosis in chest X-rays with accuracy exceeding many radiologists in one specific study. Headlines read: "Google AI Surpasses Doctors." The study was real. But the study also involved a specific patient population in South Africa, a specific type of X-ray equipment, and a specific definition of "accuracy" that measured sensitivity (catching real cases) rather than specificity (avoiding false positives). A radiologist reading the same headline would immediately ask: how does it perform on patients with other concurrent conditions? What are its false positive rates? Was it tested on the equipment actually available in low-resource clinics?

These aren't nitpicky questions. They are the difference between a research result and a deployable medical tool. The headline collapses that difference. Most readers don't have the framework to rebuild it. You now do.

Here are the specific warning phrases that should trigger your five questions:

"Our system is X% accurate" — accurate on what population, by what measure, compared to what baseline?

"AI outperforms humans" — in what specific task, under what conditions, measured how, with what error distribution?

"Objective, data-driven decision-making" — what data, collected by whom, with what gaps, optimizing for what goal?

"We take privacy seriously" — what specific data is collected, retained for how long, shared with whom, with what user controls?

The word "accountability" appears constantly in discussions of AI policy. It is used so broadly that it has almost stopped meaning anything. Here is what it actually requires in practice, based on the cases covered in this course:

Transparency means more than publishing a press release. It means making the optimization targets, training data composition, performance metrics by demographic group, and error rates accessible to people outside the company — including people who might be harmed. In 2022, the state of Illinois passed the Artificial Intelligence Video Interview Act, which requires employers who use AI to screen video interviews to tell candidates that AI is being used and to explain what it assesses. That is a meaningful transparency requirement. A company blog post describing its AI as "fair and robust" is not.

Contestability means that affected individuals can actually challenge a decision. Not theoretically challenge it by filing a lawsuit they can't afford — actually challenge it through a real process with real consequences. The Dutch court that ruled against SyRI established that affected people have a legal right to understand why an automated system flagged them. That is contestability. A form email saying "this decision was made by our system" is not.

External audit means that people with no financial interest in the system can examine it, test it, and publish findings. Joy Buolamwini's "Gender Shades" research was an external audit. ProPublica's COMPAS analysis was an external audit. Both produced findings that companies disputed — but the findings entered the public record and could not be erased. Systems that resist external audit are systems whose operators know they cannot survive examination.

In 2022, the city of New York began requiring employers with 15 or more workers to use "automated employment decision tools" only if those tools had undergone a "bias audit" — and to tell candidates when such a tool is being used. It was the most comprehensive local AI employment regulation in the United States at the time.

Critics pointed out that the auditing requirements were written broadly enough that companies could conduct their own audits or pay audit firms they selected — raising obvious conflicts of interest. Advocates said that even an imperfect audit requirement was better than none, because it created a legal hook for challenging systems that produced discriminatory outcomes. Both sides were right. The New York law was a genuine step forward and a genuine compromise that left significant gaps.

This is what progress in AI governance looks like up close: imperfect, contested, partial. A regulation that falls short of what researchers want but moves the baseline forward. A ruling that sets precedent but doesn't solve everything. A company policy change that addresses the visible problem while leaving the structural one intact.

The question for you is not: "is this enough?" Nothing will ever be enough quickly enough. The question is: do you understand the difference between a system that is genuinely more accountable and one that is performing accountability while changing nothing? Because if you do, you can participate in the conversations that shape these decisions — and you can recognize when someone is trying to convince you that window dressing is reform. That is a skill that will matter every year of your life from here forward.