Module 5 · Lesson 1

When the Algorithm Calls You a Criminal

AI systems have wrongly accused real people of crimes. What happened — and who was responsible?

If a machine makes a mistake that ruins someone's life, is that still just a "glitch"?

On a cold January morning in 2020, Robert Williams was standing in his driveway in Detroit when two police cars pulled up. Officers handcuffed him in front of his wife and two young daughters and drove him to a detention facility, where he spent the night in a cell. He had no idea what he was accused of.

The charge was felony armed robbery — a crime that carries years in prison. Williams had never been to the store that was robbed. He had no connection to the incident whatsoever. But a facial recognition system used by the Detroit Police Department had matched his driver's license photo to grainy security footage, and a human analyst had confirmed the match without investigating further.

Williams was released after 30 hours. He was the first known American to be wrongfully arrested based entirely on a facial recognition error. He was not the last. Similar cases followed in New Jersey in 2021 (Michael Oliver, charged with a crime he didn't commit) and in New Orleans in 2023 (Randal Reid, jailed for six days for a theft that occurred in a state he had never visited).

In every case, the process was roughly the same: an AI system flagged a face, a human rubber-stamped the result, and a real person's life was disrupted — sometimes severely — before anyone asked whether the machine had simply been wrong.

Why Facial Recognition Gets It Wrong — And Who It Gets Wrong More

Facial recognition AI works by measuring features of your face — the distance between your eyes, the shape of your jaw — and comparing them against a database. It doesn't "see" the way you do. It sees patterns of pixel data and calculates a probability score: this face is X% likely to match that face.

The problem is that these systems were trained mostly on data that skewed heavily toward lighter-skinned faces, especially male ones. A landmark 2018 study by MIT researcher Joy Buolamwini found that some commercial facial recognition systems had error rates as low as 0.8% on lighter-skinned men — but as high as 34.7% on darker-skinned women. That's not a small rounding error. That's one in three faces misidentified.

This means the people most likely to be wrongly accused by facial recognition AI are those who were already underrepresented in the training data. The system didn't "decide" to be biased in any human sense — it learned from biased data, and the bias became baked in.

Training data biasWhen the examples used to teach an AI system don't represent all people or situations equally, the system performs worse on groups that were underrepresented in its training.

The Human in the Loop — and the Problem with That Phrase

After Robert Williams was released, Detroit Police admitted that the facial recognition match should never have led to an arrest on its own. Their official policy said AI results were meant to be "investigative leads" — starting points, not conclusions. A human analyst was supposed to verify the result before anyone got arrested.

But here's what actually happened: the human analyst looked at the AI's match, agreed with it, and moved on. The phrase "a human was in the loop" technically described the situation. But in practice, the human deferred to the machine. That's a pattern researchers call automation bias — the tendency to trust automated systems more than our own judgment, especially when we're tired, pressured, or when the machine seems confident.

This creates a real problem. When companies or governments deploy AI in high-stakes situations — criminal justice, hiring, medical diagnosis — they often point to the human in the loop as the safety net. But if that human consistently defers to the AI, the safety net isn't really there. The accountability disappears into a gap between "the machine suggested it" and "a person confirmed it."

Automation biasThe human tendency to over-trust automated systems and accept their outputs without enough critical review, especially under pressure or time constraints.

Ethical Question

If a police analyst confirms an AI match that turns out to be wrong — and someone is wrongly arrested — who is responsible? The company that built the AI? The analyst who agreed with it? The police department that deployed it? The city that permitted it? There is no clean answer here. That's exactly the problem.

What "Standing Up" Actually Looks Like

Robert Williams didn't just quietly accept what happened to him. With support from the American Civil Liberties Union (ACLU), he sued the City of Detroit in 2021. His case — and the cases that followed — pressured the city to update its facial recognition policies and sparked legislation in multiple states restricting how law enforcement can use the technology.

Joy Buolamwini didn't just publish her research and move on. She founded the Algorithmic Justice League in 2016, a nonprofit that challenges bias in AI systems through research, art, and advocacy. She testified before the U.S. Senate. Her work directly influenced several companies to change their facial recognition products.

Neither of these people had a magic "fix AI" button. What they had was the ability to name what was wrong, document it, and bring it into spaces where decisions get made. That is what standing up looks like in the age of AI — not fighting the machine, but forcing the humans behind it to be accountable.

You Now See What Most People Miss

Most people who read a headline about a "facial recognition error" assume it was a rare malfunction — a one-time glitch. You now know it's a structural problem rooted in biased training data, compounded by automation bias, and made worse by unclear accountability. That's a completely different situation. And knowing the difference changes how you evaluate every AI deployment you'll encounter.

Lesson 1 Quiz

When the Algorithm Calls You a Criminal · 5 questions

1. Robert Williams was wrongly arrested in Detroit in 2020. What was the primary cause of his wrongful arrest?

Correct. The AI provided a match; the analyst confirmed it without verifying. This combination — algorithmic error plus automation bias — led to the arrest.

Not quite. Review the story of Robert Williams. The key was a facial recognition system that produced a false match, and a human who trusted it without checking.

2. Joy Buolamwini's 2018 MIT study found that some facial recognition systems had error rates as high as 34.7% on which group?

Correct. The disparity was stark: near-perfect accuracy on lighter-skinned men, but more than one in three darker-skinned women misidentified.

Not quite. Buolamwini's research found the worst error rates on darker-skinned women — a direct result of underrepresentation in training data.

3. A hospital deploys an AI to help diagnose rare diseases. Doctors are told to review all AI recommendations before acting. Studies later show doctors almost always follow the AI's suggestion without independent analysis. What concept best describes this pattern?

Correct. Automation bias is exactly this: humans who are nominally "in the loop" but in practice defer to the machine rather than exercising independent judgment.

Not quite. This scenario describes automation bias — the pattern of over-trusting automated systems even when independent review is required.

4. Why does training data bias cause AI to perform worse on certain groups of people?

Correct. AI learns from examples. If certain groups are underrepresented in those examples, the AI simply hasn't learned enough about them to be accurate — no malicious intent required for serious harm to occur.

Not quite. The AI isn't following rules about who to disadvantage. It learns patterns from data, and sparse data for certain groups means poor pattern learning for those groups.

5. Which of the following best describes how Joy Buolamwini and Robert Williams "stood up" against AI errors?

Correct. Standing up against AI errors doesn't mean defeating technology — it means naming harms precisely, documenting them, and engaging the humans and institutions who have the power to change things.

Not quite. The lesson's key point is that standing up means naming the problem, documenting it, and forcing accountability through legal, policy, and public channels.

Lab 1: The Accountability Gap

Role: Civil Rights Investigator · AI peer: Case Analyst

Your Scenario

A city is considering expanding its facial recognition program from law enforcement to school security — scanning students' faces at building entrances. You've been asked to assess the risks before the city council votes. Your AI partner has been briefed on the technology and the documented cases of error.

This isn't a research exercise — you need to take a position and defend it. Your AI partner will push back on weak arguments.

Start by telling the analyst: what's your biggest concern about deploying facial recognition in schools — and why does that concern outweigh the security benefits?

Case Analyst

AI Peer

I've read the Williams, Oliver, and Reid cases. I've also read the city's proposal — they're claiming the new system has a 95% accuracy rate, which they say makes it safe enough for schools. Before you make your recommendation, I want to hear your actual argument. What's the biggest problem with putting this in schools, and can you defend that position against someone who says the safety benefits justify the risk?

Module 5 · Lesson 2

The Résumé That AI Deleted

Amazon built an AI to find great job candidates. It quietly taught itself to reject women. Here's how that happened — and what it means for you.

If a biased decision is made by an algorithm instead of a human, is that still discrimination?

Starting in 2014, Amazon built a machine learning tool designed to automate the hiring process. Engineers fed it ten years of résumés that had been submitted to Amazon — and the job titles those applicants ended up with. The idea was elegant: teach the machine what a successful Amazon employee looks like, then use it to sort future applications.

By 2015, the team noticed something alarming. The system was systematically downgrading résumés that contained the word "women's" — as in, "women's chess club" or "women's college." It was also penalizing graduates of all-women's colleges. The system had noticed that most successful Amazon engineers over the past decade were men, and had learned — on its own, without being told to — to treat maleness as a positive signal and femaleness as a negative one.

Engineers tried to fix it. They removed the specific phrases the model had flagged. But they couldn't guarantee the model hadn't found other proxies for gender — writing style, extracurricular activities, anything that might correlate with being a woman. By 2018, Amazon shut the project down entirely. They later confirmed the tool had never been used to make actual hiring decisions. But the story became one of the most-cited examples in AI ethics because of what it reveals: a system can learn to discriminate without anyone programming it to.

How Discrimination Gets Hidden Inside "Neutral" Math

The Amazon engineers didn't build a sexist AI. They built a system that learned from historical data — and that data reflected a decade of hiring decisions made in an industry where women were significantly underrepresented in technical roles. The AI treated "what worked before" as "what should work now." It replicated the past.

This is sometimes called historical bias — when the unfairness of past decisions gets encoded into the patterns an AI learns, and then projected forward. The math is entirely neutral. The outcome is not.

What makes this especially difficult to challenge is that the discrimination is invisible by design. When a human hiring manager discriminates, they leave a paper trail — emails, interview notes, someone who might testify. When an algorithm discriminates, it produces a score. The score looks objective. Most of the time, the rejected candidate never even knows they were screened out by a machine, let alone why.

Historical biasWhen an AI system learns patterns from past decisions that were themselves unfair or unequal, and then reproduces those patterns in its future outputs — treating past inequality as a template.

Proxy discriminationWhen an AI system uses an apparently neutral variable (like zip code, name, or writing style) that happens to strongly correlate with a protected characteristic (like race or gender), producing discriminatory outcomes without explicitly using that characteristic.

The Hiring Algorithm Isn't Just Amazon's Problem

Amazon is not alone. A 2019 investigation by ProPublica and The Markup documented AI screening tools being used by hundreds of major employers. Studies showed that AI résumé screeners commonly favored applicants with names that sounded white and male, even when all other qualifications were identical. The 2021 HireVue facial analysis controversy revealed that some AI hiring tools were analyzing candidates' facial expressions during video interviews and generating personality scores — with no publicly available evidence that those scores predicted job performance.

This matters at an institutional level: by 2022, an estimated 99% of Fortune 500 companies were using some form of automated screening in their hiring process. The decision about whether you even get a human to look at your application is increasingly made by a system you never see and can't appeal to.

In 2023, the U.S. Equal Employment Opportunity Commission (EEOC) issued guidance confirming that employment discrimination through AI tools is still illegal under existing civil rights law — meaning companies can be sued even if "the algorithm did it." The European Union's AI Act, passed in 2024, classifies AI hiring tools as "high-risk systems" requiring mandatory transparency and human oversight. These are real policy changes, driven directly by documented cases of AI bias.

Ethical Question

If a company uses an AI hiring tool that turns out to discriminate, but the company genuinely didn't know it would — and the tool was built by a third-party vendor — who bears responsibility? The company that used it? The vendor that built it? The engineers who trained it on biased data? And what about the candidates who were rejected and never found out?

How You Audit What You Can't Fully See

You can't demand access to a company's proprietary hiring algorithm. But researchers, journalists, and advocates have developed tools for pressuring AI systems into revealing their biases — even without the source code.

One method is audit testing — sending identical applications with different names or demographic signals and comparing outcomes. This is the same method civil rights organizations used to document housing discrimination in the 1960s. It still works. In 2021, researchers at MIT used audit testing to show that AI-powered loan pricing tools charged Black applicants higher rates than white applicants with identical credit profiles.

Another tool is public disclosure: in 2021, New York City passed Local Law 144, which requires employers using AI hiring tools to commission annual independent bias audits and publish the results. It was the first law of its kind in the United States. It exists because advocates documented the problem and showed up at city council hearings.

You can't fix what's invisible. But once you know where to look, you can force the light on.

You Now See What Most People Miss

Most people think discrimination requires a discriminatory intent — someone who wants to be unfair. What you now know is that AI systems can produce discriminatory outcomes through pure pattern-matching on historical data, with no one intending any harm. That distinction matters enormously in law, in policy, and in how you think about fairness in any system that uses AI to make decisions about people.

Lesson 2 Quiz

The Résumé That AI Deleted · 5 questions

1. Amazon's AI hiring tool learned to discriminate against women even though no one programmed it to. What best explains how that happened?

Correct. The AI learned from historical data reflecting past inequality, not from explicit instructions. This is historical bias: the system replicated the patterns it found, including patterns of underrepresentation.

Not quite. No one programmed the bias deliberately. The system learned from a decade of hiring data that underrepresented women, and treated that pattern as a signal of quality.

2. A credit scoring AI doesn't ask for applicants' race. But it heavily weights zip code — and certain zip codes are historically associated with specific racial demographics due to decades of housing discrimination. What concept describes this situation?

Correct. Proxy discrimination occurs when a neutral-seeming variable (zip code) correlates strongly with a protected characteristic (race), producing discriminatory outcomes without explicitly using that characteristic.

Not quite. When an AI uses a variable that seems neutral but strongly correlates with race, gender, or another protected group, that's proxy discrimination — and it's still illegal discrimination under U.S. law.

3. Why is discrimination by AI hiring tools especially difficult for job applicants to challenge compared to discrimination by human hiring managers?

Correct. The invisibility of algorithmic screening is a core problem. If you don't know a machine rejected your application — or what signals it used — you have nothing concrete to challenge.

Not quite. The core challenge is invisibility. Rejected candidates often have no idea they were screened by an algorithm, what it evaluated, or why it scored them as it did.

4. New York City's Local Law 144 requires employers using AI hiring tools to publish annual independent bias audits. What broader strategy does this represent?

Correct. Transparency requirements are a way of making visible what would otherwise remain hidden, enabling accountability without prohibiting the technology outright.

Not quite. The law doesn't ban AI hiring tools — it makes them visible by requiring published audits, which creates accountability without eliminating the technology.

5. Researchers sent identical job applications with different names (some sounding stereotypically white, some stereotypically Black) to 100 employers using AI screening tools. They found significantly different callback rates. What research method is this?

Correct. Audit testing — sending matched pairs with different demographic signals — allows researchers to detect discriminatory patterns from the outside, without needing access to the system's code.

Not quite. This method is called audit testing: using controlled experiments with matched pairs to detect discrimination without needing to see inside the algorithm.

Lab 2: The Bias Auditor

Role: Algorithmic Bias Investigator · AI peer: Research Partner

Your Scenario

A local university is using an AI tool to screen applications for its summer scholarship program. Preliminary data suggests students from certain zip codes are being accepted at far lower rates — but the university says the AI only considers grades and test scores, not location. You need to figure out whether that claim holds up and what evidence you'd need to make a case.

Tell your research partner: given that you can't see the algorithm's source code, what's the first step in determining whether this AI is producing biased outcomes — and what specific data would you request from the university?

Research Partner

AI Peer

The university's spokesperson says the AI is "completely objective" because it only uses grades and test scores. But we've got preliminary numbers showing students from lower-income zip codes have a 23% acceptance rate versus 61% for students from wealthier zip codes — even when their grades are comparable. I want to hear your methodology. What's your first investigative move, and what data are you going to demand? Don't just say "audit testing" — tell me specifically what you'd test and why.

Module 5 · Lesson 3

When AI Invents a Story About You

AI language models can generate completely false information about real people — and present it with total confidence. What happens when that person is you?

If an AI states something false about you to millions of people, do you have the right to make it stop — and can you?

In February 2023, Mark Walters, a radio host in Georgia, discovered that the AI chatbot ChatGPT had described him — in a legal summary generated for a journalist — as having embezzled funds from a Second Amendment organization where he served as treasurer. There was one problem: none of it was true. Walters had never been accused of embezzlement. The organization mentioned didn't match any he was associated with. The case number in the summary was fake. The court where the supposed lawsuit was filed didn't exist.

ChatGPT had invented the entire thing. Not maliciously — the model doesn't have intentions — but confidently, fluently, and in exactly the format of a real legal document. Walters sued OpenAI in June 2023 in what became one of the first defamation lawsuits against an AI company in the United States.

His case was not unique. A law professor named Jonathan Turley found that ChatGPT had generated a fake sexual harassment allegation against him, complete with a fabricated Washington Post article as a source. Brian Hood, the mayor of Hepburn Shire in Australia, threatened to sue OpenAI after ChatGPT falsely described him as a convicted criminal — he had actually been a whistleblower, not a criminal. These events all occurred within a six-month window in early 2023 as large language models were first deployed at scale.

What Hallucination Actually Means — and Why It's the Wrong Word

The AI industry calls this problem hallucination — when a language model generates text that sounds accurate but is factually wrong. The word is everywhere. It's also, in a subtle way, misleading.

"Hallucination" makes it sound like the AI is confused, or like it briefly lost its grip on reality — the way a person might after not sleeping for 40 hours. But that framing implies a baseline state of being "grounded in reality" that language models don't have. A large language model doesn't know facts. It generates text that statistically resembles text it was trained on. When it produces something factually accurate, it's because accuracy correlates with how humans write. When it produces something false, it's because the false statement also fits the statistical patterns of the training data.

The more accurate framing is confabulation — generating plausible-sounding content to fill a gap, without any mechanism for checking whether it's true. It's not a bug in an otherwise reliable system. It's a structural feature of how these systems work.

Hallucination / ConfabulationWhen an AI language model generates text that is false but sounds plausible — not because the model is malfunctioning, but because it produces statistically likely text patterns rather than verified facts. "Confabulation" is the more precise term: filling in gaps with invented but convincing content.

The Legal and Personal Stakes of AI Defamation

Defamation — the legal term for a false statement of fact that damages someone's reputation — has existed as long as publishing has. What changes with AI is scale and searchability. A false statement in a small-town newspaper might reach a few thousand readers. A false statement in a widely-used AI chatbot gets reproduced every time someone asks the right question — potentially millions of times, in different conversations, with no single place to issue a correction.

The legal situation is unsettled. U.S. defamation law requires showing that the defendant acted with at least negligence — and it's still unclear how intent-based defamation frameworks apply to systems that have no intent at all. The Walters lawsuit was dismissed in 2024 on technical grounds, but similar cases continue to work through courts in the U.S., UK, and EU. Meanwhile, the EU's AI Liability Directive, proposed in 2022 and still advancing in 2024, would make it easier for people harmed by AI to access evidence and pursue legal claims.

Practically, victims of AI defamation currently have limited tools. They can contact the AI company directly — OpenAI and Google have processes for reporting false information, though correction is not guaranteed. They can publish corrections and document the false content carefully. And they can bring legal action, though courts are still figuring out the rules.

Ethical Question

AI companies argue they can't manually review everything their models generate — it's simply too much content. But if a company deploys a system that regularly generates false statements about real, named people, and those false statements are presented without any disclaimer, is the company ethically responsible for the harm? Does it matter if they "didn't know" any specific statement was false?

What You Can Do When AI Lies About Someone

If you encounter an AI generating false information about a person — yourself or someone else — the most important thing is to document it precisely. Screenshot the output. Record the exact prompt you used. Note the date, the platform, and the model version if available. This creates a record that doesn't depend on the AI maintaining that specific output (since model outputs can change over time).

Most major AI platforms have a feedback or reporting mechanism. These reports do affect model behavior over time — not for that individual conversation, but as part of the ongoing process of refining outputs. Reporting is not futile; it is a form of participation in how these systems evolve.

Beyond individual action: organizations like the Electronic Frontier Foundation (EFF), the Center for Democracy and Technology (CDT), and the AI Now Institute specifically track and respond to cases of AI harm to individuals. Connecting documented cases to organizations that can amplify them is one of the most effective ways to convert individual harm into systemic pressure.

You Now See What Most People Miss

When someone tells you an AI is "just making stuff up" about a person, they often mean it as a minor complaint about a quirky technology. You now understand why it's a serious issue with real legal, reputational, and safety stakes — and why the word "hallucination" obscures more than it explains. You also know what to do about it. Most people don't.

Lesson 3 Quiz

When AI Invents a Story About You · 5 questions

1. In February 2023, ChatGPT generated a false legal summary about radio host Mark Walters that accused him of embezzlement. What was true about this output?

Correct. Every element of the AI's output was invented: the accusation, the legal case, the court, and the organization — all fabricated with complete fluency and false confidence.

Not quite. In Walters' case, nothing in the AI's output was real. The accusation, the case number, the court, and the organization were all invented by the model.

2. Why is "confabulation" a more precise term than "hallucination" for describing AI-generated false information?

Correct. "Hallucination" implies a temporary glitch in an otherwise reality-grounded system. "Confabulation" better describes what's actually happening: generating statistically plausible text without any mechanism for verifying truth.

Not quite. The distinction is about what the word implies about the system. "Hallucination" implies a baseline of groundedness that language models don't have. "Confabulation" better describes the structural process of generating plausible-sounding filler without truth-checking.

3. You find that a popular AI chatbot is generating false information about a local teacher, claiming she was disciplined for cheating on exams. She was never disciplined for anything. What is the most useful first step?

Correct. Documentation is the foundation of any subsequent action — reporting to the platform, connecting with advocacy organizations, or supporting legal action all require a clear, timestamped record of what the AI said and how.

Not quite. Before you can report it, challenge it legally, or connect it to advocacy organizations, you need a clear record. Documentation — screenshot, prompt, date, platform — is always step one.

4. What makes AI-generated defamation potentially more harmful at scale than traditional defamation (like a newspaper article)?

Correct. Scale and diffusion are the key dangers. A newspaper correction can address a false article. There is no equivalent mechanism for an AI system that has generated the same false claim across millions of individual conversations.

Not quite. The critical difference is scale and lack of a correction mechanism. A newspaper publishes once; an AI can reproduce a false claim in millions of separate conversations, each appearing freshly generated and credible.

5. Language models generate factually accurate text sometimes — not because they verify facts, but because accuracy correlates with patterns in their training data. What does this mean for how you should use AI-generated information about real people?

Correct. Accuracy in AI outputs is a byproduct of statistical pattern-matching, not deliberate verification. For claims about real, named people, independent verification is essential — AI citations can also be fabricated.

Not quite. AI accuracy is a side effect of patterns, not a result of fact-checking. Any specific claim about a real person needs independent verification — and note that AI-generated citations can themselves be fabricated.

Lab 3: The Defamation Desk

Role: Digital Rights Advocate · AI peer: Legal Strategist

Your Scenario

A student at your school tells you that a popular AI chatbot is describing her older sister — a local community organizer — as having a criminal record for fraud. The sister has never been charged with any crime. The student wants to know what to do. You've been brought in as a digital rights advocate to advise her.

Tell the legal strategist: what steps would you recommend to this family, in what order, and what is the realistic goal at each step? Be specific — don't just say "contact the company."

Legal Strategist

AI Peer

Before you give them a plan, I want to stress-test your thinking. The family is understandably upset and wants this fixed immediately. But "contact the company" is not a plan — it's a hope. What specific steps would you actually walk them through, and what's a realistic outcome at each stage? Also: at what point, if ever, would you recommend involving lawyers or advocacy organizations — and why that point specifically?

Module 5 · Lesson 4

The Student Who Pushed Back

You don't have to be a researcher, lawyer, or senator to challenge AI systems. Here's what effective pushback actually looks like at your level — and why it matters.

What does it actually mean to hold a powerful system accountable when you're 12 years old with no lawyers and no platform?

In 2023, a group of high school students at Lowell High School in San Francisco noticed something troubling in how an AI tutoring tool recommended college prep resources. Students from lower-income families who used the tool were being steered toward community college pathways and vocational programs, while students with similar grades but markers of higher income were being recommended four-year university preparation resources. No one had programmed this difference in. The AI had inferred economic status from patterns in how students wrote their prompts and what devices they were using.

The students didn't accept it quietly. They documented specific cases — anonymized but detailed — and brought them to their school district's technology committee. They wrote a one-page report explaining what they had observed and asked three specific questions: Who selected this tool? What bias testing was done before deployment? Who is responsible if the tool disadvantages students?

The district paused use of the specific tool for review. The vendor was asked to provide its bias audit documentation. The students had not written a single line of code. They had not gone to court. They had done something simpler and more powerful: they had named the problem precisely, put it in front of the people with authority over it, and demanded a specific response.

The Anatomy of Effective Pushback

What made the Lowell students effective wasn't anger or virality — it was precision. Vague complaints about AI ("this tool is unfair") get no traction. Specific, documented observations ("this tool recommended different outcomes for students with comparable grades, and here are four examples") are much harder to dismiss.

Effective pushback against AI errors tends to have these components:

1. Specific documentation. Not "the AI said something wrong" but "on this date, using this prompt, this platform generated this specific output." Screenshots, quotes, timestamps.

2. A named harm. Not "this might be a problem" but "this caused or could cause X specific consequence to Y specific type of person." Harms that can be described concretely are much easier to respond to than harms that remain abstract.

3. A clear audience. Who has the power to act on this? A school district technology committee. A company's trust and safety team. A journalist covering AI policy. A city council member proposing AI legislation. Pushback aimed at no one in particular usually stays that way.

4. A specific ask. Not "do something about this" but "pause this deployment pending review" or "release your bias audit results" or "add a disclaimer to outputs about real people." Specific asks create accountability — a person either did the thing or they didn't.

What Changes When Young People Push Back

There is a specific credibility that comes with being a user of a tool who was directly affected by it — and there is a specific credibility that comes with being young in a conversation about technology that will shape your entire adult life. Neither of those is a small thing.

In 2022, a 16-year-old student in the UK named Shreya Agarwal became one of the named complainants in a legal challenge to an AI grading system used by the UK government during the COVID-19 pandemic. That system — called the A-level algorithm — had assigned grades to students based partly on their schools' historical performance, disadvantaging students at lower-performing schools who were individually capable of exceeding their school's average. After widespread protest by students and parents, the UK government reversed the grades of approximately 280,000 students. An AI grading decision affecting hundreds of thousands of young people was reversed, in part because students said clearly and publicly: this is wrong, this is why, and this is what we need changed.

In 2021, the Dutch government's child welfare fraud detection algorithm — known as SyRI — was ruled illegal after legal challenges partly driven by affected families who documented its discriminatory targeting of lower-income and immigrant households. The system had flagged families for potential benefits fraud based on patterns that disproportionately affected certain ethnicities. The Dutch court ruled it violated human rights law. That ruling set a precedent that influenced AI regulation debates across Europe.

Ethical Question

When you push back against a biased AI system and it gets changed or removed, the new system might have different flaws you don't know about yet. Does that make pushback pointless — or does getting the process right matter even if the outcome is imperfect? At what point is "this AI causes less harm than before" good enough?

The Skills You Already Have

This module has been about specific documented cases where AI systems caused real harm, and specific real people who did something about it. None of them had magical credentials. What they had was:

The ability to notice that something was wrong. The discipline to document it precisely rather than just getting angry. The clarity to name the harm in terms others could understand. The knowledge of who to take it to. And the persistence to follow through.

Those are not technical skills. They are not adult-only skills. You have been building them throughout this course — in every lesson where you analyzed a case, in every lab where you were asked to take a position and defend it, in every quiz where you had to apply a concept to a new situation rather than just remember a definition.

The AI systems being built and deployed right now will be part of your world for decades. The decisions being made in 2024 and 2025 about how they work, who they serve, and who holds them accountable — those decisions are still being contested. That means there is still time to influence them. The question is whether people who understand the stakes show up.

Now you understand the stakes.

You Now See What Most People Miss

Most adults assume that challenging AI systems requires technical expertise, legal resources, or institutional power. You now know that the most effective challenges have consistently come from people who could clearly describe a specific harm, name who was responsible for it, and make a specific ask. You are capable of doing all three. That is not a small thing.

Lesson 4 Quiz

The Student Who Pushed Back · 5 questions

1. The Lowell High School students who challenged their AI tutoring tool were effective primarily because they:

Correct. Their effectiveness came from precision, not power: specific documentation, named harm, correct audience (the district technology committee), and specific demands (pause, audit, accountability).

Not quite. The students had no technical or legal resources. What they had was precision: specific documentation, named harm, the right audience, and a specific ask. That combination is what got the tool paused for review.

2. In 2020, the UK government's A-level algorithm was reversed, restoring grades for approximately 280,000 students. What was the core problem with that algorithm?

Correct. By anchoring grades to each school's historical performance average, the system penalized individual students who could have outperformed their school's average — a structural bias that hurt students from disadvantaged schools most.

Not quite. The algorithm used each school's historical grade distribution as a benchmark. Individual students who might have exceeded their school's average were capped by it — a form of historical bias applied at the institutional level.

3. Which of these is the weakest form of pushback against a biased AI system and why?

Correct. Vague, undocumented complaints — even viral ones — rarely create institutional accountability. Specific, documented pushback aimed at people with authority over the system is consistently more effective.

Not quite. The weakest form is a vague complaint with no specific documentation, no named harm, and no clear ask — directed at no one in particular. All the other options target people with actual authority over the system and provide specific evidence.

4. A student notices that an AI college essay helper at her school gives much more detailed and useful feedback to essays written in formal academic English than to essays written in informal registers. She wants to push back effectively. What is her best next step?

Correct. This applies the anatomy of effective pushback: specific documentation (matched examples), named harm (unequal feedback quality by writing style), correct audience (counseling department with authority over the tool), and an implied ask (review or adjust the tool).

Not quite. The most effective move combines specific documentation with the right audience and a specific ask. Boycotts and social media posts are weaker without evidence; teaching workarounds accepts the bias rather than challenging it.

5. The Netherlands' SyRI algorithm was ruled illegal after legal challenges by affected families. What broader lesson does this carry for AI governance?

Correct. The SyRI case shows that affected communities challenging AI systems through legal channels can achieve not just local wins but precedent-setting rulings that influence policy across entire regions — in this case, contributing to EU AI regulation debates.

Not quite. The SyRI case demonstrates the full arc of effective pushback: affected families documented specific harms, brought legal challenges, won a court ruling, and set precedent that shaped European AI policy discussions — all without having any technical expertise.

Lab 4: The Accountability Report

Role: Student Advocate · AI peer: Policy Advisor

Your Scenario

Your school district is considering purchasing an AI-powered student behavior monitoring system. The system would analyze students' writing, browsing patterns on school devices, and even keystrokes to flag potential mental health risks or disciplinary issues — and would send alerts to teachers and counselors without students' knowledge.

You've been asked to present a one-minute statement to the school board before the vote. Your AI partner is a policy advisor who will help you sharpen your argument — but they'll also push back hard if your reasoning has gaps.

Draft your opening argument: what is the single most important concern about this system, and why does it outweigh whatever benefits the vendor is claiming? Make it specific enough that a school board member who hasn't read any AI ethics research would understand it immediately.

Policy Advisor

AI Peer

I've seen the vendor's pitch deck. They're claiming a 91% accuracy rate on identifying students at mental health risk, and they have testimonials from three districts saying it helped identify students before crises occurred. That's not nothing. So before you make your case to the board, I want to hear your strongest specific objection — not "privacy is important" but a concrete argument about what specifically goes wrong with this system and who gets hurt. Make me believe you've thought about the vendor's best arguments, not just your own side.

Module 5 Test

Standing Up When AI Gets It Wrong · 15 questions · Pass at 80%

1. Robert Williams was wrongfully arrested in Detroit in 2020 because a facial recognition AI produced a false match. What compounded the error into an actual arrest?

Correct. The policy required human review, but the analyst deferred to the AI — a textbook case of automation bias converting a machine error into a wrongful arrest.

Not quite. The police did have a human review policy. The problem was automation bias: the analyst accepted the AI's result instead of independently verifying it.

2. Joy Buolamwini's research found that some facial recognition systems had much higher error rates on darker-skinned women than on lighter-skinned men. What is the most accurate explanation for this disparity?

Correct. Training data bias — underrepresentation of certain groups in the data used to teach the system — produces worse learned patterns for those groups, with no discriminatory intent required.

Not quite. This is training data bias: the system learned from data that underrepresented darker-skinned women, so it had fewer examples from which to build accurate recognition patterns for that group.

3. Amazon's AI hiring tool began penalizing résumés containing the word "women's" — for example, "women's chess club" — without being programmed to do so. What type of bias does this represent?

Correct. Historical bias occurs when AI learns from data reflecting past inequalities and reproduces those inequalities in future outputs. The tool learned that maleness correlated with success in its training data — and treated that correlation as a rule.

Not quite. This is historical bias: the AI was trained on hiring data from a period and industry where women were underrepresented, so it learned to treat markers associated with women as negative signals.

4. An AI loan approval system doesn't ask for applicants' race. But researchers find it charges higher rates in zip codes with higher percentages of Black residents. What concept best describes this?

Correct. Proxy discrimination: a neutral-seeming variable that correlates with a protected characteristic (due to historical factors like residential segregation) produces discriminatory outcomes without explicitly using that characteristic.

Not quite. When a variable like zip code correlates strongly with race due to historical housing discrimination, using it in a way that produces racially disparate outcomes is proxy discrimination — still illegal under civil rights law.

5. New York City's Local Law 144 requires employers using AI hiring tools to publish annual independent bias audits. What problem is this law designed to solve?

Correct. The law doesn't ban AI hiring tools — it makes them visible. Published bias audits create accountability that didn't exist when algorithmic decisions were entirely private.

Not quite. The law's goal is transparency: forcing published audits means that algorithmic discrimination can be seen, studied, and challenged — addressing the core problem that AI bias was previously invisible to those affected by it.

6. ChatGPT generated a false legal summary accusing Mark Walters of embezzlement, with a fabricated court, fabricated case number, and fabricated organization. Why did the AI produce such a specific and detailed false output?

Correct. Confabulation: the model generates text that fits the statistical patterns of its training data. Legal summaries have particular structures; the model filled in the structure with plausible-sounding details that happened to be entirely false.

Not quite. Language models don't retrieve facts — they generate text that fits patterns. A convincing-sounding legal summary is exactly the kind of output the model's training would lead it to produce, regardless of whether the specific content is true.

7. Why is "confabulation" considered a more precise term than "hallucination" for AI-generated false information?

Correct. "Hallucination" implies a system that is normally grounded in reality having a temporary lapse. Language models don't have a baseline of verified truth to depart from — confabulation (plausible gap-filling without truth-checking) better describes the structural situation.

Not quite. The key distinction is what each word implies about the system's baseline. Hallucination implies temporary departure from normal accuracy. Confabulation describes a structural process — generating plausible content without any truth-verification mechanism — which is accurate for language models.

8. Someone tells you an AI chatbot is producing false criminal accusations about their neighbor. What should be the very first step?

Correct. Documentation is always the foundation. Every subsequent step — reporting to the platform, contacting advocacy organizations, supporting legal action — requires a precise, timestamped record of what was said and how it was generated.

Not quite. Legal action and public posting both require evidence. Without precise documentation — screenshot, prompt, date, platform — you have nothing concrete to work with. Documentation is always step one.

9. The UK A-level algorithm in 2020 disadvantaged students at lower-performing schools even when individual students were capable of exceeding their school's historical average. What type of bias does this illustrate?

Correct. The algorithm treated historical institutional performance as a predictive template for individual students — a form of historical bias that penalized students for circumstances outside their control.

Not quite. The A-level algorithm used each school's historical grade distribution to cap individual predictions, encoding the inequality of past school performance into individual students' futures — that's historical bias applied at the institutional level.

10. The Lowell High School students who challenged their AI tutoring tool successfully got it paused for review. They had no technical expertise and no legal resources. What was the key to their effectiveness?

Correct. Their effectiveness came from the anatomy of good pushback: specific documentation, named harm, correct audience, specific asks. No technical skills required.

Not quite. The students' power came from precision: documented cases, named harm, the right audience with actual authority over the decision, and specific questions that demanded specific answers.

11. A human being is described as being "in the loop" when an AI makes a recommendation and a human must confirm before action is taken. What makes this protection less effective than it sounds?

Correct. Automation bias is the specific mechanism that hollows out human-in-the-loop protections: when humans consistently defer to AI rather than genuinely reviewing it, the accountability gap between "AI suggested" and "human confirmed" disappears.

Not quite. The danger is automation bias: humans required to review AI recommendations often accept them with minimal scrutiny, especially under time pressure — meaning "a human confirmed it" doesn't actually mean a human independently evaluated it.

12. Researchers send matched résumés — identical qualifications but different names that signal different ethnicities — to 200 employers using AI screening tools and find significantly different callback rates. This research method is called:

Correct. Audit testing uses controlled experiments with matched pairs to reveal discriminatory patterns from the outside — no source code access required. The same method was used to document housing discrimination in the 1960s.

Not quite. Sending matched applications with different demographic signals and comparing outcomes is audit testing — a method with roots in civil rights research that predates AI by decades.

13. The Netherlands' SyRI algorithm flagged households for benefits fraud and was later ruled illegal by a Dutch court. What made the families' challenge effective, and what broader impact did it have?

Correct. The SyRI case shows the full arc: documented specific harm, legal challenge, court ruling, broader policy impact. Affected communities without technical expertise changed national and regional AI governance.

Not quite. The families documented the harm, pursued legal channels, won a human rights ruling, and set precedent that shaped EU AI regulation debates — all without technical expertise, through the power of documented, legally-framed pushback.

14. You want to challenge an AI system that you believe is producing biased outcomes in your school. Rank these actions from most to least effective: (A) vague social media post, (B) specific documented report to the principal with examples and questions, (C) connecting documented cases to a digital rights organization, (D) asking the AI itself to explain its bias.

Correct. Documented report to the right authority (B) is most directly actionable; connecting to advocacy organizations (C) amplifies the documented case; vague social media (A) raises awareness but lacks specificity; asking the AI to explain itself (D) is least useful since AI systems do not have reliable self-knowledge of their biases.

Not quite. Specific documented reports to authorities with actual power over the system (B) are most directly effective. Connecting cases to advocacy groups (C) amplifies documented evidence. Vague social posts (A) lack the specificity needed for accountability. Asking the AI about its own bias (D) produces unreliable outputs — models don't have accurate self-knowledge of their biases.

15. Across all four lessons in this module, what is the single most consistent pattern in effective challenges to harmful AI systems?

Correct. From Joy Buolamwini's research to Robert Williams' lawsuit to the Lowell students to the Dutch families: the pattern is consistent. Precision, documentation, correct audience, specific asks. No technical skills required.

Not quite. The consistent pattern across all cases — Buolamwini, Williams, the Lowell students, the Dutch families, the UK A-level students — is precision and specificity: documented harm, named cause, correct audience, specific demand. Technical or institutional resources help, but they are not the determining factor.