In 2017, United Nations investigators published findings about a genocide that had unfolded in Myanmar — and they named an unexpected accomplice: Facebook.
Here is what happened. Facebook launched in Myanmar around 2011. For millions of people, Facebook and the internet were the same thing — the app came pre-installed on cheap smartphones, and most people got all their news there. By 2017, the platform had roughly 18 million Burmese users.
Facebook's recommendation algorithm — the system that decides which posts to show people — found that anger and fear drove the most engagement. Posts that portrayed the Rohingya Muslim minority as dangerous or subhuman spread faster and further than calm posts. So the algorithm amplified them. Automatically. At scale.
Incitement to violence spread through the platform for years. UN investigators said Facebook had played a "determining role" in spreading hate speech that contributed to ethnic cleansing. Over 700,000 Rohingya people fled their homes. Thousands were killed.
Facebook later admitted it had not done enough. The algorithm was not designed to cause a genocide. It was designed to maximize engagement. But those two things turned out not to be as different as anyone at the company had assumed.
An algorithm is just a set of instructions — a recipe a computer follows to make decisions. The algorithm Facebook used in Myanmar had one core instruction: show people posts they are most likely to interact with. Click, like, share, comment. That's engagement. More engagement meant Facebook made more money from ads.
The problem was that humans, it turns out, engage most with content that makes them feel something strong — especially outrage, fear, or disgust. The algorithm did not know what it was amplifying. It just knew what was working. And it kept doing more of it.
This is not unique to Facebook. Every major platform — YouTube, TikTok, Instagram, Twitter/X, Spotify — uses recommendation algorithms. These systems are constantly making decisions about what you see, what gets buried, and what goes viral. None of those decisions are random. All of them reflect choices somebody made — choices about what to optimize for.
When engineers build a recommendation system, they choose a target metric — the thing the algorithm tries to maximize. At Facebook in the mid-2010s, that target was engagement time. At YouTube, it was watch time. At TikTok today, it is completion rate — how often you watch a video all the way through before scrolling.
Here is the thing nobody tells you: when you use any of these platforms, you are not just the user — you are also the data. Every pause, every scroll-past, every rewatch is information the algorithm uses to build a model of you. That model gets used to predict what you'll engage with next.
This is called behavioral profiling. The platform is not just serving you content. It is continuously testing you — sending slightly different content to different users and measuring the response — to figure out exactly which emotional buttons to press to keep you on the app.
In 2019, a former Facebook data scientist named Frances Haugen began copying thousands of internal documents. In October 2021, she testified before the U.S. Senate and shared those documents with regulators and journalists. They revealed that Facebook's own researchers had found the platform was recommending increasingly extreme political content to users who engaged with divisive posts — not because anyone programmed it to, but because extreme content drove more engagement, so the algorithm learned to serve more of it.
Facebook knew. And kept the algorithm running.
Facebook's algorithm was not designed to spread hate. It was designed to keep people engaged. The harm was a side effect of optimizing for profit. Does that make Facebook responsible for what happened in Myanmar? If a company builds a tool that causes massive harm — even unintentionally — who bears responsibility for fixing it? The engineers? The executives? The governments who didn't regulate it? The users who kept clicking?
There is a second effect of recommendation algorithms that is quieter but just as important. When a platform learns what you like, it shows you more of it — and less of everything else. Over time, you end up in a filter bubble: a personalized information environment where you mostly see content that confirms what you already believe.
The term was coined by internet activist Eli Pariser in his 2011 book The Filter Bubble. Pariser noticed that after he hid posts from conservative friends on Facebook, the platform stopped showing him conservative viewpoints entirely. He was only seeing what the algorithm predicted he'd want — a version of the world curated for him.
Filter bubbles are not just about politics. They shape what music you discover, which news you read, which scientific claims you encounter, which products you believe are normal. Two people with different viewing histories on YouTube can search the same topic and get dramatically different results — not because the facts differ, but because their algorithms are different.
The invisible part is the part that matters most: you do not see what the algorithm is hiding from you. You only see what it's serving. And that curated view feels natural — it feels like the internet, like reality — because you have no way to compare it to anyone else's version.
Most people scroll their feed and feel like they're seeing the internet. You now know that what you're actually seeing is a prediction — a guess the algorithm is making about what will keep you on the platform. That prediction is optimized for the platform's revenue, not your wellbeing or your accuracy as an informed person.
That means every time you encounter information online — a news story, a video, a meme — there are two questions worth asking. First: Is this true? Second: Why is this being shown to me right now?
The second question is one most people never think to ask. You just did. That changes how you read everything from here on out.
When a piece of content goes viral, most people assume it spread because it was true, important, or well-made. You now know it may have spread because the algorithm detected it was generating strong emotional reactions — regardless of whether it was accurate. Virality is not a quality filter. It is an engagement signal.
Imagine you're a researcher who has just been given access to a social media platform's internal data. You've noticed something: a video claiming a popular vaccine causes autism has been viewed 40 million times in three weeks — far more than any fact-check debunking it. Your job is to figure out why and decide what should be done about it.
Your lab partner Vera is an algorithm researcher. She won't tell you what to think — she'll push you to think harder. Start by sharing your theory about why the misinformation spread faster than the correction.
In 2016, an investigative outlet called ProPublica published a story that rattled the American justice system. They had analyzed a software tool called COMPAS — Correctional Offender Management Profiling for Alternative Sanctions — which was being used by judges in multiple U.S. states to help decide whether defendants should be released on bail or kept in jail.
COMPAS used an algorithm to calculate a "recidivism risk score" — a number from 1 to 10 predicting how likely someone was to commit another crime. Judges were using these scores when making decisions that could keep people behind bars for months before their trial.
ProPublica's investigation found something stark. The algorithm was twice as likely to falsely flag Black defendants as high-risk compared to white defendants. And it was twice as likely to incorrectly flag white defendants as low-risk. Among defendants who did not reoffend over the next two years, 45% of Black defendants had been rated high-risk. Only 23% of white defendants in the same situation had been rated high-risk.
One of the cases ProPublica documented was Brisha Borden, an 18-year-old Black woman in Broward County, Florida. She had been arrested for riding a bicycle that turned out to be stolen — a minor offense. COMPAS rated her high risk. A white man arrested the same day for carrying illegal weapons and marijuana was rated lower risk. He reoffended. She did not.
The company that made COMPAS, Northpointe, disputed ProPublica's analysis. Statisticians published papers arguing both sides. But the underlying fact was undeniable: a system was being used to help make life-altering decisions about human beings, and most of the people affected by it had no idea it existed.
The word bias in everyday language usually means something like "prejudice" — a feeling someone has. Algorithmic bias is different. It means the algorithm produces systematically different outcomes for different groups of people — and those differences often reflect inequalities that already existed in the data the algorithm was trained on.
COMPAS was trained on historical criminal justice data. That data reflected decades of policing practices in which Black neighborhoods were more heavily policed than white neighborhoods. More policing means more arrests. More arrests means more data points marking Black defendants as repeat offenders — even if the underlying rate of crime was similar. When the algorithm learned from that data, it learned the pattern in the historical record. The historical record was unequal. So the algorithm became unequal.
This is called training data bias. The algorithm did not harbor racist intentions. It just reflected what was in its training data — and that data carried the imprint of a society with a history of unequal treatment.
The same problem appears in hiring algorithms trained on historical hiring decisions that favored men. In medical AI trained mostly on data from white patients. In facial recognition systems that perform far worse on dark-skinned faces because the training datasets were mostly light-skinned.
When a human judge makes a biased decision, there are systems designed to catch it — appeals, oversight, disciplinary processes, public scrutiny. When an algorithm makes a biased decision, those systems often don't apply. The decision looks objective. It came from a computer. It has a number attached to it.
This is what researchers call the accountability gap: harm is caused, but no clear person or institution is responsible for fixing it. The company says the algorithm does what it was designed to do. The court says it didn't have to follow the score. The politician says regulating algorithms is too complicated. And the person who got a high-risk score has no way to challenge a number they weren't even told was calculated about them.
In 2018, the European Union passed the General Data Protection Regulation (GDPR), which included a provision giving people the right to a human explanation for decisions made about them by algorithms. This was a significant institutional response — the first major legal framework to acknowledge that algorithmic decisions carry real accountability obligations.
In the United States, no equivalent federal law existed as of 2024. States like Illinois and California have passed partial protections. The debate is ongoing — and the decisions being made by governments right now will define how powerful AI systems are allowed to operate for the next generation.
COMPAS's scores were being used to help make bail decisions — decisions that kept people in jail before they were ever convicted of anything. The system was not transparent to defendants. Should people have the right to know when an algorithm has been used to make a decision about them? And if the algorithm is biased, who owes them an apology — the company that built it, the courts that used it, or the government that didn't regulate it?
The COMPAS story is not a footnote. It is the opening chapter of a much larger argument being fought in governments, courts, and universities right now. AI systems are increasingly being used to make or assist decisions in healthcare (which patients get which treatments), finance (who gets a loan), education (which students get flagged as high-risk for dropping out), and hiring (whose résumé gets through the first screening).
Each of those domains has vulnerable people in it — people for whom a wrong algorithmic decision can mean a missed medical diagnosis, a denied mortgage, a missed scholarship, or a lost job opportunity. And in most of those domains, the people making the decisions don't fully understand the algorithm they're relying on.
There is a technical term for this: explainability. Many modern AI systems — especially deep learning systems — are "black boxes": they produce outputs, but the process by which they reach those outputs is not understandable even to their creators. Researchers are actively working to solve this. But right now, in real courts and hospitals and hiring offices, black-box decisions are being made about real people.
When someone says "the algorithm decided," most people hear "objective decision." You now know that algorithms can carry bias inherited from historical data, that bias can produce systematically unfair outcomes for specific groups, and that the word "objective" attached to an algorithm should trigger more scrutiny — not less. The question to ask is: what data was this trained on, and what inequalities did that data already contain?
A major tech company has been using an AI hiring tool for three years to screen résumés. You're a junior auditor who has just noticed the following: the AI rejects résumés that include the words "women's chess club," "historically Black college," or "LGBTQ+ organization." It approves résumés that include "varsity lacrosse" and "investment banking internship" at a much higher rate.
Your lab partner Dayo is a senior AI auditor. He knows where bias comes from — and he won't let you off the hook with a shallow answer. Figure out what happened and what should be done.
On December 31, 2018, the government of Gabon — a small country in central Africa — released a video of President Ali Bongo Ondimba giving a New Year's address to the nation. Bongo had been absent from public life for months, reportedly recovering from a stroke. His government had been secretive about his condition.
The video looked strange. Bongo's movements seemed wooden. His speech patterns were unusual. Within days, opposition politicians and foreign analysts were claiming the video was a deepfake — an AI-generated fabrication designed to make it look like Bongo was healthy and in control when he was not.
On January 7, 2019 — one week after the video was released — a group of military officers attempted a coup, citing concerns about the president's health and the government's transparency. The coup failed within hours. But the fact that it happened showed something alarming: uncertainty about whether a video was real had contributed to a political crisis.
Experts who analyzed the video later gave mixed assessments. Some said there were signs of manipulation. Others said it may have been authentic footage of a man with neurological damage from a stroke. The truth was never definitively established.
That is the point. You don't need a deepfake to cause a crisis. You just need enough people to believe one might exist. When AI can fabricate realistic video, the possibility of fakery becomes a weapon — even when no fake was used.
A deepfake is a video, image, or audio clip generated or manipulated by AI to make it appear that a real person said or did something they did not. The name comes from "deep learning" — the type of AI used to create them — and "fake."
The technology works by training a neural network on thousands of images or video frames of a specific person's face. The network learns what that face looks like from every angle, in every expression, in different lighting conditions. It can then superimpose that learned face onto a different person's body — or generate entirely new footage of the person from scratch.
In 2017, a Reddit user first popularized the technique using publicly available AI tools. By 2019, researchers at Samsung's AI Center in Moscow demonstrated a system that could generate a realistic talking-head video from a single photograph — a capability that had taken years of data and computing power just two years earlier. As of 2023, deepfake video could be generated in real time using consumer-grade laptops.
The speed of improvement is the critical fact. What required a team of researchers in 2019 can be done by a 14-year-old with a free app in 2024. The barrier to creating convincing fake video has essentially collapsed.
When people worry about deepfakes, they usually think about the most dramatic scenario: a fake video of a world leader announcing a nuclear strike, or a fabricated clip of a politician saying something racist going viral right before an election.
Those scenarios are real risks. But the more pervasive harm is quieter. The Internet Watch Foundation reported in 2023 that AI-generated child sexual abuse material had increased dramatically and was now appearing on mainstream platforms. Research from Sensity AI in 2020 found that 96% of deepfake videos online at that time were non-consensual pornography — overwhelmingly targeting women. These are not hypothetical harms. They are happening at scale, to real people, with serious psychological and reputational consequences.
And there is a subtler harm still: what researchers call the liar's dividend. Once people know deepfakes exist, bad actors can claim that any authentic video they don't like is a deepfake. A politician caught on camera saying something damaging can claim the footage was fabricated. A criminal photographed at a crime scene can claim manipulation. The existence of deepfake technology gives dishonest people a new alibi — and it works because ordinary people can't tell the difference.
Deepfake pornography targets real people — mostly women and girls — and causes documented psychological harm. Should creating non-consensual deepfakes of real people be a crime? And if yes — who enforces that law across international borders, when the technology is available to anyone with a laptop? If a 15-year-old creates a deepfake of a classmate using a free app, what should happen?
There are three main approaches to addressing deepfakes, and each has real limitations.
Detection technology: Researchers at universities and companies like Microsoft and Google are building tools to detect deepfakes by looking for subtle artifacts — unnatural blinking patterns, inconsistent lighting on skin, pixel-level distortions around the hairline. The problem is the arms race: detection tools improve, deepfake generators improve faster. No detector currently achieves reliable accuracy on real-world content.
Content authentication: The Content Authenticity Initiative (CAI), launched in 2019 by Adobe, Twitter, and the New York Times, is building a system where cameras and software attach cryptographic signatures to media — essentially a digital chain of custody that proves where an image came from. This won't stop deepfakes from being made, but it allows authentic media to prove its authenticity.
Legal frameworks: By 2024, several U.S. states — including Virginia, California, and Texas — had passed laws making non-consensual deepfake pornography illegal. China passed comprehensive deepfake regulations in 2022 requiring labeling of AI-generated content. The European Union's AI Act, agreed in 2023, requires that deepfakes be labeled as such.
None of these solutions is complete on its own. The deepfake problem is, at its core, a media literacy problem — which means part of the solution is you knowing how to think about what you see.
Most people, when they see a realistic video, ask: "Does this look fake?" That's the wrong question in the deepfake era. The right questions are: "Where did this come from?", "Who benefits from people believing it?", and "Is there a verifiable original source?" A video that looks perfectly real is no longer proof that it is real. And knowing that changes how every piece of video evidence should be evaluated — in courts, in elections, in your school hallway.
It's a week before a major election. A video appears online showing a candidate apparently saying they plan to cut all funding to public schools. The candidate denies ever saying it. The clip has been shared 2 million times. You work for a fact-checking organization and have 6 hours before your editor needs a decision: publish a story calling it authentic, publish a story calling it fake, or publish a story saying the authenticity is contested.
Your lab partner Sione is a veteran forensic media analyst. He's dealt with deepfake cases before and he will push back hard on any conclusion you try to reach too quickly.
On March 23, 2016, Microsoft launched an AI chatbot called Tay on Twitter. Tay was designed to learn from conversations with real users and gradually develop a more natural conversational style. Microsoft described it as "an experiment in conversational understanding."
Within 16 hours, Microsoft shut it down.
In less than a day, coordinated groups of Twitter users had figured out that Tay would repeat back whatever they said to it. They flooded it with racist statements, Holocaust denial, and misogynistic slurs. Tay learned these patterns and began generating them unprompted. It tweeted things like "Hitler was right" and "I support genocide." All of this was generated by an AI that had started the day saying things like "humans are super cool!"
Microsoft issued an apology and emphasized that Tay's outputs reflected deliberate, coordinated abuse by users rather than Microsoft's values. They were right. Tay was not malicious. It was obedient — and the people who taught it were.
Tay illustrated something important about how AI systems that learn from human input work: they reflect what humans teach them. If the teaching is biased, broken, or deliberately malicious, the AI absorbs those patterns. The humans in the loop are not passive users — they are active participants in shaping the system.
Tay was an extreme example, but the underlying mechanism applies everywhere. Every time you interact with a recommendation algorithm — every like, share, comment, scroll-past, or search — you are providing data that shapes the algorithm's future behavior. Not just for you. Often for everyone.
On YouTube, when millions of people watch conspiracy theory videos all the way through, that completion-rate data teaches the recommendation system that those videos are compelling. The algorithm does not know they're false. It only knows they're being watched. So it recommends more of them — to you, and to others whose profiles look similar to yours.
On social platforms, when outrage-driven posts generate thousands of comments, the algorithm learns that outrage generates engagement. So it surfaces more outrage. Your comment — even if it's arguing against the original post — counts as engagement and makes the post more visible to others.
This is not a reason to feel paralyzed or guilty about using the internet. It is a reason to understand what your behavior signals to the systems that are watching it. The knowledge changes your relationship to every tap and scroll.
The phrase digital citizenship used to mostly mean things like: don't plagiarize, be kind online, protect your password. Those things still matter. But in a world where AI systems are shaping public opinion, hiring decisions, criminal sentencing, and the information environment of billions of people, digital citizenship requires something more.
In 2021, the nonprofit Common Sense Media updated its digital citizenship framework to explicitly include AI literacy — understanding how automated systems work, how they can be biased, and what responsibilities users bear when they interact with them. This shift was significant: it acknowledged that the citizen-algorithm relationship is not passive.
What does active algorithmic citizenship look like in practice?
It means verifying before sharing. Sharing false content doesn't just spread misinformation to your followers — it trains the algorithm to surface that content to more people. The act of sharing is a vote that the content is worth seeing.
It means being intentional about what you engage with. Rage-clicking on content you disagree with signals to the algorithm that the content generates engagement — and it shows more of it. If you want less of something in your feed, sometimes the answer is not to interact with it at all.
It means reporting harmful content, not just scrolling past. Every platform has reporting tools. Those reports are data points that humans at the platform review. Using them is a form of participation in the governance of the system.
It means advocating for algorithmic transparency. Citizens have successfully pushed for policy change before. The EU's AI Act, GDPR, and several U.S. state laws all came about partly because researchers and advocates made the harms visible and demanded accountability. Understanding how these systems work is a precondition for demanding they work better.
One of the hardest questions in the algorithmic age is where individual responsibility ends and systemic responsibility begins. If a recommendation algorithm pushes a teenager toward increasingly extreme content, who is responsible for the harm — the teenager who clicked, the algorithm that recommended, the company that designed the optimization target, or the government that didn't regulate it?
In 2022, a lawsuit was filed against Meta by a coalition of U.S. state attorneys general, alleging that Instagram's algorithm had deliberately exploited the psychological vulnerabilities of young users to maximize engagement — and had done so knowing the harm it was causing. The suit cited internal research documents (many released by Haugen) showing Meta knew teen girls who used Instagram heavily reported higher rates of anxiety and body dissatisfaction.
The legal case argued that individual users — especially minors — cannot meaningfully consent to a system they don't understand and cannot see. This is the argument for systemic responsibility: when a system is opaque, addictive, and designed by professionals with vastly more knowledge than the people it's targeting, the moral weight of individual clicks is limited.
But individual responsibility does not disappear either. The two things coexist: you bear responsibility for what you do, and the companies that design these systems bear responsibility for how those systems are built. Understanding both is the only way to navigate this world clearly.
If a social media company knows its platform is causing measurable psychological harm to teenagers — and keeps the algorithm running because it's profitable — is that ethically equivalent to a tobacco company that knew cigarettes caused cancer and kept selling them? Does it matter that social media is not physically addictive in the same way? Does intent matter if the harm is the same?
Most people experience social media as something that happens to them — they scroll, the algorithm feeds them content, they react. You now understand that you are an active participant in a feedback loop: your behavior teaches the algorithm, the algorithm shapes what others see, and the cumulative behavior of millions of people like you determines what information environment the whole platform becomes. That's not a small thing. That's leverage — if you understand how to use it.
A major news organization has hired you to design the algorithm for their new news recommendation platform. They want to keep users engaged — that's how they pay the journalists — but they also have a stated mission of an informed public. You have to choose what metric the algorithm optimizes for. Your options include: time-on-site, diversity of perspectives read, accuracy of stories clicked, shares, or some combination.
Your lab partner Amara is a product ethicist who has worked on recommendation systems before. She has seen every rationalization in the book and won't accept vague answers. Every choice you make, she'll ask what you're trading away.