The checkpoint appeared ordinary: a gate, a camera, a brief pause. But the camera was running facial recognition software linked to a database that flagged individuals based on ethnicity, mosque attendance records, and contact lists. Abdulhakim Idris, a Uyghur teacher returning from visiting relatives, was pulled aside within seconds. Officers already knew his name, his employer, and that his cousin had applied for a passport three months earlier. He had not been charged with anything. He never would be. He simply disappeared into what the Chinese government calls "vocational training."
By 2019, Human Rights Watch and the Australian Strategic Policy Institute had documented over 380 detention facilities in Xinjiang. Satellite imagery and leaked government procurement documents showed that companies including Hikvision, Dahua, and SenseTime had supplied AI-powered surveillance infrastructure — facial recognition cameras, gait-analysis systems, and predictive-policing software — to what amounted to the largest mass internment of an ethnic minority since World War II.
Privacy is not a preference. Under international law it is a right. Article 12 of the Universal Declaration of Human Rights (1948) states: "No one shall be subjected to arbitrary interference with his privacy, family, home or correspondence." The International Covenant on Civil and Political Rights (ICCPR, 1966), binding on 173 states, repeats this protection in Article 17 and adds that any legal restriction must be proportionate, necessary, and non-discriminatory.
AI surveillance systems challenge all three of those conditions simultaneously. A facial-recognition camera in a public square does not target suspects — it processes every face it sees. That is mass collection, not targeted investigation. When the data collected is then cross-referenced with religion, ethnicity, or political affiliation — as documented in Xinjiang — the system becomes a machine for discrimination at scale.
Between 2016 and 2019, the Metropolitan Police Service trialled automated facial recognition (AFR) at public events including the Notting Hill Carnival and Champions League finals. Independent evaluations by the University of Essex (2019) found that 80% of matches were false positives — innocent people flagged as suspects. Officers stopped and demanded ID from individuals based solely on algorithm output. A legal challenge by civil liberties organization Liberty resulted in the Court of Appeal ruling in 2020 (R (Bridges) v Chief Constable of South Wales) that South Wales Police's AFR deployment violated the Human Rights Act and the Equality Act because no adequate legal framework governed it.
Modern AI surveillance is not a single camera. It is a layered infrastructure. China's "Sharp Eyes" program, announced in 2017, aimed to achieve full coverage of public spaces nationwide by 2020 using over 600 million cameras. The system integrates facial recognition, license-plate readers, mobile phone location data, purchasing history from Alipay and WeChat, and — in some cities — a "Social Credit Score" that restricts travel, loans, and employment for those deemed non-compliant.
Researchers from Carnegie Mellon and the Oxford Internet Institute have documented how this architecture migrates. Between 2008 and 2023, Chinese technology companies exported AI surveillance infrastructure to at least 80 countries, including Ecuador, Zimbabwe, Pakistan, and Serbia. The Carnegie Endowment for International Peace's AI Global Surveillance Index (2019) found that authoritarian and semi-authoritarian governments were the fastest adopters.
In 2019, San Francisco became the first city in the United States to ban government use of facial recognition technology. Oakland, Boston, and Portland followed. Their rationale was explicit in the legislation: the technology's error rates, combined with the chilling effects on free assembly and speech, made it fundamentally incompatible with civil liberties even when used for legitimate law-enforcement purposes.
The European Union's AI Act (2024) classifies real-time remote biometric identification in public spaces as a prohibited AI practice with narrow exceptions for specific terrorist threats — and only with prior judicial authorization. This represents the most comprehensive legal constraint on AI surveillance in any democratic jurisdiction.
Critics of outright bans argue that the technology, if accurate and governed by strict warrants, is simply a faster version of existing ID checks. Proponents of bans respond that speed and scale are not morally neutral: a system that can process ten million faces an hour is not qualitatively the same as a detective recognizing a suspect. It is infrastructure for control of populations, not investigation of individuals.
Every government that has deployed mass AI surveillance has cited public safety or counter-terrorism as justification. The human rights question is not whether safety matters — it does — but whether population-wide biometric monitoring is ever a proportionate response to specific threats, and who gets to decide when that threshold is crossed.
In this lab you will examine real-world AI surveillance deployments and apply the legal standard of proportionality — asking whether the rights restriction is necessary, non-discriminatory, and no greater than the legitimate aim requires.
Discuss each scenario with the AI assistant. Push back, ask for counterarguments, and explore where the human rights line should be drawn.
Vernon Prater was white, 41 years old, and had two armed robbery convictions. Brisha Borden was Black, 18 years old, and had been arrested for taking a bike with friends. A risk-assessment algorithm called COMPAS — Correctional Offender Management Profiling for Alternative Sanctions — scored Prater as low risk for reoffending. It scored Borden as high risk. Two years later, Prater had committed a series of new felonies. Borden had not been rearrested.
ProPublica journalists Julia Angwin and Jeff Larson published their analysis in May 2016: among defendants who did not reoffend, Black defendants were nearly twice as likely as white defendants to be labeled higher risk. Among those who did reoffend, white defendants were more likely to have been labeled lower risk. The algorithm had inverted the pattern it claimed to detect.
Non-discrimination is not a peripheral principle of human rights law — it is its foundation. Article 2 of the UDHR guarantees all rights "without distinction of any kind, such as race, colour, sex, language, religion, political or other opinion." The International Convention on the Elimination of All Forms of Racial Discrimination (ICERD, 1965), ratified by 182 states, extends this to prohibit not only intentional discrimination but disparate impact — practices that are neutral on their face but produce racially unequal outcomes.
COMPAS was not designed to discriminate. Its developers at Northpointe (now Equivant) argued that the algorithm was "race-neutral" because race was not an input variable. But the training data — prior arrests, prior convictions, neighborhood data — encoded decades of racially biased policing. Feeding biased history into a model produces biased predictions. The neutrality of the inputs does not launder the discrimination of the outputs.
HireVue's AI-powered video interview tool analyzed candidates' word choice, facial expressions, and vocal tone to score their suitability for jobs. The Electronic Privacy Information Center (EPIC) filed a complaint with the Federal Trade Commission in November 2019, arguing the system was a "black box" that could encode discrimination based on disability, race, and national origin. HireVue responded that the system had been audited for fairness. Faced with regulatory pressure from the Illinois Artificial Intelligence Video Interview Act (2020) — the first state law requiring bias audits of hiring AI — HireVue announced in January 2021 that it would discontinue its facial analysis feature. The company acknowledged it could not sufficiently validate that the feature was measuring job performance rather than demographic proxies.
Modern algorithmic discrimination rarely operates through explicit protected characteristics. Instead it operates through proxies — variables that correlate strongly with race, gender, or disability without naming them. Zip code as a proxy for race. Job title gaps as a proxy for gender. "Cultural fit" scores as a proxy for both.
In 2018, Reuters reported that Amazon had quietly scrapped an internal AI recruitment tool after discovering it systematically downgraded résumés containing the word "women's" (as in "women's chess club") and penalized graduates of all-women's colleges. The model had been trained on a decade of Amazon's own hiring decisions — decisions made in a tech workforce that was roughly 74% male. The algorithm had learned to reproduce the existing gender imbalance, not to overcome it.
Dutch tax authorities used an algorithmic fraud-detection system between 2013 and 2021 that flagged applicants for child benefit fraud based on having dual nationality. By 2020, parliamentary investigators confirmed the system had targeted some 26,000 families, the majority from minority ethnic backgrounds, with devastating consequences including bankruptcy, divorce, and loss of child custody. The scandal forced the resignation of the Dutch cabinet in January 2021.
Following the ProPublica COMPAS exposé, Northpointe responded with a counter-analysis demonstrating that by a different mathematical definition of fairness — calibration, meaning the algorithm's score predicts equally well across races — the tool was not biased. Both claims were mathematically true. The 2016 work by computer scientists Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan proved formally that when base rates differ between groups (i.e., actual reoffending rates differ between groups, as they do due to structural inequality), you cannot simultaneously satisfy calibration and equal false-positive rates. You must choose.
This is not merely a technical puzzle. It is a political and ethical choice about which kind of error is more acceptable. Accepting a higher false-positive rate for Black defendants — labeling more innocent people as dangerous — to achieve calibration means encoding a systematic human rights harm into the algorithm by design. The choice should be made explicitly, democratically, and with full legal accountability — not buried in a vendor's technical specification.
In most jurisdictions, defendants subject to algorithmic risk assessment have no right to see the algorithm, no right to challenge its output in court, and no right to know which variables drove their score. The right to a fair trial — Article 10 of the UDHR — arguably encompasses the right to confront the evidence against you. An opaque risk score that cannot be examined, cross-examined, or appealed arguably violates that right.
In this lab you will grapple with the impossible fairness problem: when base rates differ between groups, which definition of fairness should a system optimize for — and who should make that decision?
Work through concrete scenarios. Ask the AI to explain the trade-offs, play devil's advocate on both sides, and help you articulate a principled position on how algorithmic decision-making should be governed in contexts that affect human rights.
United Nations investigators later described it as a "textbook example of ethnic cleansing." Between August and September 2017, over 700,000 Rohingya Muslims fled Myanmar after military operations that included mass killings, rape, and arson. Years before the violence peaked, Facebook had become Myanmar's primary internet — and its primary news source. Hate speech targeting Rohingya was rampant. Myanmar military accounts posted content directly inciting violence. Facebook's AI content moderation system had almost no capacity to read Burmese and had fewer than five Burmese-speaking content reviewers for a country of 54 million.
A 2018 UN fact-finding mission stated explicitly that Facebook played a "determining role" in spreading hate speech that contributed to the violence. Meta acknowledged in 2021, through litigation in the United States and Kenya, that it had known of the problem years earlier. The lawsuits alleged that Meta's engagement-maximizing algorithm had specifically amplified inflammatory content because it generated more reactions — making the algorithmic architecture itself complicit in atrocity.
Article 19 of the UDHR protects the right to "freedom of opinion and expression" including the freedom "to receive and impart information and ideas through any media." The ICCPR's Article 19 allows restrictions only where necessary for respect of the rights or reputations of others, or for national security, public order, or public health — and only if those restrictions are provided by law and are proportionate.
Article 20 of the ICCPR goes further: it requires states to prohibit advocacy of national, racial, or religious hatred that constitutes incitement to discrimination, hostility, or violence. The Myanmar case therefore presents a dual failure: the algorithm amplified content that states are legally obligated to prohibit, while simultaneously failing to moderate it — a violation of both the right to free expression and the right to be protected from incitement.
The structural problem is one of scale and architecture. A human editor reviewing a post can apply context, consider the speaker's history, and assess the likely audience. An engagement-maximizing algorithm does none of this. It asks: will this content drive interaction? Content that triggers emotional reactions — outrage, fear, disgust — reliably does. The algorithm is not neutral; it has a value embedded in its objective function, and that value is engagement, not truth or safety.
In May 2021, during the Israeli military operation in Gaza, Human Rights Watch, Amnesty International, and dozens of journalists documented a wave of Instagram and Facebook removals of Palestinian content — including first-hand documentation of airstrikes, photos of destroyed homes, and posts using the term "Nakba" (the Arabic word for the 1948 Palestinian exodus). Meta's own Oversight Board acknowledged in a September 2021 report that Arabic-language content moderation had significant accuracy deficits. A 2021 internal audit commissioned by Meta and conducted by Business for Social Responsibility concluded that the company's human rights policies had failed to prevent suppression of Palestinian voices. Meta agreed to remediation steps but did not commit to equal enforcement standards across language groups.
Content moderation AI operates at extraordinary scale: Meta processes approximately 100 billion pieces of content per day. No human team could review this volume. But the systems that handle this volume embed asymmetries that consistently disadvantage minority-language speakers, journalists in conflict zones, and human rights defenders documenting abuses.
The Global Network Initiative, a multistakeholder body that includes Google and Meta, has produced principles requiring companies to assess the human rights impact of their moderation systems. But these principles are voluntary and their implementation is self-assessed. In 2022, the UN Special Rapporteur on Freedom of Expression, Irene Khan, called for legally binding standards requiring platforms to conduct and publish human rights impact assessments before deploying automated content moderation systems.
The DSA — Digital Services Act, which entered full force in the EU in February 2024 — requires very large online platforms to conduct annual systemic risk assessments including fundamental rights impacts, submit to third-party audits, and make their algorithmic recommender systems accessible to vetted researchers. This is the most substantive binding framework yet applied to platform AI and freedom of expression.
The free expression threat is not only algorithmic error. In multiple documented cases, AI companies have actively assisted government censorship. Google launched "Project Dragonfly" in 2018 — a censored version of its search engine designed for China that would have blacklisted searches for "human rights," "student protest," and "Nobel Prize" (Liu Xiaobo had won in 2010). Internal protests by Google employees and reporting by The Intercept forced the project's suspension in 2019, though Google has not publicly committed to never resuming it.
Apple removed apps from its Chinese App Store at the request of Chinese authorities, including VPN tools used by activists and journalists, a Quran app, and the New York Times app. LinkedIn shuttered its social features in China in 2021 rather than comply with requests to censor political content — a decision critics called overdue and others called a model for principled withdrawal. The UN Guiding Principles on Business and Human Rights (Ruggie Principles, 2011) establish that corporations have a responsibility to respect human rights even where governments do not require it — meaning compliance with censorship requests is not automatically a defense.
AI content moderation simultaneously threatens free expression from two directions: by failing to remove incitement to violence (Myanmar, 2017), and by over-removing protected speech from marginalized communities (Palestine, 2021). Both failures are not random — they are predictable consequences of training data that underrepresents minority languages, of engagement objectives that reward inflammatory content, and of governance structures that place these decisions inside private companies with no democratic accountability.
In this lab you will work through the structural tensions in AI content moderation: how do you build a system that removes incitement to violence without suppressing minority voices? How should accountability be structured when private companies make speech decisions affecting billions?
Engage the AI assistant to develop principled frameworks. Challenge its proposals, explore the enforcement gaps, and consider what binding governance would look like.
A UN Panel of Experts report, published in March 2021, described an incident during the Libyan civil war in which a Kargu-2 drone — a Turkish-made loitering munition capable of autonomous target engagement — had "hunted down and remotely engaged" retreating fighters without requiring human command input for each strike. This may represent the first documented use of a lethal autonomous weapons system (LAWS) in combat. The panel's language was careful, the evidence fragmentary, but the implication was historic: a machine may have made a kill decision without a human in the loop.
The incident received limited press coverage. There were no war crimes trials, no Security Council resolution, and no international mechanism capable of investigating the incident authoritatively. This accountability vacuum — not the drone itself — is what human rights organizations including Human Rights Watch and the International Committee of the Red Cross called the most alarming aspect of the event.
International humanitarian law (IHL) — the body of law governing armed conflict, codified primarily in the Geneva Conventions (1949) and their Additional Protocols (1977) — requires combatants to observe four core principles: distinction (between combatants and civilians), proportionality (no excessive civilian harm relative to military advantage), precaution (taking all feasible measures to minimize civilian harm), and military necessity (attacks limited to what is necessary to achieve a legitimate military objective).
These principles require judgment. Distinction requires assessing whether a person is a combatant or a civilian — a determination that can depend on whether someone is actively participating in hostilities, their location, the time of day, and context that changes moment to moment. No existing AI system has demonstrated the capacity to make these assessments reliably in dynamic combat environments. Yet fully autonomous weapons are currently under development by at least nine states including the United States, Russia, China, Israel, South Korea, and the United Kingdom.
Investigative reporting by +972 Magazine and Local Call (April 2024), based on interviews with Israeli intelligence officers, disclosed that the Israeli military had used AI systems called "Gospel" and "Lavender" to generate bombing target lists in Gaza. "Lavender" reportedly processed data on 37,000 individuals and assigned each a probability score for being a Hamas militant. Officers described the system as having a "machine-like" error rate of approximately 10%, meaning roughly 3,700 people flagged were likely civilians. The system reportedly allowed strikes on private homes of identified individuals, with officers describing target approval as taking "20 seconds" per case. Human Rights Watch and Amnesty International called for independent investigation; the Israeli military disputed characterizations of the system's autonomy. No binding international accountability mechanism has yet been triggered.
Traditional IHL assigns responsibility through a chain of command. A soldier commits a war crime; their commanding officer who ordered or failed to prevent it bears command responsibility. This framework presupposes a human decision-maker at each link in the chain. Autonomous weapons systems break the chain. When a machine selects a target and fires without human authorization for that specific decision, determining who violated IHL — and how they can be held accountable — becomes structurally impossible under existing frameworks.
Peter Asaro, philosopher at The New School and co-founder of the International Committee for Robot Arms Control, has argued that this creates what he calls a "responsibility gap" — a space in which atrocities can occur without legal accountability because no individual human being made the lethal decision. The gap is not an accident; it functions as legal insulation. Designing a system to be autonomous is, on this analysis, designing a system to evade accountability.
The Campaign to Stop Killer Robots, a coalition including over 170 organizations, has called for a legally binding treaty prohibiting fully autonomous weapons. As of 2024, the Convention on Certain Conventional Weapons (CCW) has held informal talks since 2014 but has not produced a binding instrument, primarily because states developing autonomous capabilities — including the United States, Russia, and China — have blocked progress toward binding obligations.
The "Lavender" system's disclosed logic — assigning probability scores for militant status and using those scores to authorize lethal action — represents the migration of predictive policing into armed conflict. In both contexts, the fundamental human rights problem is the same: a person is harmed based on a statistical probability, not a specific act, and has no opportunity to challenge the evidence against them.
The right to life — Article 3 of the UDHR, Article 6 of the ICCPR — is the most fundamental of all human rights. IHL permits killing in armed conflict only within strict constraints. Algorithmic targeting that operates at speed, at scale, with admitted error rates, and without meaningful human review of individual cases, does not obviously satisfy those constraints. The right to life does not become negotiable because the system that threatens it is efficient.
In 2023, the Political Declaration on Responsible Military Use of Artificial Intelligence and Autonomy — signed by 50 states — affirmed that IHL applies to AI-enabled weapons and that states must exercise human judgment over lethal decisions. Critics noted that the Declaration is non-binding, contains no verification mechanism, and was explicitly declined by Russia and China. It represents aspiration, not accountability.
Whether or not any specific autonomous weapon system has yet committed a war crime, the direction of development is clear: weapons systems are becoming faster, more autonomous, and more capable of lethal action without human authorization. The question human rights law must answer — urgently, before the technology races past the law — is whether the right to life requires a human being to make the decision to end it.
In this lab you will work through the legal and philosophical frameworks needed to govern lethal autonomous weapons. What does "meaningful human control" actually require? How should IHL be updated to address algorithmic targeting? And what binding treaty language could close the responsibility gap?
Engage the AI assistant to stress-test proposed frameworks, explore state objections, and develop principled positions on where the line between human decision-making and machine autonomy must be drawn in armed conflict.