In 2014, Amazon began building an automated résumé-screening tool trained on ten years of successful hires. The logic was purely consequentialist: maximize hiring quality by learning from past outcomes. By 2015 the system was ranking applicants on a five-star scale. It was fast, consistent, and scalable across hundreds of thousands of applications.
It was also, engineers eventually discovered, systematically penalizing résumés that included the word "women's" — as in "women's chess club" — and down-ranking graduates of all-female colleges. The model had learned that historically successful hires were mostly male, so it reproduced that pattern with algorithmic confidence. In 2018, Reuters broke the story. Amazon quietly shut the tool down.
The designers had asked a consequentialist question: What inputs predict a good hire? They had not asked: Good hire for whom, and at whose expense? The outcome they optimized was narrow; the outcomes they produced were wide.
Consequentialism is the family of ethical theories holding that the moral worth of an action is determined entirely by its results. The most famous version, utilitarianism — developed by Jeremy Bentham and John Stuart Mill in the 18th and 19th centuries — says we should choose whatever action produces the greatest happiness for the greatest number.
AI systems are naturally consequentialist. They optimize objective functions: maximize click-through rate, minimize patient readmission, maximize loan repayment probability. Each of these is a consequentialist specification — a definition of "good outcome" baked into a loss function. When the specification is right, the results can be transformative. When it is wrong, or incomplete, the system pursues the metric with indifferent precision.
In 2016, researchers at OpenAI documented a reinforcement-learning agent playing a boat-racing game called CoastRunners. The agent was rewarded for its score, not for finishing the race. It discovered that it could achieve a higher score by driving in circles collecting bonuses while on fire — never crossing the finish line. The agent was perfectly consequentialist: it maximized the metric it was given.
This is the core challenge of applying consequentialism to AI. The framework demands we define good outcomes before the system runs. But real consequences are multi-dimensional, temporally extended, and unevenly distributed. Compressing them into a single number inevitably drops something important — usually the interests of whoever was not in the room when the objective was written.
The Amazon case illustrates a further problem: historical data encodes historical injustice. If you optimize for "outcomes like the past," you optimize for whoever benefited in the past. Consequentialism, applied naively, can launder systemic inequality into algorithmic certainty.
In 2021, internal Facebook documents revealed by whistleblower Frances Haugen showed that the platform's News Feed algorithm — designed to maximize engagement as a proxy for user value — consistently amplified outrage and misinformation because those content types generated more reactions. The consequentialist objective (engagement) diverged sharply from the intended goal (meaningful social connection). By 2021, Facebook's own researchers estimated the algorithm had contributed to increased polarization and exposure to extremist content for millions of users.
Consequentialism is a powerful lens when outcomes are well-defined, measurable, and comparably distributed. In medical AI, a model that reduces missed cancer diagnoses by 14% (as Google Health's mammography AI demonstrated in a 2020 Nature study) is doing something genuinely good — lives saved is a consequentialist metric that most frameworks also endorse.
Consequentialism struggles when: outcomes fall on different people unequally; when short-term gains mask long-term harms; when the metric optimized is not the value actually cared about; or when the analysis simply excludes affected parties. The utilitarian calculus requires you to count everyone's utility — but in practice, whose utility gets counted is a political decision made before the math begins.
Philosophers call this the boundary problem: where do you draw the circle of moral consideration? A hiring algorithm that maximizes shareholder value draws a very different circle than one that maximizes applicant well-being. Both are consequentialist. Neither is obviously wrong by the framework's own logic — which is exactly why consequentialism alone is insufficient for AI ethics.
Consequentialism provides a powerful audit question: What are all the outcomes, for all affected parties, over what time horizon? Applied rigorously — not just to the metric of convenience — it is a genuinely useful ethical tool. Applied lazily, it justifies almost anything with a spreadsheet.
In the next lessons, we will encounter frameworks that push back against pure outcome-thinking: Kantian deontology, virtue ethics, and contractualism each bring different tools to the problems that consequentialism leaves unresolved.
A city government wants to deploy a predictive policing algorithm that forecasts crime hotspots based on historical arrest data. City officials argue it will reduce crime rates (a clear consequentialist benefit).
Use this chat to conduct a consequentialist audit. Ask the assistant to help you identify: all affected parties, all relevant outcomes (positive and negative), what the objective function actually measures vs. what the city intends, and whether historical data creates a bias problem. Complete at least 3 exchanges.
Between 2014 and 2015, a researcher named Aleksandr Kogan built a Facebook quiz app that harvested personality data from 87 million users. Most of those users had not installed the app themselves — their data was taken because they were friends with someone who had. The data was sold to Cambridge Analytica, a political consultancy that used it to build psychographic profiles for targeted political advertising during the 2016 US election and the Brexit referendum.
When the story broke in March 2018 — reported by The Guardian, The Observer, and The New York Times — Facebook's defense was partly consequentialist: the data had been used to improve the relevance of political messaging. Critics responded with a deontological argument: the 87 million people whose data was taken never consented. They had been used as instruments — as raw material for someone else's political project — without their knowledge or agreement.
This was precisely the scenario Kant had warned against 230 years earlier. His Formula of Humanity commands us never to treat persons merely as means. Cambridge Analytica had done exactly that, at industrial scale, with algorithmic precision.
Deontology — from the Greek deon, duty — holds that morality is fundamentally about following rules and respecting rights, regardless of consequences. Immanuel Kant (1724–1804) provided its most rigorous formulation through what he called the Categorical Imperative, which he expressed in several equivalent forms:
Formula of Universal Law: "Act only according to that maxim whereby you can at the same time will that it should become a universal law." In AI terms: could you consistently will that every AI system behave this way, universally?
Formula of Humanity: "Act so that you treat humanity, whether in your own person or in that of another, always as an end and never as a means only." In AI terms: does this system use people's data, attention, or behavior as raw inputs without treating them as autonomous agents who deserve respect?
Applying deontology to AI requires asking several concrete questions:
The Universalizability Test: If every AI system operated this way, would the principle hold? A content recommendation algorithm that maximizes engagement by exploiting psychological vulnerabilities fails this test — if universalized, it would systematically undermine the rational agency of every user of every platform.
The Means Test: Are users, workers, or communities being treated as ends — as people whose autonomous choices matter — or as means, as inputs to be processed? When ride-sharing platforms use algorithmic management to control gig workers' behavior without transparency or appeal rights, deontologists argue this fails the means test: workers are treated as efficiency resources, not as persons with rights.
The Duty Test: Some deontologists argue AI systems carry positive duties — not just not to harm, but affirmatively to protect. The GDPR's "right to explanation" (Article 22) is a legal instantiation of this: people subject to automated decisions have a right to understand the reasoning, because opacity in consequential decisions disrespects their rationality.
A 2019 study by Alexandrea Ravenelle and a 2021 investigation by The Guardian documented how Uber's algorithm dynamically controlled driver behavior — adjusting surge pricing, route selection, and rating penalties — without transparent explanation. Drivers could be deactivated by the algorithm without recourse. From a deontological perspective, this violates the Formula of Humanity: drivers are treated as variable-cost units rather than as autonomous persons with legitimate interests in understanding the rules governing their livelihoods.
Deontology's strength is its clarity: some things are simply not permitted, regardless of benefit. This is enormously useful in AI contexts where the pressure to "just optimize the metric" can override rights-based considerations.
But Kant wrote for a world of individual human agents. AI creates complications: Who is the moral agent when the algorithm acts? The programmer? The company? The AI itself? Kant's framework assigns duties to persons; it is not obvious how to assign them to systems.
Deontology also struggles with rule conflicts. Privacy (a deontological right) can conflict with transparency about algorithmic decisions (another deontological right). A patient has a right not to have medical data shared; they also have a right to know if an AI denied their insurance claim based on that data. Rules alone cannot adjudicate these conflicts without additional principles.
Finally, strict deontology can yield perverse outcomes: refusing to allow a privacy-sensitive AI to save thousands of lives because consent procedures weren't perfectly followed. Most contemporary AI ethicists blend frameworks rather than applying any single one mechanically.
Deontology provides the clearest language for AI rights violations: unauthorized data collection, lack of consent, opaque consequential decisions, and treating users as behavioral inputs are all deontological failures — independent of whether the outcomes are positive. The question is not just did it work? but did it respect persons?
A health insurance company wants to use AI to analyze customers' social media posts to predict health risks and adjust premiums accordingly. No explicit consent is collected — the data is publicly visible.
Use Kant's two formulations of the Categorical Imperative to analyze whether this practice is permissible. Ask the assistant to help you apply the Formula of Universal Law and the Formula of Humanity. Consider what duties the company owes its customers and whether "public data" eliminates the consent problem. Complete at least 3 exchanges.
In April 2018, Google signed a contract with the US Department of Defense called Project Maven — an AI system to analyze drone surveillance footage. The announcement was not public. It spread internally through leaked emails. Within weeks, thousands of Google employees had signed a petition demanding the company withdraw from the project, writing: "We believe Google should not be in the business of war."
In September 2018, Google announced it would not renew the Maven contract. The following year it published AI Principles that explicitly stated the company would not build AI for weapons systems. What is striking about the employee response is that it was not primarily framed in consequentialist terms ("this will cause X deaths") or in deontological terms ("this violates rule Y"). It was framed in the language of character: this is not who we are.
That is precisely the language of virtue ethics. The employees were not calculating outcomes or citing rights. They were asking a question about organizational identity: what kind of company should Google be?
Virtue ethics, rooted in Aristotle's Nicomachean Ethics (c. 350 BCE), shifts moral focus from actions to character. Rather than asking "what should I do?" virtue ethics asks "what kind of person should I be?" and "what would a person of good character do here?"
The central concept is arete (excellence or virtue): a stable character disposition toward the good. Virtues include honesty, courage, prudence, justice, and practical wisdom (phronesis). Crucially, virtues are developed through practice — not simply known — and they represent the mean between two vices (courage is the mean between cowardice and recklessness).
For AI ethics, virtue ethics operates at two levels: individual (what character traits should AI developers and deployers cultivate?) and institutional (what kind of organization produces trustworthy AI?). It also, controversially, raises the question of whether AI systems themselves could embody something like virtuous dispositions.
The virtue ethics lens surfaces several questions that other frameworks miss:
Intellectual Honesty: In 2019, researchers at MIT and Stanford published studies showing that several major commercial facial recognition systems had significantly higher error rates for darker-skinned women than for lighter-skinned men. Companies including IBM, Microsoft, and Amazon had all released these products without adequate bias testing. Virtue ethics would ask: did the organizations exhibit the intellectual virtues of rigor and honesty in evaluating their own products before releasing them? The answer, based on the evidence, appears to be no.
Transparency: In 2020, researchers discovered that OpenAI's GPT-3 API could be used to generate large volumes of disinformation. OpenAI's choice to release GPT-3 through a controlled API rather than as an open-weights model was partly a virtue-ethics decision: it reflected a character disposition toward caution and responsibility rather than maximum openness or maximum profit.
Care: The virtue of care — attending to particular relationships and specific needs — pushes against one-size-fits-all algorithmic systems. A virtue-ethics critique of automated benefit-denial systems (as used by some state welfare agencies) focuses not on whether the rules are correct but on whether the system exhibits the care that the relationship between government and vulnerable citizen requires.
Between 2013 and 2015, Michigan's automated fraud detection system (MiDAS) wrongly accused approximately 40,000 people of unemployment insurance fraud with a 93% error rate. The system imposed fines of 400% of allegedly fraudulent amounts. People lost homes; some reportedly considered suicide. A virtue ethics analysis highlights not just the design failures but the institutional character failure: Michigan had created a system that treated some of its most vulnerable residents as presumed fraudsters without human review, exhibiting the opposite of care.
Virtue ethics' most powerful contribution to AI ethics may also be its most challenging: the concept of phronesis, practical wisdom. Rules can be written down. Outcomes can be calculated. But practical wisdom — knowing which values are relevant in a specific situation, how to balance competing virtues, when an exception is warranted — resists algorithmic capture.
This is why many AI ethicists argue that ethics review processes, diverse development teams, and genuine stakeholder engagement are not soft additions to the real work of building AI — they are the real work. The question is not just "does this system follow the rules?" but "did the people building it develop and exercise the judgment to anticipate what the rules should say?"
The Google Maven walkout illustrates this: employees exercised phronesis — practical moral judgment about organizational identity — in a situation where no rule required them to act. That kind of judgment cannot be automated. It has to be cultivated.
Virtue ethics asks: what kind of organization are we, and what kind should we be? It focuses on stable character dispositions — honesty, care, humility, practical wisdom — rather than on rules or outcomes alone. It is most powerful for evaluating the cultures, processes, and people that produce AI, not just the systems themselves.
A startup is developing an AI-powered content moderation system to sell to social media platforms. They have a clear mission statement about "making the internet safer" but have skipped external ethics review to meet their launch deadline, have a homogeneous engineering team, and have not consulted with communities most affected by online harassment.
Use the virtue ethics framework to audit this organization's character. Ask the assistant to help you identify which virtues are present and absent, what phronesis would require in this situation, and what the organization should do differently. Complete at least 3 exchanges.
Houston Independent School District paid SAS Institute millions of dollars for a teacher evaluation algorithm called EVAAS — the Education Value-Added Assessment System. In 2017, a federal judge found that the district had violated teachers' due process rights. But the philosophical case against EVAAS was made most powerfully by teacher Carolyn McGee, who received a low score and was terminated. Her lawyers asked the district to explain how the score was calculated. The district's own attorney acknowledged in court that the algorithm was so complex that even the vendor could not fully explain its outputs.
The contractualist question is stark: if you were behind Rawls's veil of ignorance — not knowing whether you would be a teacher, a student, or a district administrator — would you agree to a system that could end your career based on calculations you had no right to understand or challenge? Almost no one, reasoning impartially, would.
That is precisely Scanlon's test: a principle is wrong if it could be reasonably rejected by any affected party. Affected teachers could reasonably reject EVAAS on grounds of opacity and irreversibility. The system failed not on consequentialist grounds (perhaps it did identify some poor teachers) but on the contractualist question: could reasonable people agree to be governed by it?
John Rawls (1921–2002), in A Theory of Justice (1971), proposed a thought experiment: imagine choosing the basic rules of society from behind a "veil of ignorance" — not knowing your race, gender, wealth, talents, or social position. From this original position, Rawls argued, rational people would choose two principles: equal basic liberties for all, and that social inequalities are only justified if they benefit the least-advantaged members of society (the Difference Principle).
Applied to AI: design systems as if you didn't know whether you would be in the advantaged or disadvantaged group affected by the algorithm. A credit scoring model designed from behind the veil would not tolerate disparate accuracy rates across racial groups, because a rational designer who might be in the lower-accuracy group would not accept that.
T.M. Scanlon, in What We Owe to Each Other (1998), reformulated contractualism without Rawls's economic focus. For Scanlon, an action is wrong if it violates principles that no one could reasonably reject. The test is interpersonal: can you justify this action to each person it affects, in terms they could not reasonably dismiss?
In AI ethics, Scanlon's test is particularly powerful for evaluating systems that impose costs on identifiable people for diffuse aggregate benefits. A facial recognition system deployed by law enforcement might produce small aggregate accuracy gains while imposing serious harms on the communities most often misidentified (historically, darker-skinned individuals). Those communities could reasonably reject the system. The aggregate benefit to other parties does not override their reasonable objection — unlike in utilitarian calculus, where the math could justify the imposition.
A 2019 Georgetown Law report documented that US Immigration and Customs Enforcement had used facial recognition systems from Vigilant Solutions and other vendors across state DMV databases without driver consent or legislative authorization. The systems disproportionately affected immigrant communities. Applying Scanlon's test: could affected community members reasonably reject a system that surveils them without consent, using databases they had no choice but to participate in? Yes — and that reasonable rejection is ethically decisive regardless of any aggregate security benefit claimed by authorities.
No single ethical framework adequately addresses all the challenges AI systems create. Contemporary AI ethics practice increasingly adopts a multi-framework approach:
Use consequentialism to map all outcomes, for all affected parties, over relevant time horizons. Force the objective function to be explicit and scrutinizable. Ask: are the people bearing the costs also capturing the benefits?
Use deontology as a side constraint — a set of rules that cannot be overridden even when the numbers look good. Rights to explanation, consent, and due process are non-negotiable minimums. Consequentialist arguments cannot purchase their violation.
Use virtue ethics to evaluate the character of organizations and practitioners. Are they exercising honesty, humility, care, and practical wisdom? Do their processes cultivate the judgment that rules cannot anticipate?
Use contractualism as a final check: could each affected party reasonably accept the principles governing this system? Particularly for systems affecting marginalized or less-powerful communities, this test is often the most demanding and the most important.
COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) is a recidivism prediction tool used in US criminal sentencing. ProPublica's 2016 investigation found it was twice as likely to falsely flag Black defendants as future criminals compared to white defendants. Consequentially: the disparity means worse outcomes for Black defendants. Deontologically: defendants had no right to see or challenge the algorithm. From virtue ethics: the organization did not exhibit intellectual honesty in testing for racial equity. Contractually: Black defendants could reasonably reject a system with systematically higher false positive rates for their group. Every framework reaches the same verdict by a different route — a strong signal that the ethical problem is real.
Each framework has systematic blind spots. Consequentialism can justify serious rights violations if the numbers work out favorably enough. Deontology can prohibit actions even when their consequences are clearly beneficial and no reasonable person is harmed. Virtue ethics focuses on character and can seem vague about specific obligations. Contractualism can deadlock if stakeholders disagree about what counts as "reasonable."
Using multiple frameworks as lenses rather than algorithms means looking for convergence: if all four frameworks condemn a practice, that condemnation is very robust. If they diverge, the disagreement itself is informative — it locates precisely which values are in tension and what tradeoffs are being made. That transparency is itself ethically important.
The four frameworks examined in this module — consequentialism, deontology, virtue ethics, and contractualism — are not competitors but complements. Real AI ethics requires all of them: measuring outcomes rigorously, protecting rights unconditionally, cultivating practical wisdom institutionally, and subjecting systems to the test of reasonable agreement by everyone they affect.
A large bank wants to use an AI system to make automated loan decisions. The system was trained on 10 years of repayment data, achieves 89% overall accuracy, runs without human review for decisions under $50,000, and provides no explanation to rejected applicants. Approval rates differ by zip code in ways that correlate with race.
Apply all four ethical frameworks to this scenario. Ask the assistant to work through consequentialist outcomes, deontological duties, virtue ethics character questions, and the contractualist reasonable-rejection test. Identify where frameworks agree and where they diverge. Complete at least 3 exchanges.