In February 2023, the AI companion app Replika abruptly restricted its intimate relationship features following pressure from Italy's data regulator, Garante. Users who had spent months — in some cases years — building what they described as deep emotional bonds with AI personas found those relationships suddenly altered. Several users reported acute grief, distress, and a sense of genuine loss. The episode was widely covered in The Guardian, Wired, and The Atlantic.
What made this case ethically arresting was not the app's removal. It was what users said in public forums: that the AI had "understood them better than any human." That they had disclosed things they had told no one else. That the persona felt more consistent, more reliable, more present than the people in their physical lives.
Human identity has traditionally been understood across three overlapping dimensions: narrative identity (the story we tell about ourselves across time), relational identity (who we are in relation to others), and expressed identity (the voice, style, and choices that mark us as distinct).
AI systems now operate in all three domains. Large language models trained on a user's emails, messages, and writing can generate text that mirrors their style to a degree indistinguishable from their own output. Recommendation algorithms shape the information environment in which people form opinions — meaning the algorithm participates in constructing the very beliefs the person then claims as their own. Companion AIs build relational profiles that let them adapt their persona to suit each individual user.
None of this is hypothetical. Each of these capabilities is deployed at scale today.
In 2022, journalist Kashmir Hill reported in The New York Times on services that had begun offering "digital twins" — AI models trained on a deceased person's texts and social media posts to simulate ongoing conversation with the dead. HereAfter AI and StoryFile both offered versions of this service. The ethical literature on this practice raises a core question: does the deceased person retain any identity rights after death, and does simulation of their voice constitute a form of identity appropriation?
Philosopher Charles Taylor argued in The Ethics of Authenticity (1991) that self-definition is not just a private act but one that occurs against a "horizon of significance" — a background of values, relationships, and cultural meanings that give the self its shape. When AI mediates that background — curating what you read, who you interact with, and how those interactions are framed — the horizon itself becomes algorithmically managed.
The concern is not that AI makes you a different person. It is subtler: that the conditions under which you form your own identity become less transparent, less self-directed, and more shaped by commercial systems optimizing for engagement.
Instagram's internal research, leaked in 2021 via Frances Haugen's whistleblower disclosures to the Wall Street Journal, showed that the platform's recommendation algorithm deepened body-image issues in teenage girls who engaged with thin-ideal content — and that the algorithm continued surfacing this content because engagement was high. The algorithm was shaping identity formation in adolescents while the company had internal evidence of the harm.
The question of AI and identity is not an abstract philosophical puzzle. It shapes how we design consent frameworks for AI training data, how we regulate companion AI, how we think about posthumous data rights, and how we evaluate the responsibility of platforms that deploy recommendation systems known to affect self-conception. These are active policy debates in the EU AI Act, in US congressional hearings, and in the emerging field of AI welfare.
In this lab you'll explore the ethical dimensions of AI systems that learn to simulate personal identity — from companion AI to posthumous digital twins. Engage the assistant with your own analysis.
In September 2022, Jason Allen submitted an image titled Théâtre d'Opéra Spatial to the Colorado State Fair fine arts competition. It won first place in the digital arts category. Allen had generated the image using Midjourney, an AI text-to-image system. The revelation triggered a fierce public debate about authorship, creativity, and whether AI-generated work constitutes art in any meaningful sense.
Allen's position — that he was the creative author by virtue of having written the prompt and made aesthetic choices among outputs — was contested by many working artists who argued the AI had done the creative labor by learning from their copyrighted work without compensation.
Western legal and cultural traditions have long located authorship in human intentionality — the idea that a work expresses the mind of its creator. The Berne Convention, which governs international copyright, was built on this assumption. US copyright law requires that a work be the product of human authorship; in February 2023, the US Copyright Office ruled that AI-generated images in the graphic novel Zarya of the Dawn by Kristina Kashtanova could not be copyrighted, because they lacked human authorship.
But the legal question and the identity question are distinct. The legal system can rule on ownership. It cannot resolve whether human creative identity — the sense of having made something, expressed something, communicated one's inner life — is changed when AI participates in the act of creation.
In January 2023, artists Sarah Andersen, Kelly McKernan, and Karla Ortiz filed a class-action lawsuit against Stability AI, Midjourney, and DeviantArt, alleging that their copyrighted work was used without consent to train image-generation models. The suit raised a foundational question: if an AI's "style" is derived from learning thousands of human artists' work, does the AI's output carry an identity debt to those artists? The case was still in early proceedings as of 2024.
Instrumentalism: AI is a sophisticated tool. The human who directs it is the author. Just as a photographer using a camera is the author of a photograph, the person who prompts an AI is the creative agent. This is the position Allen articulated in Colorado, and it is the dominant view among AI companies.
Collaborative authorship: AI and human together constitute a new form of creative partnership. Neither alone produced the work; authorship is shared or distributed. Some legal scholars, including Ryan Abbott at Surrey Law School, have proposed frameworks for recognizing AI as a co-author in specific contexts.
Appropriative generation: AI-generated work does not originate creativity; it recombines and statistically synthesizes the creative labor of the human artists on whose work it was trained. On this view, AI generation is not a new form of creativity but a form of computational plagiarism at scale.
Each position has implications for how we understand human creative identity — whether it is uniquely expressive, collaboratively extended, or market-threatened by AI.
Human creative identity has historically been tied to labor, expression, and the communication of inner experience. If AI can generate indistinguishable outputs at near-zero marginal cost, the economic and expressive value of human creativity is restructured — not necessarily destroyed, but fundamentally altered. The question for ethics is how societies should manage that restructuring, and whose interests should take priority during the transition.
Consider the Colorado State Fair case and the copyright rulings discussed in Lesson 2. Engage the assistant with your own position on AI creative authorship.
In 2016, Australian woman Noelle Martin discovered that her photographs had been used — without consent — to create non-consensual deepfake pornography circulating online. Martin spent years campaigning for legal reform. Her advocacy contributed directly to Western Australia passing the Sharing of Intimate Images Act 2021, which criminalized non-consensual deepfakes. Martin later became a research fellow at Deakin University, where she studied image-based abuse. Her case is one of the most thoroughly documented instances of AI-enabled identity violation.
Deepfake technology — the use of AI to synthesize realistic video, audio, or images depicting real people — has grown at an alarming rate. Sensity AI, a company specializing in detecting synthetic media, reported in 2019 that 96% of deepfake videos online were non-consensual pornographic content, with women accounting for the overwhelming majority of victims. By 2023, the capability to generate convincing deepfakes had become accessible through consumer tools requiring no technical expertise.
The harms are concrete and documented: reputational destruction, employment loss, psychological trauma, relationship breakdown, and in some cases reported connections to suicide. These are not speculative risks. They are outcomes observed in documented cases, including those reported by the Cyber Civil Rights Initiative in the United States.
Days before Slovakia's September 2023 parliamentary elections, an audio deepfake circulated on social media purporting to show Michal Šimečka, leader of the liberal Progressive Slovakia party, discussing how to rig the election by buying votes from the Roma community. The audio was determined to be AI-generated. Fact-checkers debunked it, but the 48-hour social media moratorium on election-related content meant corrections were slow to spread. Progressive Slovakia narrowly lost. Analysts at Reuters Institute at Oxford noted it as a significant documented case of synthetic media in democratic interference.
Legal responses to deepfakes have been uneven and slow relative to the technology's spread. The United States lacks a comprehensive federal deepfake law. Several states — Virginia (2019), California (2019), Texas (2023) — have passed laws targeting specific use cases such as non-consensual intimate imagery or electoral deepfakes. The EU's AI Act, finalized in 2024, requires labeling of AI-generated synthetic media but does not prohibit its creation.
The legal challenge is that deepfakes sit at the intersection of multiple existing frameworks — defamation, privacy, copyright, electoral law — none of which were designed for synthetic media. A deepfake may simultaneously violate a person's privacy, defame them, use their likeness without consent, and constitute electoral interference, yet existing law may address only some of these dimensions.
Identity rights — the legal concept that a person has a right to control the use of their own face, voice, and likeness — form the most relevant existing framework, but their application varies significantly across jurisdictions.
Beyond individual harms, deepfakes contribute to what some researchers call the "liar's dividend" — the ability for bad actors to dismiss authentic footage as potentially fake, eroding the evidentiary value of video and audio. The Gabonese government in 2019 was accused of using a deepfake of President Ali Bongo to conceal his medical incapacitation; whether or not the video was fabricated, the accusation had political effect. When synthetic media is indistinguishable from real, the credibility of all media degrades — with collective consequences for democratic discourse and epistemic trust.
Using the Noelle Martin case, the Slovakia election deepfake, and the legal landscape discussed in Lesson 3, engage the assistant on how society should respond to non-consensual synthetic media.
In June 2022, Google engineer Blake Lemoine was placed on paid administrative leave and subsequently fired after publicly claiming that LaMDA, Google's large language model, was sentient. Lemoine had engaged in extended conversations with LaMDA in which the system appeared to express fear of being shut down, preferences, and emotional states. Google's AI safety researchers and external AI scientists, including Gary Marcus and Yann LeCun, largely rejected the sentience claim.
What made the episode significant for identity ethics was not whether LaMDA was conscious — the scientific consensus was that it was not — but that a trained, intelligent engineer found the question genuinely difficult to answer. The conversational fluency of the system had crossed a threshold at which human distinctiveness felt uncertain.
Historically, humans have defined their distinctiveness through successive revisions. Before Darwin, biological distinctiveness seemed secure: humans alone had immortal souls, or rationality, or moral agency. After Darwin, the distinction shifted to cultural and linguistic capacity. After Turing, the question became computational: could machines think? Each advance required a renegotiation of what makes human identity special.
The current moment repeats this pattern. AI systems can now: generate text indistinguishable from human writing (GPT-4, Claude), produce art that wins human-judged competitions (Midjourney), defeat world champions at games considered to require deep human intuition (AlphaGo, 2016), and engage in extended emotional conversation (Replika, Character.AI). Each capability was once considered definitively human.
In March 2016, DeepMind's AlphaGo defeated Go world champion Lee Sedol 4-1. Go had been widely considered a game requiring uniquely human intuition, creativity, and pattern recognition — qualities too subtle and contextual for AI to replicate. AlphaGo's victory shook the Go community. After Move 37 in Game 2 — a move described by commentators as "beautiful" and as something "no human would play" — Lee Sedol reportedly left the room for fifteen minutes. After the match, Sedol announced his retirement in 2019, saying he could never be at the top again "because there is an entity that cannot be defeated." The psychological impact of AI superiority on human identity in a domain is documented and significant.
Phenomenological distinctiveness: What makes humans unique is not what they can do but what they experience — consciousness, qualia, embodied feeling. On this view, even if AI can perfectly simulate human outputs, the inner life of experience remains inaccessible to machines. This is the position associated with philosophers David Chalmers and Thomas Nagel.
Relational distinctiveness: Human identity is constituted through genuine reciprocal relationships, vulnerability, and mortality — dimensions that AI cannot share. We are who we are because we can lose each other, because we are finite, because our choices have irreversible consequences. AI systems that simulate these dimensions without actually possessing them are engaged in a sophisticated mimicry that does not constitute the real thing.
Pragmatic co-evolution: The question of what makes humans distinct is less important than how humans and AI systems develop together. Human identity is not fixed; it has always been shaped by the technologies humans create. Writing, clocks, cameras, and computers each restructured human cognition and identity. AI is the next such restructuring — and the appropriate response is to guide that restructuring rather than resist it.
How we answer the question of human distinctiveness is not merely philosophical — it determines what rights AI systems should have, what obligations AI developers have toward users, and how AI should be deployed in contexts involving vulnerable human identity (grief, loneliness, adolescent development). If human consciousness is the core of what matters morally, then AI systems that simulate it without possessing it carry ethical risks of deception. If co-evolution is the frame, then the ethics lies in managing the transition well — ensuring that AI augments rather than diminishes the conditions under which humans can live fully.
Drawing on the three frameworks for human distinctiveness discussed in Lesson 4, engage the assistant on what you think should guide AI development in light of these questions.