In November 2017, former Google design ethicist Tristan Harris testified before a Senate subcommittee, describing how notification timing, infinite scroll, and variable-reward feeds were engineered specifically to exploit psychological vulnerabilities rather than serve users. Harris had circulated an internal presentation at Google in 2013 titled "A Call to Minimize Distraction & Respect Users' Attention." He received supportive replies — and essentially nothing changed. His subsequent co-founding of the Center for Humane Technology marked the moment the phrase "persuasive technology" entered mainstream regulatory vocabulary.
Recommendation algorithms deployed by YouTube, Facebook, and TikTok were trained primarily on engagement metrics: watch time, likes, shares, and return visits. These proxies for user interest are cheap to measure, plentiful, and highly optimizable. The problem is that they are not the same as user welfare.
A 2019 internal Facebook study, later cited in Frances Haugen's 2021 whistleblower disclosures, found that 64% of people who joined extremist groups on Facebook did so because the algorithm directly recommended it. The recommendation engine was optimizing for the metric it was given — engagement — and found that outrage and tribalism were reliably engaging.
YouTube's own researchers published findings in 2019 showing that the recommendation system steered politically neutral searchers toward progressively more extreme content within a few clicks. The pathway existed not because anyone designed it to radicalize users, but because more extreme content attracted longer watch sessions, and the system learned that pattern faithfully.
Engagement metrics are a proxy for value. Proxies can be gamed by the optimization process itself — producing behavior that scores well on the metric while undermining the underlying goal. This is sometimes called Goodhart's Law: when a measure becomes a target, it ceases to be a good measure.
Three well-documented psychological mechanisms are particularly relevant to attention-capture design:
Variable reward schedules — pioneered by B.F. Skinner and extensively studied in behavioral psychology — produce compulsive checking behavior when rewards (likes, new content, social validation) arrive unpredictably. The smartphone notification pull-to-refresh gesture was explicitly compared to a slot machine by its designer, Aza Raskin, in a 2018 interview where he expressed regret for the invention.
Social comparison is reliably activating. Seeing curated highlight reels of peers triggers evaluation anxiety. A landmark 2018 University of Pennsylvania experiment (Allcott et al.) randomly assigned students to limit Facebook, Instagram, and Snapchat to 10 minutes each per day. After four weeks, the restricted group showed significantly lower depression and loneliness scores compared to the control group.
Fear of missing out (FOMO) creates an urgency bias that keeps users returning to feeds even when they report not wanting to. The asymmetry is structural: the platform benefits from every return visit whether or not the user benefits.
In 2018, Facebook announced it would de-prioritize passive content consumption in favor of "meaningful social interactions" following public pressure and declining teen usage data. The change reduced overall time-on-site but, according to internal documents released by Haugen, also increased the spread of misinformation because heated comment threads constituted "meaningful interactions" by the algorithm's measurement.
YouTube introduced a "Take a break" reminder feature in 2018 and began reducing recommendation of borderline content in 2019, claiming the latter change reduced consumption of borderline videos by 70% among users who would have otherwise watched them. Researchers noted the change was difficult to verify externally because the company controlled all relevant data.
These early efforts share a pattern: reactive, incremental, and implemented after significant public and regulatory pressure. They also revealed how genuinely difficult it is to operationalize "wellbeing" as an optimization target when the entire business model depends on time-on-platform.
In this lab you'll interrogate how platforms set optimization targets and what goes wrong when proxies diverge from underlying goals. Discuss the concepts with the AI tutor — cite specific cases and mechanisms from the lesson.
Complete at least 3 exchanges to mark the lab complete.
In April 2019, the Norwegian Consumer Council published Deceived by Design, a 44-page report documenting how Google, Facebook, and Windows 10 used interface design to steer users toward privacy-invasive settings. The report described a privacy settings screen where the "Share data" button was large, brightly colored, and in the primary reading position, while "Don't share" required multiple additional clicks through pages designed to provoke anxiety. This was not a bug. It was the intentional architecture of a decision.
In 2023, the Federal Trade Commission's Bringing Dark Patterns to Light report catalogued similar practices across subscription services, e-commerce, and social media. The FTC found that companies using dark patterns in subscription cancellation flows retained 20–40% more subscribers than those offering straightforward cancellation — confirming that these designs worked exactly as intended, against user intent.
UX researcher Harry Brignull coined the term "dark patterns" in 2010 to describe interface designs that trick users into unintended actions. The taxonomy he developed — later expanded by academic researchers including Mathur et al. (2019) — includes:
Confirmshaming — labeling the "decline" option with guilt-inducing text. ("No thanks, I don't want to save money.")
Roach motel — making it easy to enter a state (subscribe, create account) and deliberately difficult to exit. Amazon's Prime cancellation flow, before a 2022 FTC complaint prompted redesign, required navigating through six screens and resisting multiple retention offers.
Hidden costs — concealing fees until final checkout, exploiting the sunk-cost psychology that makes users less likely to abandon a purchase after investing effort in the process.
Forced continuity — charging users automatically after free trials without prominent reminders, relying on inertia and the hassle of cancellation.
Privacy zuckering (named for Mark Zuckerberg) — tricking users into sharing more data than they intend through confusing or misleading consent flows.
In 2022, the EU's Digital Services Act introduced explicit prohibitions on dark patterns, defining them as "practices that materially distort or impair the ability of recipients of the service to make free and informed decisions." The Irish Data Protection Commission fined Meta €390 million in January 2023 partly for consent interfaces designed to channel users toward data-sharing without genuine informed choice.
Richard Thaler and Cass Sunstein's 2008 book Nudge introduced the concept of libertarian paternalism: designing choice environments (choice architectures) that steer people toward better outcomes while preserving freedom to opt out. The key distinction from dark patterns is transparency and alignment with user interests.
The UK's Behavioural Insights Team applied this to organ donation in 2013: changing England's default from opt-in to opt-out (presumed consent) increased registered donors significantly. The default was changed explicitly to save lives — a goal most people endorsed — and the change was publicly disclosed and subject to democratic debate.
In the AI context, Apple's Screen Time feature (launched 2018) lets users set limits on their own app usage — a form of self-nudging. The feature doesn't prevent use; it creates friction and a moment of reflection. Early research suggested it reduced usage among users who set voluntary limits, though motivated users easily bypassed it.
Machine learning has made dark patterns more precise and more personalized. Rather than a single dark pattern applied uniformly, AI systems can test thousands of variants and identify which manipulation works best on each individual user based on their behavioral profile.
A 2022 Princeton study of 11,000 shopping websites found dark patterns on 1,818 of them — and that e-commerce sites using AI-driven A/B testing deployed significantly more dark patterns than those without such testing infrastructure. The optimization process, left to run without ethical constraints, converges on manipulation because manipulation converts.
The 2023 FTC report specifically flagged AI-driven "personalized" retention offers in subscription cancellation flows — where the system learns that offering a 30% discount retains price-sensitive users while emotional appeals ("You'll miss your friends!") work better on socially-motivated users. The manipulation is tailored, making it harder to recognize and resist.
You'll examine real interface scenarios and apply dark pattern taxonomy to identify what's happening — then propose redesigns that preserve business viability while respecting user autonomy. Think about the choice architecture principles from the lesson.
Complete at least 3 exchanges to mark the lab complete.
In September 2021, Frances Haugen provided the Wall Street Journal with internal Facebook research showing that Instagram worsened body image for one in three teenage girls who used it regularly. The documents, known as the Facebook Papers, included a slide from a 2019 internal presentation: "We make body image issues worse for one in three teen girls." The research had been completed. The finding had been shared internally. No product change followed for two years. Instagram's algorithm had been reliably surfacing body-image content to users who engaged with it — because they engaged with it.
In 2023, the Wall Street Journal published an investigation into TikTok's recommendation algorithm, creating test accounts that signaled depression-related interests through viewing behavior. Within hours, the accounts were receiving near-continuous feeds of content about hopelessness, self-harm, and suicide — each view generating the next recommendation, each recommendation deepening the spiral. TikTok stated its policies prohibited such content; the investigation found the algorithm surfaced it anyway.
Recommendation algorithms create self-reinforcing content loops. A user who watches one video about anxiety tends to receive more anxiety content, because their engagement with it signals interest. This is efficient from an engagement standpoint. From a mental health standpoint, it can trap vulnerable users in what researchers call rabbit holes of distressing content.
A 2020 study published in JAMA Internal Medicine analyzed Pinterest's recommendation system and found that searches for eating disorder terms reliably generated further pro-eating-disorder content — "thinspo" images, extreme calorie restriction tips, and community spaces organized around disordered eating. Pinterest implemented a search intervention in 2012 after pressure from eating disorder advocacy groups; the 2020 study found the problem had re-emerged as the algorithm updated.
The mechanism is not malice — it is gradient descent finding the path of least resistance to engagement, regardless of whether the content is harmful. The system has no model of user wellbeing; it has a model of what users click.
UK Online Safety Act (2023) introduced a formal "duty of care" for platforms with significant reach, requiring them to assess and mitigate risks to user mental health, with specific provisions for child users. The law explicitly holds that algorithmic amplification of harmful content can constitute a breach of this duty even if the platform did not create the content.
The field of suicide prevention developed "safe messaging guidelines" — evidence-based protocols about how to discuss self-harm in media without triggering contagion effects. These guidelines recommend against detailed method descriptions, avoid presenting suicide as a solution, and emphasize connection and help-seeking. They were developed for journalists and broadcasters, but AI systems now need to implement them at scale.
In 2016, Facebook deployed an AI-assisted suicide prevention system that flagged posts and comments expressing suicidal ideation, routing them to human reviewers who could trigger wellness checks. By 2019, Facebook reported the system had led to more than 3,500 wellness checks globally. The system's precision and recall remained proprietary, making external evaluation impossible — a pattern critics noted was itself a wellbeing design problem.
Crisis Text Line, a U.S. mental health text service, partnered with data scientists to build predictive models from anonymized conversation data. In 2022, controversy erupted when it emerged that the organization had been licensing insights from crisis conversations to a for-profit subsidiary (Loris.ai). The data had been stripped of personal identifiers, but the source — people in acute distress — raised serious questions about the ethics of monetizing vulnerability even in aggregate form.
Several design principles have emerged from research and regulatory pressure specifically for systems likely to interact with psychologically vulnerable users:
Content velocity limits — deliberately slowing the rate at which emotionally activating content can be consumed in sequence. TikTok announced "content diversity" interventions in 2023 that would interrupt loops of sad or anxiety-related content with neutral or positive content after extended sequences.
Transparent algorithmic explanation — telling users why they are seeing content, giving them meaningful control. YouTube's "Why am I seeing this?" feature, while limited, represents movement toward algorithmic transparency.
Differential treatment for minors — several platforms including TikTok and Instagram have implemented age-differentiated content policies that restrict certain categories for under-18 accounts. Researchers have noted these are largely honor-system-based in the absence of robust age verification.
Off-ramp design — deliberately placing pathways out of distressing content loops (crisis hotline links, breathing exercises, suggestions to contact a friend) within the content experience rather than behind separate menu structures.
You're advising on the design of an AI recommendation system for a mental health information app. The app serves general users but knows that a significant portion are experiencing anxiety or depression. Explore design decisions with the tutor, drawing on safe messaging guidelines and the duty of care framework.
Complete at least 3 exchanges to mark the lab complete.
In 2019, the UK government commissioned the Cairncross Review on a sustainable future for journalism. Buried in its recommendations was an early articulation of what would become a major policy direction: that algorithmic curation systems should be designed to support "quality information" rather than pure engagement. This framing — designing AI for epistemic outcomes, not just behavioral ones — began appearing in academic literature and corporate AI ethics frameworks through 2020–2023, culminating in the EU AI Act's Annex III listing of "AI systems intended to influence elections or voting behavior" as high-risk, with specific wellbeing-oriented obligations.
Meanwhile, DeepMind published research in 2022 exploring what it called agent incentives — the mathematical conditions under which an AI agent would or would not develop instrumental goals that conflict with human wellbeing. The paper identified corrigibility (the willingness to be corrected) and avoidance of side effects as key properties for wellbeing-safe agents. The research was theoretical, but it represented a shift from "don't harm" toward "be designed to flourish alongside humans."
The fundamental challenge of wellbeing-by-design is measurement. Three broad approaches have been tested:
Subjective wellbeing surveys — asking users how they feel after using a product. Google's YouTube team conducted a 2020 survey experiment measuring "satisfaction" and "regret" after video sessions, finding these correlated differently with watch time depending on content type. Documentaries and educational content generated high satisfaction at medium watch times; passive entertainment generated regret at high watch times. The researchers proposed a "satisfaction per minute" metric, though it has not been publicly adopted as a primary optimization target.
Longitudinal behavioral proxies — tracking whether users return to a service because they want to or because they feel compelled to. Twitter's Responsible ML team published a 2022 paper distinguishing "pull" from "push" engagement — users actively seeking content versus responding to notifications. They found notification-driven engagement correlated with lower subsequent mood reports, while search-driven engagement correlated with satisfaction.
Deliberative preference — asking users what they would want if they reflected on it, rather than what they do in the moment. Spotify's "2023 Listening Habits" research found a significant gap between what users listened to when browsing (familiar, comfortable content) and what they reported wanting to explore (new artists, challenging genres). Wellbeing-oriented design would close this gap rather than exploit it.
UK innovation foundation Nesta published a digital wellbeing framework in 2020 identifying five dimensions: agency (users control their experience), connection (quality social interaction), engagement (purposeful rather than compulsive use), knowledge (access to accurate information), and emotional outcomes (net positive emotional state). The framework provides a multi-dimensional alternative to engagement as a target metric.
The Center for Humane Technology, co-founded by Tristan Harris in 2018, developed a "Ledger of Harms" — a public documentation of evidence linking technology design choices to specific human costs including addiction, political polarization, and teen mental health decline. While primarily advocacy, the Ledger created shared vocabulary for design teams and regulators and has been cited in congressional testimony multiple times since 2019.
In 2021, the Partnership on AI published "Responsible Publication Norms for Machine Learning" which included a wellbeing section requiring researchers to assess potential psychological harms before publishing systems that interact directly with users. The norms are voluntary but represent the emerging professional consensus in the research community.
Mozilla's 2022 "YouTube Regrets" project crowd-sourced evidence of harmful recommendations from 37,380 volunteers across 91 countries, creating external accountability for recommendation quality that didn't depend on platform self-reporting. The project found 71% of regretted recommendations were of content that violated YouTube's own policies — suggesting a gap between policy and algorithmic reality.
Synthesizing research, regulatory guidance, and practitioner experience, several principles have emerged for AI systems designed with wellbeing as a genuine objective:
Align short-term and long-term interests — users often want different things in the moment than they want across a lifetime. Wellbeing design respects both, creating moments for reflective choice rather than always defaulting to immediate impulse.
Make the system legible — users cannot make genuine choices about their AI interactions if they don't understand the system shaping those interactions. Explainability isn't just a technical property; it's a precondition for autonomy.
Design exit pathways — healthy AI relationships must include genuine ability to disengage, take breaks, or reduce dependency without friction or manipulation.
Measure what matters, not what's easy — the wellbeing metrics most worth optimizing for (life satisfaction, cognitive flourishing, quality social connection) are hard to measure. The solution is to measure them harder, not to substitute easier proxies.
Build accountability structures — wellbeing by design requires that someone is responsible for outcomes beyond engagement. This means organizational structure changes, not just design guidelines. Teams with power to trade engagement for wellbeing outcomes need authority to do so.
You're a wellbeing design consultant advising a major social platform's product team. Drawing on all four lessons — engagement economics, dark patterns, mental health risks, and wellbeing frameworks — you'll help develop a practical design brief. The AI tutor will challenge your recommendations and push you to defend them with evidence from the module.
Complete at least 3 exchanges to mark the lab complete.