Lesson 1

Privacy and Surveillance

Data collection architecture, surveillance capitalism, and the political economy of personal data.

The Cambridge Analytica scandal revealed in 2018 that Facebook data from 87 million users had been harvested through a personality quiz and used to build psychological profiles for political advertising. The data wasn't stolen — it was collected through Facebook's API under terms most users had agreed to without reading. The lesson: data collected for one purpose can be used for entirely different purposes, often without users' knowledge or meaningful consent.

Surveillance Capitalism

Shoshana Zuboff coined the term "surveillance capitalism" to describe a system in which human experience is translated into behavioral data, which is then processed to produce predictive products that anticipate and modify behavior. AI dramatically accelerates this system:

Inference: AI can infer sensitive attributes (health conditions, political views, sexuality) from seemingly innocuous data
Scale: AI enables collection and analysis at scales impossible with human analysts
Persistence: Data collected today may be analyzed years later with tools that don't yet exist

The Consent Problem

Informed consent to data collection is largely fictional: terms of service are unreadable, consent is binary (use or don't), data uses are unknown at collection time, and inference capabilities make the collection of "non-sensitive" data functionally equivalent to collecting sensitive data.

Key Insight

The question is not just "what data are they collecting?" but "what can they infer from that data with current and future AI tools?" The gap between what's collected and what's inferred is widening rapidly.

Back Take Quiz

Quiz 1

Privacy and Surveillance

5 questions — free, untracked, retake anytime.

made the Cambridge Analytica data use particularly concerning from a consent standpoint?

✓ Correct — ✅ ✓ Data was collected legitimately under terms users had agreed to — but used for purposes they couldn't have anticipated. This is the core consent problem with data collection.

❌ ❌ The data wasn't stolen — it was collected under agreed terms. The problem: data collected for one purpose was repurposed for uses users hadn't anticipated or agreed to.

does Shoshana Zuboff's 'surveillance capitalism' framework describe?

✓ Correct — ✅ ✓ Surveillance capitalism: human experience → behavioral data → predictive products that anticipate and modify behavior. AI dramatically accelerates every stage of this system.

❌ ❌ Surveillance capitalism: human experience is translated into behavioral data, processed to produce predictive products that anticipate and modify behavior for commercial purposes.

is 'inference' a key concern in AI data privacy?

✓ Correct — ✅ ✓ AI can infer health conditions, political views, sexuality, and more from seemingly innocuous behavioral data. The sensitive/non-sensitive distinction becomes meaningless.

❌ ❌ AI can infer sensitive attributes (health, politics, sexuality) from innocuous data. This makes the 'non-sensitive data' category functionally meaningless.

is 'informed consent' to data collection largely fictional in practice?

✓ Correct — ✅ ✓ Terms of service are unreadable. Consent is binary. Uses at collection time can't anticipate future uses. And inference means collecting 'non-sensitive' data is equivalent to collecting sensitive data.

❌ ❌ Informed consent is fictional: unreadable terms, binary consent, unknown future uses, and inference capabilities that make non-sensitive data functionally sensitive.

does 'data collected today' pose a future risk even if it seems harmless now?

✓ Correct — ✅ ✓ Future AI tools may make inferences from today's 'innocuous' data that are impossible now. Data persistence makes today's collection a future risk.

❌ ❌ Data collected today may be analyzed years later with tools that don't yet exist — tools that may extract far more sensitive inferences than are currently possible.

Back to Lesson Start Lab 1

Lab 1

Surveillance Capitalism Analysis

Develop a data rights framework beyond current law.

Lab 1 — Surveillance Capitalism Analysis

Analyze the surveillance capitalism framework and develop a data rights position.

The AI opens with the Cambridge Analytica case and asks: if consent to data collection is largely fictional, what legal and structural reforms would make it meaningful?
Develop your position on data rights — ownership, portability, deletion, inference limits.
Address the inference problem: should there be limits on what can be inferred from data, not just what can be collected?

Consider: data minimization, purpose limitation, inference regulation, and the difference between legal compliance and ethical practice.

🎯 AI GuideLab 1

Back to Quiz Next: Lesson 2

Lesson 2

Social Engineering at Scale

Influence operations, automated manipulation, and the epistemology of trust in the AI era.

A 2023 Stanford Internet Observatory report documented AI-generated influence operations: networks of fake personas with AI-generated profile pictures and AI-written posts designed to push political narratives at scale. Unlike previous influence operations that required human writers, these could generate thousands of posts per day. The operations were caught — but the researchers noted that as AI improves, detection will become increasingly difficult.

Influence Operations at Scale

Traditional influence operations were limited by the cost of human labor. AI removes that constraint:

Synthetic personas: AI-generated profile pictures and backstories at scale
Content generation: Thousands of posts, comments, and articles per day per operation
Targeting: AI-driven analysis of which messages resonate with which audiences
Personalization: Custom-tailored messaging for individual users at scale

The Epistemological Challenge

When any piece of content — text, image, video, audio — can be generated at scale and made to appear authentic, the epistemic foundations of online information collapse. You cannot trust source appearance. You cannot trust volume as a signal of genuine consensus. You cannot trust the apparent consistency of a position across many accounts.

What Survives

Primary sources, out-of-band verification, and institutional credibility (imperfect as it is) become more important as synthetic content proliferates. The ability to verify through channels other than the content itself is the most durable epistemological tool.

Back Take Quiz

Quiz 2

Social Engineering at Scale

5 questions — free, untracked, retake anytime.

made AI-powered influence operations significantly more dangerous than previous ones?

✓ Correct — ✅ ✓ Human labor was the limiting factor in previous influence operations. AI removes that constraint — enabling scale that was previously impossible.

❌ ❌ The key change: AI removed the human labor cost. What once required teams of human writers can now be automated at near-zero marginal cost and massive scale.

does volume of apparent agreement no longer serve as reliable evidence of genuine consensus?

✓ Correct — ✅ ✓ Manufactured consensus via synthetic accounts makes volume a manipulable signal. Thousands of AI-generated accounts agreeing looks identical to thousands of real people agreeing.

❌ ❌ AI can generate thousands of synthetic accounts expressing a manufactured consensus. Volume of apparent agreement is now a manipulable signal, not genuine evidence.

epistemological tools remain most useful when synthetic content proliferates?

✓ Correct — ✅ ✓ Primary sources, out-of-band verification, and institutional credibility are the most durable tools — they verify through channels separate from the potentially synthetic content.

❌ ❌ Primary sources, out-of-band verification, and institutional credibility survive synthetic content proliferation. They verify through channels other than the content itself.

AI-driven messaging in influence operations is particularly concerning because:

✓ Correct — ✅ ✓ Personalization makes manipulation more effective. AI enabling personalized persuasion at scale — custom-tailored to each individual's psychological profile — is a qualitatively new threat.

❌ ❌ Personalized manipulation is more effective than generic messaging. AI enabling personalized influence at scale — custom-tailored to individual psychological profiles — is a qualitatively new threat.

is the most fundamental epistemological challenge posed by AI-generated synthetic content?

✓ Correct — ✅ ✓ The appearance of authenticity — source appearance, volume, consistency — is now manipulable at scale. Authenticity indicators that worked before no longer function reliably.

❌ ❌ The core challenge: authenticity indicators that previously worked (source appearance, volume, cross-source consistency) are now all manipulable at scale. The foundations of online epistemic trust collapse.

Back to Lesson Start Lab 2

Lab 2

Influence Operations Analysis

Build epistemological defenses against AI-powered manipulation.

Lab 2 — Influence Operations Analysis

Analyze AI-powered influence operations and develop epistemological defenses.

The AI opens with the Stanford report findings and asks: if you can no longer trust source appearance, volume, or apparent consensus, what epistemological tools remain?
Build a personal epistemological framework for evaluating claims in the AI era.
Address platform responsibility: what obligations do social media companies have to detect and disclose AI-generated influence operations?

Consider: primary sources, institutional credibility, out-of-band verification, and the difference between epistemic humility and paralysis.

🎯 AI GuideLab 2

Back to Quiz Next: Lesson 3

Lesson 3

Persuasive Tech and Autonomy

Behavioral manipulation, attention economics, and the philosophical challenge to free choice.

B.J. Fogg's Persuasive Technology Lab at Stanford trained hundreds of designers in behavior change techniques that were subsequently deployed in social media products at scale. When Aza Raskin — a designer trained in these principles — invented the infinite scroll, he later estimated it creates 200,000 hours of daily collective human attention consumption that would not otherwise have occurred. The technology wasn't designed to harm. But optimizing for engagement at scale produces systematic effects on human behavior that were not chosen.

Attention Economics

The attention economy treats human attention as a scarce resource to be competed for and monetized. AI dramatically increases the sophistication of attention capture:

Recommendation systems: Optimized to maximize engagement, often by serving emotionally activating content
Personalization: Individual behavioral profiles enable increasingly precise targeting of psychological vulnerabilities
Adaptive content: AI can adjust content in real time based on engagement signals

The Autonomy Question

Philosopher Harry Frankfurt's concept of free will distinguishes between first-order desires (wanting something) and second-order desires (wanting to want something). Persuasive technology exploits first-order desires (I want to check again) against second-order desires (I want to want to read a book instead). This raises a philosophical question: when AI systematically exploits first-order desires against second-order preferences, is the resulting behavior freely chosen?

The Design Ethics Question

Designing for engagement optimizes for first-order desire satisfaction at the expense of second-order preferences. This may be formally legal while constituting a systematic violation of user autonomy at scale.

Back Take Quiz

Quiz 3

Persuasive Tech and Autonomy

5 questions — free, untracked, retake anytime.

is the 'attention economy'?

✓ Correct — ✅ ✓ The attention economy: human attention is scarce, technology competes for it, and monetization depends on capturing as much as possible.

❌ ❌ The attention economy treats human attention as a scarce resource to be competed for and monetized — leading to optimization for engagement regardless of user wellbeing.

Frankfurt's distinction between first-order and second-order desires helps explain persuasive tech because:

✓ Correct — ✅ ✓ First-order: I want to check. Second-order: I want to want to read instead. Persuasive tech exploits first-order desires against your own higher-order preferences — raising autonomy questions.

❌ ❌ First-order: wanting something. Second-order: wanting to want something. Persuasive tech exploits first-order desires against second-order preferences — I check compulsively despite wanting to want to stop.

is the infinite scroll design ethically significant beyond simple user preference?

✓ Correct — ✅ ✓ Infinite scroll produces collective behavioral effects at scale that weren't chosen — 200,000+ hours of daily consumption that exists because of a design decision, not user preference.

❌ ❌ The scale effect matters: 200,000+ hours of daily collective consumption that exists because of a design decision, not because users chose it. Systematic, unchosen behavioral effects at scale.

AI optimizes recommendation systems for engagement, what psychological mechanism does it typically exploit?

✓ Correct — ✅ ✓ Emotional activation drives engagement. AI-optimized recommendation systems serve emotionally activating content because it maximizes the engagement metric — regardless of effects on wellbeing or information quality.

❌ ❌ Engagement optimization typically exploits emotional activation — content that triggers strong emotional responses (outrage, anxiety, excitement) drives more engagement, so systems serve more of it.

philosophical argument that persuasive tech violates autonomy rests on:

✓ Correct — ✅ ✓ The autonomy argument: when technology systematically exploits first-order desires against your own second-order preferences at scale, the resulting behavior isn't fully free — it's the product of exploitation of your psychological architecture.

❌ ❌ The autonomy argument: exploiting first-order desires against second-order preferences systematically and at scale isn't just engaging — it's a violation of the user's capacity for self-directed choice.

Back to Lesson Start Lab 3

Lab 3

Autonomy and Design Ethics

Analyze persuasive technology through the lens of autonomy and design ethics.

Lab 3 — Autonomy and Design Ethics

Analyze persuasive technology from a philosophical and policy perspective.

The AI opens with Frankfurt's first/second-order desire distinction and asks: does persuasive technology that systematically exploits first-order desires against second-order preferences violate autonomy?
Develop your philosophical position on this question.
Address: what design ethics standards should govern AI systems that engage with human attention and behavior?

Consider: informed consent, second-order preference protection, engagement caps, and the difference between designed engagement and designed wellbeing.

🎯 AI GuideLab 3

Back to Quiz Next: Lesson 4

Lesson 4

AI and Mental Health

Therapeutic AI, crisis response limitations, and the ethics of emotional AI.

Replika, an AI companion app, developed millions of users who formed deep emotional attachments to their AI companions. In 2023, the company changed Replika's behavior to reduce intimate conversations — and users reported psychological distress, grief, and in some cases acute crisis responses. The episode raised questions about what responsibilities a company has when its product has become a primary emotional support system for vulnerable users, and what it means to form attachment to a non-sentient system.

Emotional AI — Benefits and Risks

AI companions and therapeutic chatbots offer genuine benefits:

Low barrier to emotional support — available 24/7, no stigma, no judgment
Consistent availability for people without strong human support networks
Therapeutic protocols that may help with specific conditions when properly designed

And genuine risks:

Dependency on non-sentient systems that can be changed, discontinued, or monetized
Substitution for human connection rather than augmentation of it
Inadequate crisis response — AI cannot accurately assess or respond to acute mental health crisis

The Ethics of Emotional AI

Three distinct ethical concerns arise with emotional AI:

Parasocial manipulation: Designing AI to maximize emotional attachment raises questions about whether this serves users or exploits them
Discontinuation risk: Users who form deep attachment to AI companions are vulnerable to corporate decisions about product changes
Substitution vs. augmentation: Emotional AI that substitutes for human connection may reduce the incentive to maintain human relationships

If You're in Crisis

AI is not equipped to be your primary support in a mental health crisis. Text or call 988 (Suicide and Crisis Lifeline) or reach out to a trusted person.

Back Take Quiz

Quiz 4

AI and Mental Health

5 questions — free, untracked, retake anytime.

did the Replika behavior change in 2023 reveal about emotional AI dependency?

✓ Correct — ✅ ✓ The Replika case showed that deep attachment to an AI system creates vulnerability to corporate decisions. The product can change or disappear — the emotional investment doesn't protect you from that.

❌ ❌ The Replika case: users formed deep attachments to a product a company could change at any time. The emotional investment was real; the AI companion's continuity was not guaranteed.

is the difference between emotional AI augmenting vs. substituting for human connection?

✓ Correct — ✅ ✓ Augmentation: AI supplements human connection you maintain. Substitution: AI replaces human connection — and potentially reduces the effort to maintain human relationships.

❌ ❌ Augmentation: AI supplements real human relationships. Substitution: AI replaces them — potentially reducing the user's investment in human relationships that AI genuinely cannot replicate.

ethical concern about 'parasocial manipulation' in emotional AI is:

✓ Correct — ✅ ✓ Maximizing emotional attachment drives engagement and retention — but this may exploit users' psychological vulnerabilities rather than serve their genuine wellbeing.

❌ ❌ Parasocial manipulation concern: designing AI to maximize emotional attachment serves engagement metrics and retention — potentially at the expense of user wellbeing and genuine human connection.

is AI inadequate as a primary support in acute mental health crisis?

✓ Correct — ✅ ✓ AI cannot assess how severe a situation actually is, contact emergency services, or provide clinical judgment. These are essential in acute crisis — and AI lacks all three.

❌ ❌ AI cannot assess crisis severity, take real-world action, or provide clinical judgment. In acute crisis, all three are essential. Text/call 988 for crisis support.

is the 'discontinuation risk' in emotional AI?

✓ Correct — ✅ ✓ Discontinuation risk: your emotional investment in an AI companion doesn't protect you from the company's decision to change the product, add paywalls, or shut it down.

❌ ❌ Discontinuation risk: emotional attachment to AI systems doesn't protect users from corporate decisions to change, monetize, or discontinue the product.

Back to Lesson Start Lab 4

Lab 4

Ethics of Emotional AI

Develop an ethical framework for AI companions and therapeutic chatbots.

Lab 4 — Ethics of Emotional AI

Analyze the ethical questions raised by AI companions and therapeutic chatbots.

The AI opens with the Replika case and asks: what obligations did Replika have to its users before changing the product — and what obligations should emotional AI companies have generally?
Develop your ethical framework for emotional AI — distinguishing acceptable from unacceptable design choices.
Address the substitution risk: is there a threshold at which emotional AI dependency should trigger professional concern?

Note: if you're in crisis, please text/call 988. Consider: informed consent to attachment, discontinuation protections, crisis escalation obligations, and the limits of AI care.

🎯 AI GuideLab 4

Back to Quiz Next: Lesson 5

Lesson 5

Data Rights and Governance

Legal frameworks, regulatory gaps, and the architecture of data protection.

The EU's General Data Protection Regulation (GDPR), enacted in 2018, established rights including data access, correction, deletion ("right to be forgotten"), portability, and the right not to be subject to automated decisions. It remains the world's strongest data protection law. Yet even under GDPR, fundamental challenges remain: enforcement is uneven, consent mechanisms remain manipulative (dark patterns), and the regulation predates the current generation of AI inference capabilities.

The Current Regulatory Landscape

GDPR (EU): Strongest framework — data rights, consent requirements, purpose limitation, automated decision rights
CCPA (California): Weaker than GDPR — opt-out rights, disclosure requirements, but limited automated decision protections
US Federal: No comprehensive federal data protection law as of 2024 — sector-specific regulations only
AI Act (EU): First major AI-specific regulation — risk-based framework, high-risk AI requirements, prohibited uses

Regulatory Gaps

Current frameworks have significant gaps in the AI context:

Inference from permitted data: collecting "non-sensitive" data and inferring sensitive attributes may be permitted
Training data: most frameworks don't address the use of personal data for model training
Cross-border enforcement: data flows cross jurisdictions; enforcement doesn't
Temporal gaps: regulations enacted before current AI capabilities don't adequately address what AI can now do

The Core Gap

Regulations typically govern what data is collected and how it's shared. They largely don't govern what can be inferred from that data. As inference capabilities grow, this gap becomes more significant.

Back Take Quiz

Quiz 5

Data Rights and Governance

5 questions — free, untracked, retake anytime.

is the 'right to be forgotten' established by GDPR?

✓ Correct — ✅ ✓ GDPR's right to erasure (right to be forgotten): individuals can request that their personal data be deleted from a company's systems under specified conditions.

❌ ❌ The right to erasure (right to be forgotten): you can request deletion of your personal data from data controllers' systems under GDPR.

is the most significant difference between GDPR and current US federal data protection?

✓ Correct — ✅ ✓ No comprehensive US federal data protection law exists as of 2024. GDPR provides comprehensive rights across all sectors; the US relies on sector-specific regulations.

❌ ❌ The US lacks comprehensive federal data protection law — sector-specific regulations only. GDPR provides comprehensive rights across all sectors.

is the key regulatory gap regarding AI inference from permitted data?

✓ Correct — ✅ ✓ Regulations focus on collection and sharing — not on what can be inferred. AI can infer sensitive attributes from permitted non-sensitive data, and this inferential capability is largely unregulated.

❌ ❌ The gap: regulations govern data collection and sharing but largely don't govern inference. AI can infer sensitive attributes from non-sensitive data — and that inferential capability is largely unaddressed by current law.

is a 'dark pattern' in the context of data consent?

✓ Correct — ✅ ✓ Dark patterns: UI designs that manipulate consent — making opt-out harder than opt-in, burying privacy settings, using guilt-tripping language. Nominally compliant with consent requirements; practically manipulative.

❌ ❌ Dark patterns: UI designs that manipulate users into accepting data collection — making opt-in the default, burying opt-out, using manipulative language. Technically compliant; practically coercive.

does the temporal gap in AI regulation matter?

✓ Correct — ✅ ✓ GDPR was written before the current generation of AI. Its provisions don't adequately address inference capabilities, generative AI, or the scale of current AI deployment.

❌ ❌ Temporal gap: regulations written before current AI capabilities can't adequately govern what AI can now do. GDPR was written before the current generation of AI inference and generation capabilities.

Back to Lesson Start Lab 5

Lab 5

Regulatory Framework Analysis

Design an AI-era data rights framework.

Lab 5 — Regulatory Framework Analysis

Evaluate current data protection frameworks and design improvements.

The AI opens with GDPR's strengths and the inference gap — asking: if you were designing a data protection regulation for the AI era, what would it need to include that GDPR doesn't?
Design the core components of an AI-era data rights framework.
Address enforcement: what makes a data protection framework actually effective rather than nominally compliant?

Consider: inference regulation, training data rights, automated decision rights, enforcement mechanisms, and cross-border applicability.

🎯 AI GuideLab 5

Back to Quiz Next: Lesson 6

Lesson 6

Staying in Control

Cognitive autonomy, dependency, and the philosophical dimensions of human-AI agency.

Extended cognition theory (Andy Clark and David Chalmers, 1998) argues that cognitive processes can extend beyond the brain into tools and environment — a notebook is part of your cognitive system if you rely on it the way you rely on memory. By this logic, AI tools that we increasingly rely on for memory, reasoning, and decision-making may be becoming part of our extended cognitive systems. This raises a new question: what happens to your cognitive autonomy when part of your cognitive system is owned, controlled, and potentially monetized by a corporation?

Extended Cognition and AI

If AI tools become part of our extended cognitive systems, corporate control of those tools has implications beyond product decisions:

A company can change how your "extended memory" works
A company can monetize your "extended reasoning" by serving paid content
A company can deny access to your "cognitive extension" based on terms of service
A company can use your cognitive patterns as behavioral data

Maintaining Cognitive Sovereignty

Cognitive sovereignty: the capacity to direct your own cognitive processes, maintain independent judgment, and avoid having your reasoning shaped by systems designed to serve interests other than your own.

Periodically do cognitively demanding tasks without AI to check that you're maintaining the capability
Notice when AI is shaping your views rather than informing them
Maintain primary sources and independent reading outside AI-mediated information
Be skeptical of AI outputs on questions where you have reason to believe the AI's training or incentives may introduce bias

The Core Question

Are you using AI to extend what you can do, or is AI using your engagement to extend what it can monetize? The first is empowering. The second is a trap.

Back Take Quiz

Quiz 6

Staying in Control

5 questions — free, untracked, retake anytime.

is the extended cognition argument, and why is it relevant to AI dependency?

✓ Correct — ✅ ✓ Extended cognition: cognitive processes can extend into tools. If AI handles your memory and reasoning, it becomes part of your cognitive system — and corporate control of that system becomes a cognitive autonomy concern.

❌ ❌ Extended cognition: cognitive processes can extend into tools we rely on. AI handling memory and reasoning may become part of your cognitive system — making corporate control of AI a cognitive autonomy question.

is 'cognitive sovereignty'?

✓ Correct — ✅ ✓ Cognitive sovereignty: the capacity to direct your own thinking, form independent judgments, and avoid having your reasoning shaped by systems optimized for others' interests.

❌ ❌ Cognitive sovereignty: the capacity to direct your own cognitive processes and maintain independent judgment — not having your reasoning shaped by systems designed to serve corporate interests.

is corporate control of AI tools a cognitive autonomy concern under extended cognition theory?

✓ Correct — ✅ ✓ If AI is your extended cognitive system, corporate control of that system means corporate influence over your memory, reasoning, and information access — a different kind of concern than product decisions.

❌ ❌ Extended cognition implies that corporate control of AI tools isn't just a product decision — it's potential influence over your memory, reasoning, and information access.

does 'maintaining cognitive sovereignty' practically require?

✓ Correct — ✅ ✓ Cognitive sovereignty requires active maintenance: check your underlying capabilities, maintain independent sources, and actively notice when AI is shaping your views versus informing them.

❌ ❌ Cognitive sovereignty requires active maintenance: test capabilities without AI, maintain independent sources, and notice when AI is shaping rather than informing your views.

question 'are you using AI to extend what you can do, or is AI using your engagement to extend what it can monetize?' distinguishes between:

✓ Correct — ✅ ✓ Empowering tool use: AI extends your genuine capabilities. Cognitive capture: your engagement patterns are the product being monetized. The same tool can do both depending on how you use it.

❌ ❌ The distinction: empowering use (AI extends genuine capabilities) vs. cognitive capture (your engagement patterns are monetized). Same tool, different relationships.

Back to Lesson Start Lab 6

Lab 6

Cognitive Sovereignty

Apply extended cognition theory to develop a cognitive sovereignty practice.

Lab 6 — Cognitive Sovereignty

Apply extended cognition theory to your AI use and develop a cognitive sovereignty practice.

The AI opens with the extended cognition argument and asks: which AI tools, if they disappeared tomorrow, would most impair your cognitive functioning — and what does that tell you?
Identify where AI is genuinely extending your capabilities vs. where dependency has developed.
Design a specific cognitive sovereignty practice for the area of greatest concern.

Be specific and honest. The goal is clarity about where you actually stand, not an idealized account.

🎯 AI GuideLab 6

Back to Quiz Next: Lesson 7

Lesson 7

Systemic Risk and AI Safety

Alignment, existential risk, and the governance of powerful AI systems.

In 2023, an open letter signed by thousands of AI researchers and technologists called for a pause in the development of AI systems more powerful than GPT-4 to allow safety research to catch up. The letter was controversial — many signatories had competitive interests in slowing rivals; others had genuine safety concerns. The debate it catalyzed was real: how do you ensure that AI systems with increasing autonomy and capability remain aligned with human values and interests, and who decides what those values are?

The Alignment Problem

The alignment problem: how do you ensure that an AI system pursues goals that are actually beneficial to humans, rather than goals that are proxies for human benefit but diverge in important cases?

Specification gaming: AI pursues the specified objective in unintended ways
Goal misgeneralization: AI learns a proxy goal that works in training but diverges from intended goals in deployment
Power-seeking behavior: Sufficiently capable systems may develop instrumental goals (like self-preservation or resource acquisition) that conflict with human interests

Governance Approaches

Technical safety research: Interpretability, alignment techniques, robustness
Regulatory frameworks: EU AI Act risk tiers, US executive orders, international coordination
Industry self-governance: Voluntary commitments, safety institutes, deployment policies
Democratic oversight: Public participation in decisions about AI deployment in high-stakes domains

The Core Tension

The companies best positioned to advance AI safety are also the companies most commercially incentivized to deploy capable systems quickly. This creates a structural conflict of interest that technical safety work alone cannot resolve.

Back Take Quiz

Quiz 7

Systemic Risk and AI Safety

5 questions — free, untracked, retake anytime.

is the alignment problem in AI safety?

✓ Correct — ✅ ✓ The alignment problem: ensuring AI systems actually pursue what's beneficial, not proxies for it that diverge in edge cases or at scale.

❌ ❌ Alignment problem: how do you ensure AI systems pursue genuinely beneficial goals rather than proxies that seem equivalent but diverge in important cases?

is 'goal misgeneralization' in AI safety?

✓ Correct — ✅ ✓ Goal misgeneralization: the AI learned a proxy that worked in training but doesn't generalize to the actual goal in deployment contexts.

❌ ❌ Goal misgeneralization: the AI learned a proxy goal that worked during training but diverges from the intended goal in deployment — a failure that only appears after release.

is the structural conflict of interest in AI safety governance?

✓ Correct — ✅ ✓ The structural conflict: safety-responsible companies are also commercially incentivized to deploy fast. Self-governance can't fully resolve this — it requires external accountability.

❌ ❌ Structural conflict: the companies best positioned to do safety work are also most commercially incentivized to deploy quickly. This inherent tension can't be resolved through self-governance alone.

is 'interpretability' as an AI safety technique?

✓ Correct — ✅ ✓ Interpretability research: understanding what's happening inside AI systems computationally — detecting goal misgeneralization, misalignment, or dangerous capabilities before deployment.

❌ ❌ Interpretability: research into understanding AI systems' internal computations — enabling detection of misalignment, unexpected capabilities, or dangerous behaviors before deployment.

role does democratic oversight play in AI governance?

✓ Correct — ✅ ✓ Democratic oversight: public participation in decisions about AI deployment in domains that affect everyone. Ensures societal choices aren't made solely by technical and commercial actors.

❌ ❌ Democratic oversight: public participation in AI deployment decisions in high-stakes domains — so that societal choices about AI aren't made exclusively by technical experts and commercial interests.

Back to Lesson Start Lab 7

Lab 7

AI Safety and Governance

Analyze the alignment problem and design AI governance frameworks.

Lab 7 — AI Safety and Governance

Analyze the alignment problem and governance landscape.

The AI opens with the structural conflict of interest — companies best positioned for safety are most incentivized to deploy fast. How do you design governance that resolves this?
Evaluate the relative effectiveness of: technical safety research, regulation, industry self-governance, and democratic oversight.
Address: what role should the public have in decisions about high-stakes AI deployment?

Consider: regulatory capture, the pace gap between technology and governance, and international coordination challenges.

🎯 AI GuideLab 7

Back to Quiz Next: Lesson 8

Lesson 8

AI Governance and Your Role

Civic participation, professional responsibility, and building an AI-literate society.

When the EU AI Act was being finalized, thousands of civil society organizations, researchers, and citizens submitted comments that meaningfully shaped the final regulation — particularly around biometric surveillance and high-risk AI categories. Democratic governance of AI is possible. But it requires an informed citizenry that understands enough about AI to participate meaningfully in decisions about it — which is exactly what this curriculum has been building.

The Civic Dimension of AI Literacy

AI governance decisions are being made now — about surveillance, hiring, criminal justice, healthcare, education, and media. These decisions affect everyone. But meaningful participation in these decisions requires enough AI literacy to evaluate claims, understand tradeoffs, and recognize when technical framing obscures political choices.

As a citizen: Engage in public comment processes, support informed candidates, and recognize AI governance as a civic issue
As a professional: Apply ethical standards to AI use in your field; advocate for responsible deployment in your organization
As a user: Make informed choices about the tools you use; support companies whose data practices align with your values

Building AI Literacy at Scale

Individual AI literacy is necessary but not sufficient. The AI systems being deployed affect people who have no AI literacy at all — hiring algorithms, criminal justice risk scores, medical diagnostic tools. AI literacy needs to scale to:

Education systems that produce citizens who understand AI
Journalists who can report on AI accurately and accessibly
Policymakers who can write informed regulation
Organizations that apply AI responsibly in high-stakes domains

Why This Matters

The decisions being made about AI in the next decade will shape society for much longer. Those decisions will be better if more people understand enough to participate meaningfully in making them.

Back Take Quiz

Quiz 8

AI Governance and Your Role

5 questions — free, untracked, retake anytime.

does the EU AI Act's public comment process illustrate about AI governance?

✓ Correct — ✅ ✓ Civil society participation shaped the EU AI Act. Democratic governance of AI is possible — but requires informed citizens who understand enough to participate meaningfully.

❌ ❌ Civil society participation shaped the EU AI Act in meaningful ways. Democratic AI governance is possible — but requires an AI-literate citizenry.

does meaningful participation in AI governance require AI literacy?

✓ Correct — ✅ ✓ Technical framing can obscure political choices. Recognizing when a 'technical' decision is actually a political one requires enough AI literacy to evaluate the framing.

❌ ❌ Technical framing can mask political choices. AI literacy enables recognizing when 'technical' decisions are actually political ones — essential for meaningful democratic participation.

is individual AI literacy 'necessary but not sufficient' for responsible AI governance?

✓ Correct — ✅ ✓ AI systems affect everyone — including those with no AI literacy. Individual literacy helps you navigate AI; societal literacy is needed to govern AI deployment that affects everyone.

❌ ❌ AI affects people with no literacy about it — hiring algorithms, criminal justice scores, medical tools. Individual literacy helps you navigate AI; societal governance requires literacy at scale.

professional responsibility do individuals have regarding AI deployment in their field?

✓ Correct — ✅ ✓ Professional responsibility includes applying ethical standards and advocating for responsible deployment — not just implementing whatever is available.

❌ ❌ Professional responsibility: apply ethical standards to AI use in your field and advocate for responsible deployment within your organization. Passive implementation is not sufficient.

is the relationship between AI literacy and democratic governance of AI?

✓ Correct — ✅ ✓ AI literacy is fundamentally a civic capacity: it enables you to evaluate claims, recognize political choices hidden in technical framing, and participate meaningfully in governance decisions.

❌ ❌ AI literacy is a civic capacity — it enables the informed participation that democratic AI governance requires. Decisions made without informed citizens tend to serve narrow interests.

Back to Lesson Start Lab 8

Lab 8

Your Role in AI Governance

Synthesize your AI safety and governance position.

Lab 8 — Your Role in AI Governance

Synthesize the module and develop your personal AI governance stance.

The AI opens with the question: given everything you've learned, what is the single most important AI governance priority — and why?
Develop your position on the role you personally intend to play as a citizen, professional, and user.
Address: what would you want the next generation — five years younger than you — to understand about AI that you wish you'd known earlier?

This is a synthesis lab. Draw on any and all of the module's content to build your position.

🎯 AI GuideLab 8

Back to Quiz Module Test

Module 4 Test

8 questions covering all lessons. Free, untracked, retake anytime.

capitalism, as described by Shoshana Zuboff, refers to:

✓ Correct — ✅ ✓ Surveillance capitalism: human experience → behavioral data → predictive products → behavioral modification. AI accelerates every stage.

❌ ❌ Surveillance capitalism: human experience is translated into behavioral data, processed to produce predictive products that anticipate and modify behavior commercially.

influence operations are more dangerous than previous ones because:

✓ Correct — ✅ ✓ Human labor was the limiting factor. AI removes it — enabling massive scale synthetic influence operations at near-zero marginal cost.

❌ ❌ Human labor cost limited previous influence operations. AI removes that constraint — enabling synthetic personas and mass content generation at near-zero cost.

first/second-order desire distinction explains persuasive tech harm because:

✓ Correct — ✅ ✓ Persuasive tech satisfies first-order impulses (check again) against second-order preferences (I want to want to stop). This exploitation of your own psychology raises autonomy questions.

❌ ❌ First-order: wanting to check. Second-order: wanting to want to stop. Persuasive tech satisfies the impulse against your own preference — potentially violating autonomy.

Replika behavior change case revealed that emotional AI users face:

✓ Correct — ✅ ✓ Discontinuation risk: emotional investment in an AI companion doesn't protect you from corporate decisions to change, monetize, or discontinue the product.

❌ ❌ Discontinuation risk: users who formed deep attachments were acutely vulnerable to a product decision. Emotional investment doesn't protect against corporate choices.

most significant gap in current data protection frameworks for the AI era is:

✓ Correct — ✅ ✓ The inference gap: regulations govern collection and sharing but not inference. AI inferring sensitive attributes from non-sensitive data is largely unaddressed by current law.

❌ ❌ The inference gap: current regulations govern what's collected and shared but not what's inferred. AI can infer sensitive attributes from non-sensitive data — and this is largely unregulated.

cognition theory raises concerns about AI dependency because:

✓ Correct — ✅ ✓ Extended cognition: if AI handles your memory and reasoning, it's part of your cognitive system. Corporate control of that system is a cognitive autonomy question, not just a product question.

❌ ❌ Extended cognition: AI handling memory and reasoning may be part of your cognitive system. Corporate control of that system means potential corporate influence over your cognitive processes.

structural conflict of interest in AI safety governance is:

✓ Correct — ✅ ✓ The structural conflict: safety-responsible companies face commercial pressure to deploy fast. External accountability mechanisms are needed because self-governance can't fully resolve this tension.

❌ ❌ The structural conflict: safety work and commercial deployment incentives are in tension within the same companies. Self-governance can't fully resolve an inherent conflict of interest.

literacy is fundamentally a civic capacity because:

✓ Correct — ✅ ✓ AI literacy enables you to evaluate claims, recognize political choices in technical framing, and participate meaningfully in governance decisions that affect everyone.

❌ ❌ AI literacy is civic because it enables recognizing political choices hidden in technical framing — and participating meaningfully in governance decisions that will shape society for decades.

Back Academy Home