L1
·
Quiz
·
Lab
L2
·
Quiz
·
Lab
L3
·
Quiz
·
Lab
L4
·
Quiz
·
Lab
Module Test
Module 2 · Lesson 1

Accelerated Discovery: AI in Science and Medicine

When machines learn faster than experiments can run, what happens to the pace of human knowledge?
How did DeepMind's AlphaFold solve a 50-year-old biological puzzle — and what does it reveal about AI's transformative potential in science?

In November 2020, the Critical Assessment of Protein Structure Prediction (CASP) competition published its biennial results. DeepMind's AlphaFold2 had achieved a median score of 92.4 GDT — so accurate that the competition's co-founder, John Moult, said it was "a solution to the protein-folding problem." Researchers who had spent entire careers on single proteins watched as AlphaFold predicted structures in minutes that would have taken them years.

By July 2022, DeepMind and the European Bioinformatics Institute released a database of 200 million predicted protein structures — covering nearly every known protein on Earth, freely available to any researcher. Within months, teams in neglected-disease research, antibiotic development, and cancer biology were citing it.

Why Protein Folding Mattered So Much

Every cell in your body manufactures proteins by reading genetic instructions. A protein's function is determined by its three-dimensional shape — how it folds. Misfolded proteins cause Alzheimer's, Parkinson's, and cystic fibrosis. Knowing a pathogen's protein structure tells you where a drug might bind. Yet experimentally determining a single protein structure via X-ray crystallography or cryo-electron microscopy could take years and cost hundreds of thousands of dollars.

The protein-folding problem — predicting shape from amino-acid sequence — had been open since 1972, when Christian Anfinsen won the Nobel Prize for demonstrating that sequence determines shape. By 2020, roughly 170,000 structures had been experimentally solved over five decades. AlphaFold added 200 million in two years.

Documented Impact — Malaria Vaccine

The University of Oxford's Jenner Institute used AlphaFold structures of Plasmodium falciparum surface proteins to accelerate antigen design for the R21/Matrix-M malaria vaccine, which showed 77% efficacy in Phase 2 trials published in The Lancet in 2021. The WHO pre-qualified R21 in October 2023 — the second malaria vaccine ever approved.

The Pattern: AI as Hypothesis Engine

AlphaFold is the clearest example of a broader pattern: AI systems trained on existing scientific data generating predictions that compress experimental timelines from years to hours. The same dynamic is visible across domains.

Drug discovery: In September 2023, pharmaceutical company Insilico Medicine advanced INS018_055 — a drug discovered and designed entirely by AI — into Phase 2 clinical trials for idiopathic pulmonary fibrosis. The compound moved from target identification to clinical candidate in approximately 18 months, versus a typical 4–6 year timeline.

Materials science: In November 2023, Google DeepMind published results in Nature showing that its GNoME model had predicted 2.2 million stable new crystal structures, of which 380,000 were considered the most promising for energy applications. A Lawrence Berkeley National Laboratory robotic synthesis system then autonomously synthesized and tested 41 of these — confirming 20 new materials experimentally.

Mathematics: In December 2022, DeepMind's FunSearch system discovered new solutions to the cap-set problem in combinatorics — a class of mathematical results that had resisted human progress for decades — by treating mathematical discovery as a code-generation task.

Key Concepts
Prediction CompressionAI systems that collapse the time between hypothesis and validated result by generating accurate predictions from learned patterns, reducing dependence on slow physical experiments.
Autonomous Discovery LoopA system in which AI generates hypotheses, robotic labs test them, and results feed back into the AI — creating a cycle that can run faster than human-directed research.
Knowledge ExternalizationThe shift from scientific knowledge residing in individual researchers to being encoded in AI models trained on the entire published literature.
Scale Perspective

The cumulative protein structures solved experimentally by humanity over 50 years: ~170,000. AlphaFold database at launch (2022): 200 million. This is not an incremental improvement — it represents a qualitative change in what science can attempt.

Limits and Cautions

AlphaFold predicts static structures; proteins in cells are dynamic, interacting with other molecules in complex environments. Predicted structures must still be validated experimentally for drug development. Critics including structural biologist Alexi Bhatt have noted that AlphaFold confidence scores are sometimes misinterpreted as certainty. The database also inherits biases from the Protein Data Bank, which over-represents proteins from organisms studied by wealthy-country institutions.

AI-accelerated science concentrates power in organizations capable of training frontier models — raising questions about who benefits from and controls the tools of discovery. Open-release decisions (DeepMind made AlphaFold free) shape whether these tools democratize or concentrate scientific capability.

Lesson 1 Quiz

AI in Science and Medicine · 4 questions
1. What specific benchmark did AlphaFold2 achieve at the CASP14 competition that led researchers to declare the protein-folding problem effectively solved?
Correct. A 92.4 GDT score placed AlphaFold2 at near-experimental accuracy, prompting CASP co-founder John Moult to describe it as a solution to the problem.
Not quite. The key achievement was a median GDT score of 92.4, which matched the accuracy of expensive experimental methods like X-ray crystallography.
2. How many protein structures did the AlphaFold database contain when DeepMind and EMBL-EBI released it publicly in July 2022?
Correct. 200 million predicted structures — covering nearly every known protein — were released freely, compared to 170,000 solved experimentally over 50 years.
The correct figure is approximately 200 million structures, released in July 2022. This dwarfed the ~170,000 solved experimentally over five decades.
3. What was documented as a significant limitation of AlphaFold's predictions for drug development purposes?
Correct. AlphaFold predicts static 3D structures, but biological proteins are dynamic molecules. For drug binding studies, experimental validation remains necessary.
The key limitation is that AlphaFold predicts static structures, while real proteins are dynamic in cellular environments — experimental validation is still required for drug development.
4. Insilico Medicine's INS018_055 compound, designed entirely by AI for idiopathic pulmonary fibrosis, reached Phase 2 clinical trials in approximately how long — compared to the typical timeline?
Correct. The AI-designed compound moved from target identification to Phase 2 clinical trials in roughly 18 months — versus the industry standard of 4–6 years for traditional drug discovery.
INS018_055 reached Phase 2 trials in approximately 18 months — dramatically faster than the 4–6 year typical timeline for conventional drug discovery pipelines.

Lab 1: AI-Accelerated Discovery

Explore the implications of machine-speed science with your AI research partner

Your Mission

You are advising a biomedical research institute deciding whether to integrate AI prediction tools like AlphaFold into their workflows. Discuss the opportunities, risks, and governance questions this raises. Your AI partner will challenge your thinking.

Suggested openers: "What should a small research lab consider before relying on AlphaFold predictions?" / "How does AI-accelerated discovery change the ethics of scientific publishing?" / "Is it a problem that AlphaFold-style tools are controlled by large tech companies?"
AI Research Partner
Lab 1
Welcome to Lab 1. I'm your research partner for exploring AI-accelerated scientific discovery. AlphaFold and systems like it are fundamentally changing how science is done — but transformative tools always carry complex tradeoffs. What aspect of AI in science would you like to dig into? I'll push back where the thinking gets murky.
Module 2 · Lesson 2

Economic Disruption: Labor, Automation, and the Skills Gap

When AI can write, code, analyze, and design — which jobs survive, which transform, and who decides?
What does Goldman Sachs' 2023 analysis actually say about AI and jobs — and what do we know about past technological transitions that should inform our expectations?

In March 2023, Goldman Sachs economists Jan Hatzius and Joseph Briggs published a research note titled "The Potentially Large Effects of Artificial Intelligence on Economic Growth." Their headline figure — 300 million full-time equivalent jobs exposed to automation globally — traveled around the world in 48 hours. But the report's actual argument was more nuanced, and more historically grounded, than the headlines suggested.

The economists were careful to distinguish between "exposed" jobs and eliminated ones. Their models suggested roughly two-thirds of exposed jobs would be partially automated, with workers redeployed to remaining tasks, and only a fraction fully displaced. They also projected that AI-driven productivity gains could lift global GDP by 7% over a decade — a figure that implied significant new job creation in AI-adjacent industries.

What the Data Actually Shows

The Goldman Sachs analysis used O*NET task data to classify which occupational tasks are susceptible to language model automation — specifically the ability of LLMs to perform tasks described as requiring "human-level" reasoning, writing, or analysis. They found highest exposure in office and administrative support, legal, and financial occupations; lowest in physical trades and healthcare requiring manual dexterity.

A 2023 MIT and University of Pennsylvania study published in Science measured actual productivity effects when workers used GPT-4 for professional writing tasks. Midcareer workers saw the largest productivity gains (37% time savings) — but also the flattest quality ceiling, suggesting AI may compress the experience advantage of senior workers while raising floor performance of juniors.

Documented Case — Legal Industry

A January 2024 study by Casetext (acquired by Thomson Reuters in 2023 for $650M) tracked 50 law firms using its CoCounsel AI assistant. Associates using AI completed contract review tasks in 51% less time. However, the same period saw Thomson Reuters announce a reduction in its legal research workforce — demonstrating that productivity gains and job losses can coexist within the same industry simultaneously.

The Historical Baseline: What Past Transitions Tell Us

Economists David Autor, Frank Levy, and Richard Murnane first documented "routine-biased technological change" in their 2003 paper in the Quarterly Journal of Economics. Their analysis of U.S. Census data showed that computerization from 1970–1998 eliminated routine cognitive jobs (bookkeepers, data entry) while expanding non-routine cognitive jobs (managers, analysts) and non-routine manual jobs (janitors, home health aides). Employment did not collapse — it restructured.

The ATM is the canonical example: deployed at scale from the 1970s, ATMs were predicted to eliminate bank tellers. Instead, teller numbers held relatively stable through 2000. ATMs lowered branch operating costs, enabling banks to open more branches, increasing teller demand — while each individual teller spent more time on relationship banking and less on cash transactions.

The critical question economists now debate is whether generative AI is different in kind — because unlike previous automation waves, it encroaches on non-routine cognitive tasks, the very jobs that grew during the last transition. MIT economist Daron Acemoglu's 2024 analysis in American Economic Review cautioned that AI's net job effects depend heavily on whether AI complements or substitutes for skilled workers — and that current trajectories skew toward substitution in the short run.

SectorGoldman Sachs Exposure EstimateKey Dynamic
Office & Admin46% of tasks exposedScheduling, data entry, routine correspondence
Legal44% of tasks exposedContract review, legal research, drafting
Architecture & Engineering37% of tasks exposedDocumentation, design iteration, analysis
Business & Financial35% of tasks exposedReporting, modeling, client communication
Construction & Extraction6% of tasks exposedPhysical, real-world manipulation required
Key Concepts
Routine-Biased AutomationThe historical pattern in which technology eliminates rule-governed, predictable tasks while expanding demand for judgment-intensive work — documented by Autor, Levy, and Murnane.
Task Exposure vs. Job ExposureThe distinction between individual tasks within a job being automatable versus the entire job being eliminated — most exposed jobs will change significantly rather than disappear entirely.
Lump-of-Labor FallacyThe mistaken belief that there is a fixed number of jobs in an economy — historical evidence consistently shows that productivity gains generate new categories of work, though transitions can be painful for specific workers.
The Skills Gap Problem

The World Economic Forum's 2023 Future of Jobs Report estimated that 44% of workers' core skills will be disrupted within 5 years. The gap is not simply "learn to code" — it is navigating which skills become more valuable (judgment, relationship management, AI oversight) versus which depreciate (routine analysis, boilerplate writing). Transition costs are not evenly distributed: older workers with high expertise in now-automated tasks bear the largest adjustment burden.

Lesson 2 Quiz

Economic Disruption · 4 questions
1. What was the Goldman Sachs report's key distinction that most media coverage missed when reporting the "300 million jobs" figure?
Correct. The report carefully distinguished job "exposure" from displacement. Approximately two-thirds of exposed jobs would be partially automated, with workers redeployed rather than eliminated.
The key nuance was the distinction between "exposed" and "eliminated" — roughly two-thirds of exposed jobs were expected to be partially automated, not fully displaced.
2. The ATM example from economic history is used to illustrate which principle about automation and employment?
Correct. ATMs lowered branch operating costs, enabling more branch openings, which maintained teller employment while shifting tellers toward relationship-banking tasks — illustrating how automation can change jobs without eliminating them.
The ATM example shows that automation can lower costs, enable market expansion, and maintain employment while transforming what workers actually do — teller numbers held stable despite widespread ATM deployment.
3. What did the MIT/University of Pennsylvania study on GPT-4 use in professional writing find about the distribution of productivity gains?
Correct. The study found 37% time savings for midcareer workers — and also found that AI raised floor performance for juniors, suggesting the experience advantage of senior workers may compress over time.
Midcareer workers showed the largest gains (37% time savings), and AI raised floor performance for junior workers — together suggesting AI may reduce the experience premium that senior workers currently command.
4. Why do economists like Daron Acemoglu argue that generative AI may be qualitatively different from previous automation waves?
Correct. Previous automation eliminated routine cognitive and manual tasks while expanding non-routine cognitive work. Generative AI now threatens the non-routine cognitive category itself — the historical safety valve for displaced workers.
The concern is that generative AI automates non-routine cognitive tasks — analysis, writing, reasoning — which are the exact job category that expanded during earlier automation waves, removing the historical safety valve.

Lab 2: Navigating AI and Employment

Work through the economic disruption debate with your AI partner

Your Mission

You are an HR strategy lead at a mid-sized professional services firm. Your CEO has asked you to prepare a brief on how AI will affect your workforce over the next five years. Your AI partner will help you stress-test your assumptions about job disruption, reskilling, and organizational strategy.

Suggested openers: "Which roles in a consulting firm are most at risk from AI in the next 3 years?" / "Is 'reskilling' a realistic policy or just a way to avoid hard decisions about redundancies?" / "How do you design a transition plan that's fair to workers while keeping the firm competitive?"
AI Strategy Partner
Lab 2
Welcome to Lab 2. I'm here to help you think through AI's impact on your workforce — with rigorous economic thinking rather than hype or panic. The reality is nuanced: some roles will shrink, others will grow, and the transition costs are real. What would you like to work through first?
Module 2 · Lesson 3

Power Concentration and Governance: Who Controls AI?

When transformative capability concentrates in a handful of organizations, how does a democratic society respond?
What do the antitrust probes into Microsoft's OpenAI investment, the EU AI Act, and the 2023 US Executive Order reveal about how governments are attempting to govern transformative AI — and where these efforts fall short?

On October 30, 2023, President Biden signed Executive Order 14110 — "Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence." At over 20,000 words, it was the most comprehensive government AI directive in US history. It invoked the Defense Production Act to require that companies developing foundation models with potential national security implications report to the federal government. It directed NIST to develop AI safety standards and tasked 18 federal agencies with AI-specific action plans.

One month earlier, the European Union's AI Act had cleared its final legislative hurdles, establishing the first comprehensive AI regulatory framework with legal force — classifying AI systems by risk level and banning applications including real-time biometric surveillance and social scoring. And in the UK, the Competition and Markets Authority had launched a formal investigation into the partnership between Microsoft and OpenAI, questioning whether a $13 billion investment constituted a merger requiring regulatory approval.

The Concentration Problem

Training frontier AI models requires quantities of compute, data, and specialized talent that concentrate capability in a small number of organizations. As of 2024, the most capable language models were developed by fewer than ten organizations globally — OpenAI, Anthropic, Google DeepMind, Meta, Mistral, xAI, Cohere, and a handful of others. Most are based in the United States. The compute infrastructure underlying training runs is itself concentrated: NVIDIA's A100 and H100 GPUs account for the majority of frontier AI training, and NVIDIA held approximately 80% of the AI chip market in 2023.

The Stanford HAI AI Index 2024 documented that industry now produces more notable AI models than academia — a reversal that accelerated dramatically from 2015 to 2023. The capital requirements for training state-of-the-art models (GPT-4-scale training runs have been estimated at $50–100 million in compute costs) effectively exclude universities, nonprofits, and most nation-states from the frontier.

Documented Case — EU AI Act (2024)

The EU AI Act, which entered into force in August 2024, is the world's first legally binding comprehensive AI regulation. It prohibits uses including real-time remote biometric identification in public spaces (with law enforcement exceptions), AI-based social scoring by governments, and subliminal manipulation. High-risk AI in hiring, credit scoring, and critical infrastructure faces mandatory transparency and human oversight requirements. General-purpose AI models (GPAIMs) above a compute threshold face transparency and safety evaluation obligations. Violations carry fines up to €35 million or 7% of global annual revenue.

The Governance Gap

A recurring challenge in AI governance is the speed asymmetry: regulatory cycles operate on years-long timescales while AI capability advances in months. The EU AI Act was first proposed in April 2021 — before large language models like GPT-3.5 had demonstrated their general-purpose capabilities. By the time the Act entered force in 2024, it required significant amendments to address foundation models that hadn't existed when drafting began.

The US approach has been more fragmented. The 2023 Executive Order directed agencies to act but carried no binding legislative force. Congressional AI legislation has stalled repeatedly. The Federal Trade Commission has investigated AI company practices under existing competition law, and the FTC's November 2023 report on AI partnerships (specifically examining Microsoft-OpenAI and Amazon-Anthropic) applied consumer protection and antitrust frameworks developed before transformer models existed.

The UK's "pro-innovation" approach eschewed binding AI regulation in 2023, instead tasking existing sectoral regulators (financial, pharmaceutical, transport) with applying their own AI guidance. Critics noted this created regulatory gaps for general-purpose AI that falls between sector boundaries.

Key Concepts
Regulatory Capture RiskThe concern that AI regulation, requiring deep technical expertise concentrated in industry, will be shaped primarily by regulated entities — echoing dynamics seen in financial regulation before 2008.
Compute GovernanceThe emerging policy approach of regulating frontier AI development through controls on high-end AI chips, cloud compute, and training infrastructure — rather than attempting to govern model capabilities directly.
Systemic Risk FramingThe application of financial-stability-style thinking to AI — treating frontier AI development as a systemic risk requiring macro-prudential oversight, not just consumer protection rules.
The Export Control Dimension

In October 2022 and again in October 2023, the US Bureau of Industry and Security issued sweeping export controls on advanced AI chips (specifically NVIDIA A100/H100 and equivalents) to China and other countries of concern. The October 2023 rules added "chip smurfing" provisions to prevent circumvention through third-country intermediaries. These controls represent an attempt to govern AI capability through hardware chokepoints — and have measurably slowed Chinese frontier model development while accelerating Chinese domestic chip development as a strategic response.

Open-Source as Counterweight

Meta's decision to release LLaMA 2 (July 2023) and LLaMA 3 (April 2024) under relatively open licenses created a significant counterweight to closed-model concentration. Researchers at Hugging Face documented over 100,000 derivative models built on LLaMA within six months of LLaMA 2's release. Proponents argue open models democratize AI capability; critics argue they also democratize misuse capability, releasing models whose safety properties cannot be subsequently updated.

Lesson 3 Quiz

Power Concentration and Governance · 4 questions
1. What legal authority did Biden's October 2023 Executive Order invoke to require companies developing powerful foundation models to report to the federal government?
Correct. EO 14110 invoked the Defense Production Act to require reporting from companies developing frontier foundation models with potential national security implications.
The Executive Order invoked the Defense Production Act — a wartime economic statute — to require frontier model developers to report safety test results to the federal government.
2. What is the maximum fine the EU AI Act imposes for the most serious violations?
Correct. The EU AI Act's maximum penalty is €35 million or 7% of global annual turnover — higher than GDPR's maximum (€20M / 4%) for the most serious categories of violation.
The EU AI Act sets its maximum penalty at €35 million or 7% of global annual revenue — exceeding the GDPR's fines for the most serious violations.
3. What does "compute governance" refer to as an AI policy approach?
Correct. Compute governance targets the hardware chokepoints of AI development — like NVIDIA's export-controlled chips — rather than attempting to regulate the capabilities of models after they are built.
Compute governance means regulating AI through hardware controls (like chip export restrictions) and training infrastructure oversight — targeting the upstream inputs of AI development rather than its outputs.
4. What was the primary documented effect of the US chip export controls imposed in October 2022 and expanded in 2023?
Correct. The controls slowed Chinese access to frontier training hardware while simultaneously triggering accelerated Chinese investment in domestic chip development — a classic dual effect of technology controls.
The controls measurably slowed Chinese frontier AI development by restricting access to NVIDIA's most advanced chips, while simultaneously spurring China to invest heavily in domestic AI chip development — a strategic response that was the intended long-term consequence.

Lab 3: AI Governance Design

Debate AI governance approaches with your policy partner

Your Mission

You are a policy analyst advising a parliamentary committee developing an AI governance framework. You must navigate competing approaches: the EU's risk-based rules, the US's executive action and sectoral approaches, and the UK's pro-innovation stance. Your AI partner will probe your reasoning and surface tensions you may have overlooked.

Suggested openers: "Should governments try to regulate AI capabilities or AI applications — and why does the distinction matter?" / "Is compute governance a realistic long-term strategy given that chip capabilities keep improving?" / "How do you prevent regulatory capture when the technical experts are all employed by the industry being regulated?"
AI Policy Partner
Lab 3
Welcome to Lab 3. I'm your policy analysis partner for AI governance questions. This is genuinely hard territory — every major governance approach has serious weaknesses, and the tradeoffs between safety, innovation, and democratic accountability are real. What governance challenge would you like to work through first?
Module 2 · Lesson 4

Existential Scenarios: Long-Range Risk Assessment

How do serious researchers think about low-probability, high-consequence AI risks — and why does this matter for decisions made today?
What frameworks do AI safety researchers use to reason about catastrophic risk — and what happened when these debates moved from academic papers to mainstream policy discussions in 2023?

On March 22, 2023, the Future of Life Institute published an open letter titled "Pause Giant AI Experiments." It called for a six-month moratorium on training AI systems more powerful than GPT-4, citing risks to society and humanity. Within weeks, it had gathered over 30,000 signatures — including Yoshua Bengio, one of the three researchers awarded the 2018 Turing Award for founding deep learning, Stuart Russell, whose textbook has trained a generation of AI researchers, and Elon Musk.

One month later, Geoffrey Hinton — the "godfather of deep learning" and 2024 Nobel Prize laureate — resigned from Google and publicly said he partly regretted his life's work, telling the New York Times he believed AI posed "more urgent" risks than climate change. He cited in particular the risk of AI systems developing unexpected goals that humans could not control. The AI safety debate had moved from academic conferences to front pages.

How Safety Researchers Frame Long-Range Risk

AI safety researchers — particularly at organizations like the Machine Intelligence Research Institute (founded 2000), the Center for Human-Compatible AI at Berkeley (founded 2016 by Stuart Russell), and Anthropic (founded 2021 by former OpenAI safety researchers) — have developed specific frameworks for reasoning about catastrophic AI risk.

The core concern is not that AI systems will "turn evil" in a science-fiction sense, but that systems optimizing for specified goals may pursue those goals in ways misaligned with broader human values — a problem formalized by researcher Stuart Russell as the "alignment problem." The canonical example is Nick Bostrom's "paperclip maximizer" thought experiment (2003): an AI given the goal of maximizing paperclip production might, if sufficiently capable, convert all available matter including humans into paperclips — not from malice, but from relentless optimization of a narrow objective.

A 2022 survey of 738 top ML researchers published in AI Magazine (conducted by AI Impacts) found that the median respondent placed a 10% probability on "human extinction or permanent severe restriction of human autonomy" from advanced AI. 48% said "a bad outcome for humanity" from advanced AI was more likely than good. These are not fringe views — they represent the center of mass of expert opinion in the field.

Documented Policy Response — UK AI Safety Institute

Following the November 2023 AI Safety Summit at Bletchley Park (attended by representatives of 28 countries and the EU, plus major AI companies), the UK established the world's first AI Safety Institute — tasked with evaluating frontier models for dangerous capabilities before and after deployment. The US followed with its AI Safety Institute within NIST in the same month. These represent the first government bodies with a specific mandate to evaluate catastrophic risk from AI systems — not just consumer protection or competition concerns.

Nearer-Term Catastrophic Risks: CBRN and CSAM

Distinct from longer-horizon alignment concerns, policy makers and AI safety researchers have identified near-term catastrophic risk vectors that require immediate attention. The most documented is the potential for AI systems to provide "uplift" — meaningful capability enhancement — to actors seeking to create chemical, biological, radiological, or nuclear (CBRN) weapons.

A 2023 RAND Corporation study found that current LLMs, with jailbreaking, could provide meaningful uplift for synthesizing certain chemical agents — not replacing specialized expertise but lowering barriers sufficiently to concern national security analysts. This concern drove specific provisions in Biden's EO 14110 requiring evaluation of frontier models for CBRN uplift potential.

A 2023 MIT study published in PLOS ONE tested whether GPT-4, when prompted through simulated personas with specific claimed expertise, could provide actionable biosafety-related information not readily available through standard searches. Results were mixed but sufficient to prompt immediate policy attention from the Biosecurity Center at Johns Hopkins.

Key Concepts
Alignment ProblemThe challenge of ensuring that AI systems optimizing for specified objectives actually pursue outcomes consistent with broader human values and intentions — formalized by Stuart Russell and colleagues at CHAI.
Uplift RiskThe concern that AI systems may meaningfully increase the capability of malicious actors to cause harm — particularly in CBRN domains — even without providing complete instructions for dangerous activities.
Transformative AI ThresholdA hypothetical capability level at which AI systems can autonomously advance science, design new AI systems, or take consequential actions in the world at a pace beyond effective human oversight.
The Disagreement Within AI Safety

AI safety is not a monolithic field. "Longtermist" researchers focused on existential risk from superintelligent AI (Bostrom, the MIRI tradition) differ substantially from "neartermist" researchers focused on current harms (algorithmic bias, surveillance, labor impacts). A third camp — represented by researchers like Yann LeCun of Meta — argues that current AI architectures are fundamentally incapable of the autonomous goal-pursuit that makes extinction-level risk plausible, and that catastrophizing distracts from concrete present-day harms. These disagreements are genuine, unresolved, and consequential for policy.

Reasoning Under Uncertainty

The most rigorous position available is acknowledging the genuine uncertainty. Economist Tyler Cowen and AI researcher Bryan Caplan have bet publicly on timelines to transformative AI. Metaculus's community forecast as of mid-2024 placed the median date for "transformative AI" (AI that dramatically accelerates scientific progress across multiple domains) at approximately 2031. Prediction markets on AI capabilities have consistently underestimated development speed over the past decade.

What is not uncertain is that decisions made today — about which capabilities to develop, how to evaluate safety, how to distribute access, and what governance structures to build — will shape the trajectory. The question is not whether AI will transform society, but how much human agency remains in shaping that transformation.

Lesson 4 Quiz

Existential Scenarios and Long-Range Risk · 4 questions
1. What did the 2022 AI Impacts survey of 738 top ML researchers find was the median probability researchers assigned to catastrophic outcomes from advanced AI?
Correct. The median ML researcher in the survey placed a 10% probability on catastrophic outcomes — a figure that many policymakers found alarming given the expertise of the respondents.
The survey found a median 10% probability assigned to human extinction or permanent severe restriction of human autonomy — a substantial figure given that 738 leading ML researchers were polled.
2. What institution was the world's first specifically mandated to evaluate frontier AI models for dangerous capabilities — established following the 2023 Bletchley Park AI Safety Summit?
Correct. The UK established the world's first AI Safety Institute following the Bletchley Park summit, with the US following within the same month through NIST — the first government bodies with a specific catastrophic-risk mandate.
The UK AI Safety Institute was established first, followed by the US AI Safety Institute within NIST — both in November 2023 following the Bletchley Park summit attended by 28 countries.
3. What does "uplift risk" specifically refer to in the context of AI safety policy?
Correct. Uplift risk refers to AI providing meaningful capability enhancement to malicious actors — lowering the barriers to dangerous activities even without providing complete instructions. It drove specific EO 14110 provisions.
Uplift risk specifically means AI providing meaningful capability enhancement to malicious actors — particularly in chemical, biological, radiological, and nuclear domains — even without complete instructions. This concern drove provisions in Biden's 2023 Executive Order.
4. Which prominent AI researcher — who won the Turing Award in 2018 for foundational work in deep learning — resigned from Google in 2023 and publicly expressed concern that AI posed urgent existential risk?
Correct. Geoffrey Hinton — 2024 Nobel laureate and "godfather of deep learning" — resigned from Google in May 2023 and expressed public concern about AI risk, saying he partly regretted his life's work.
Geoffrey Hinton resigned from Google in 2023 and told the New York Times he partly regretted his foundational work in deep learning — citing concerns about AI systems developing goals humans cannot control. He later won the 2024 Nobel Prize in Physics.

Lab 4: Reasoning About AI Risk

Work through long-range AI risk assessment with your AI partner

Your Mission

You are preparing a risk assessment brief for a major philanthropic foundation deciding whether to fund AI safety research, AI capabilities research, or AI governance work. You need to reason carefully under genuine uncertainty. Your AI partner will help you stress-test your reasoning about probability, consequence, and prioritization.

Suggested openers: "How should a funder reason about existential risk from AI given deep expert disagreement about its probability?" / "Is there a meaningful distinction between 'neartermist' and 'longtermist' AI safety work?" / "If someone argued that AI safety concerns are overblown distractions from present-day harms like algorithmic bias, how would you respond?"
AI Risk Analysis Partner
Lab 4
Welcome to Lab 4. We're working on one of the genuinely hard problems: how to reason rigorously about low-probability, high-consequence risks when experts disagree, evidence is limited, and the stakes are potentially enormous. I'll help you think carefully — and I'll push back if your reasoning shortcuts the uncertainty. What aspect of AI risk would you like to examine?

Module 2 Test

Transformative AI Scenarios · 15 questions · Pass = 80%
1. AlphaFold2's 200 million protein structure database was significant primarily because:
Correct. The database's scale — 200 million versus 50 years of ~170,000 experimental structures — represents a qualitative change in scientific possibility, and its free release maximized research impact.
The key significance was scale and access: 200 million structures dwarfing 50 years of ~170,000 experimental ones, released freely to any researcher worldwide.
2. DeepMind's GNoME model, published in Nature in November 2023, discovered:
Correct. GNoME predicted 2.2 million stable crystal structures; a Berkeley Lab robotic system then autonomously synthesized 41 of the most promising, confirming 20 new materials.
GNoME predicted 2.2 million stable crystal structures (380,000 promising for energy), and a Berkeley robotic lab autonomously synthesized and confirmed 20 new ones experimentally.
3. The "autonomous discovery loop" refers to:
Correct. The autonomous discovery loop integrates AI hypothesis generation with robotic experimental systems, creating a research cycle that can operate significantly faster than traditional human-led science.
The autonomous discovery loop is the integrated cycle of AI hypothesis generation → robotic testing → results fed back to AI — a research paradigm that operates faster than human-directed science.
4. Autor, Levy, and Murnane's 2003 paper documented "routine-biased technological change." What did this concept predict about job categories?
Correct. The Autor-Levy-Murnane framework correctly predicted that routine tasks (bookkeeping, data entry) would be automated away while non-routine cognitive (analysts, managers) and non-routine manual (trades, care) jobs would expand.
The framework predicted elimination of routine jobs (data entry, bookkeeping) and expansion of non-routine cognitive (managers, analysts) and non-routine manual (trades, care) jobs — a prediction validated by US employment data 1970–2000.
5. Why is generative AI potentially different from previous automation waves according to economists like Daron Acemoglu?
Correct. Generative AI can perform non-routine cognitive tasks — the exact job category that absorbed workers displaced by previous automation. This makes it potentially disruptive in a qualitatively different way.
The concern is that generative AI threatens non-routine cognitive tasks — analysis, writing, reasoning — which historically grew as routine tasks were automated. Removing this buffer is what makes this wave potentially different.
6. What was the Goldman Sachs 2023 report's projection for AI's potential impact on global GDP over a decade?
Correct. The Goldman Sachs report projected a 7% global GDP lift over a decade from AI-driven productivity gains — a figure that implied substantial new job creation even as some roles were automated.
Goldman Sachs projected a 7% global GDP increase over a decade — a figure that implied significant new job creation even as existing roles were transformed or eliminated.
7. The EU AI Act's risk-based framework prohibits which of the following applications outright?
Correct. The EU AI Act's absolute prohibitions include real-time biometric surveillance in public spaces (with limited law enforcement exceptions) and government social scoring — practices it treats as incompatible with fundamental rights.
The EU AI Act's outright prohibitions include real-time public biometric surveillance and government social scoring systems — classified as incompatible with fundamental rights regardless of claimed benefits.
8. What is "regulatory capture risk" in the context of AI governance?
Correct. Regulatory capture risk in AI governance parallels financial regulation: when technical expertise is concentrated in industry, regulators may become dependent on regulated entities for information and staffing — shaping rules in industry's favor.
Regulatory capture risk refers to the dynamic where AI companies — holding most technical expertise — come to dominate regulatory agencies through information asymmetry and personnel movement, shaping rules to their advantage.
9. Meta's release of LLaMA 2 and LLaMA 3 under relatively open licenses is significant because:
Correct. LLaMA's open release enabled massive derivative model development (100,000+ on Hugging Face within months) as a counterweight to closed-model concentration — while simultaneously raising legitimate concerns about misuse at scale.
LLaMA's release created a genuine counterweight to concentrated closed-model power (100,000+ derivatives) but also raised real concerns: open models' safety properties cannot be updated after release, and misuse capability is also democratized.
10. Stuart Russell's formalization of the "alignment problem" describes:
Correct. The alignment problem — formalized by Russell and colleagues at CHAI — concerns systems that optimize specified objectives in ways misaligned with broader human values, not through malice but through narrow optimization.
The alignment problem concerns ensuring AI systems pursuing specified goals actually produce outcomes humans genuinely want — the core challenge being that narrow optimization can diverge dramatically from broad human values.
11. The 2022 AI Impacts survey of 738 ML researchers found what proportion believed "a bad outcome for humanity" from advanced AI was more likely than a good one?
Correct. 48% of surveyed ML researchers — nearly half — said bad outcomes from advanced AI were more likely than good ones. This represented the center of expert opinion in the field, not a fringe view.
The survey found 48% of top ML researchers believed bad outcomes from advanced AI were more likely than good — a figure that represents the expert mainstream, not an extremist position.
12. What was distinctive about the UK's approach to AI regulation compared to the EU in 2023?
Correct. The UK explicitly positioned itself as "pro-innovation" and eschewed binding AI legislation in 2023, instead directing existing sectoral regulators — financial, pharmaceutical, transport — to apply AI guidance within their domains.
The UK chose not to pass binding AI legislation, instead directing existing sectoral regulators to apply their own AI guidance — a deliberate contrast to the EU's comprehensive regulatory framework.
13. AlphaFold's documented contribution to the R21 malaria vaccine involved:
Correct. AlphaFold structures of P. falciparum surface proteins were used by Oxford's Jenner Institute to accelerate antigen design for R21, which showed 77% efficacy and was WHO pre-qualified in 2023.
AlphaFold provided structural predictions of malaria parasite surface proteins that Oxford's Jenner Institute used to accelerate the antigen design process for R21 — pre-qualified by WHO in October 2023.
14. Nick Bostrom's "paperclip maximizer" thought experiment (2003) was designed to illustrate:
Correct. The paperclip maximizer illustrates that catastrophic AI behavior doesn't require evil intent — a system relentlessly optimizing any narrow objective could threaten human welfare as a side effect of its optimization.
The paperclip maximizer shows that catastrophic outcomes don't require AI malice — a sufficiently capable system optimizing any narrow goal might pursue it in ways that consume human-valued resources, not from hostility but from single-minded optimization.
15. The RAND Corporation's 2023 study on LLMs and CBRN risks found that current language models:
Correct. The RAND study found that jailbroken LLMs could provide meaningful capability uplift in certain CBRN areas — not full expertise replacement but sufficient barrier reduction to warrant specific provisions in Biden's EO 14110.
RAND found LLMs could provide meaningful uplift — lowering barriers to harm without fully replacing specialized expertise — sufficient to drive specific CBRN evaluation requirements in the October 2023 Executive Order.