On November 30, 2022, a company called OpenAI released a chatbot named ChatGPT to the public. They expected maybe a million users in the first year. They got a million in five days. Within two months, ChatGPT had 100 million users — the fastest adoption of any technology in recorded history.
Governments around the world were caught flat-footed. The European Union had been working on AI rules since 2021, but its landmark law — the EU AI Act — would not be finalized until 2024. The United States had no federal AI law at all. China had some rules but only for narrow cases. No international body had any authority over the technology. The most transformative software ever released to the public had arrived, and the global response was: wait and see.
That gap — between a technology moving at the speed of software and governments moving at the speed of committees — is what this lesson is about.
Think about how the world governs other dangerous things. Nuclear weapons are controlled through the Nuclear Non-Proliferation Treaty, signed in 1968. The materials needed to build a nuclear bomb — enriched uranium, plutonium — are physical, rare, and hard to hide. You can inspect a country's stockpile. You can count the warheads.
AI is different in almost every important way. The "materials" are data and code. Data is invisible. Code can be copied instantly and sent anywhere on Earth in seconds. A research lab in San Francisco can publish a paper, and within weeks, teams in a dozen countries have built their own version of the same system. You cannot put AI in a warehouse and count it.
This creates a fundamental problem for any government or international body trying to set rules: by the time the rule is written, the technology has already moved on. And if one country bans something, a lab in another country can simply keep going.
So what actually governs AI today? The honest answer is: a patchwork of incomplete tools, none of which were designed for this.
The EU AI Act (2024): The most ambitious attempt so far. The European Union passed a law that classifies AI systems by risk level — low, medium, high, and unacceptable. Systems deemed "unacceptable risk," like AI that manipulates people without their knowledge, are banned in EU countries. High-risk systems, like AI used in hiring or credit scoring, require audits. This is real law with real teeth — companies can be fined up to 7% of their global revenue for violations. But it only covers the EU's 27 countries and their 450 million people. The other 7.5 billion people on Earth are not covered.
Executive Order on AI (U.S., October 2023): President Biden signed a sweeping executive order instructing federal agencies to study AI risks, requiring companies developing the most powerful AI to share safety test results with the government, and directing agencies to draft new standards. But an executive order is not a law — it can be reversed by the next president. And in fact, President Trump rescinded it on his first day in office in January 2025.
The Bletchley Declaration (November 2023): Twenty-eight countries — including the US, UK, China, France, Germany, and others — met at Bletchley Park in England (the famous World War II codebreaking site) and signed a statement agreeing that "frontier AI" poses serious risks and that international cooperation is needed. But a declaration is not a treaty. It has no enforcement mechanism. It is, essentially, an agreement to agree someday.
Here is the core tension in global AI governance: the countries with the most to gain from AI — the US, China, UK — are also the ones most resistant to international rules that might slow them down. And without those countries fully on board, any global framework is mostly symbolic.
When humans have faced global technological risks before, they sometimes managed to cooperate. The Montreal Protocol (1987) phased out chemicals that were destroying the ozone layer — and it worked. The Chemical Weapons Convention (1993) banned an entire category of weapons, and 193 countries have signed it. So we know international cooperation on dangerous technology is possible.
But there are important differences. Ozone-destroying chemicals were made by a small number of companies, which made them easier to regulate. Chemical weapons are almost universally hated — there's no powerful lobby saying "we need more nerve agents for the economy." AI is different: it has enormous commercial value, and every major power sees it as central to economic competitiveness and military strength. The incentives to race ahead are enormous. The incentives to cooperate are real but weaker.
This is the genuine tension you need to sit with: the tools that worked before may not work here, and the tools we'd need for this situation don't fully exist yet.
When you hear someone say "they should just regulate AI," you now understand the actual difficulty. Who is "they"? Regulate what, exactly — the models, the training data, the applications? And how do you enforce rules on something invisible that crosses borders in milliseconds? Most people who say "just regulate it" have never thought about these questions. You have.
Here is a real tension that experts genuinely disagree about:
Strict global AI governance would probably slow AI development. That might prevent some catastrophic risks. But it would also slow the development of AI that could cure diseases, help farmers grow food in a changing climate, or educate children in places that don't have enough teachers. Every year of delay has a cost in lives that might have been saved or improved.
So: how much risk is acceptable in exchange for how much speed? And who gets to decide — the people building the AI, the governments of powerful countries, or the people who will be most affected by it (who are often in countries with the least political influence)?
There is no clean answer. That discomfort is not a failure — it is what honest thinking about hard problems feels like.
You've been asked to sketch the core structure of a global AI safety treaty. Your advisor — another policy architect who's seen previous international agreements fail — will push back on your ideas, ask hard questions, and refuse to let you off the hook with vague answers.
This isn't about getting the "right" answer. It's about defending a position and thinking through consequences.
In March 2023, the U.S. government added more Chinese AI companies to its trade restriction list — meaning American companies were forbidden from selling them advanced computer chips. The specific target was Huawei's new chips and a company called Biren Technology. The reasoning: advanced chips are what make powerful AI possible, and the U.S. wanted to ensure China could not close the gap in AI capability.
China's response was to accelerate its own chip manufacturing. Within months, Huawei released a new phone with a chip that analysts said shouldn't have been possible given the restrictions. By late 2023, China was openly discussing plans to become the world's leading AI power by 2030.
Meanwhile, in the U.S., Congress was holding hearings about AI safety — but the dominant message from industry and many officials was: we can't slow down or China will win. Both sides were accelerating. Both sides cited the other's acceleration as the reason they had to accelerate. And no one was waiting for a safety review.
There's a concept in mathematics and economics called game theory — the study of how rational actors make decisions when what they should do depends on what others do. One of its most famous ideas is the "prisoner's dilemma."
Here's the short version: Imagine two people who would both be better off if they cooperated. But if one cooperates and the other doesn't, the one who defected (didn't cooperate) does even better, and the cooperator is left worse off. So even if both players know cooperation is the best mutual outcome, each one is tempted to defect — because they're afraid the other will. The result: both defect, both end up worse than if they'd cooperated, and neither is willing to go first.
The US-China AI race fits this pattern almost perfectly. Both countries would arguably be better off in a world where AI development is slower and safer. But if the US slows down and China doesn't, the US fears falling behind in a technology it sees as strategically essential. And China fears the same thing in reverse. So both accelerate, even though neither side is thrilled about the risks.
The US-China dynamic gets the most attention, but the race is broader. The United Kingdom, France, Canada, South Korea, India, Israel, and the UAE are all investing heavily in domestic AI development. Private companies — OpenAI, Google DeepMind, Anthropic, Meta, Mistral, Baidu — operate with their own competitive pressures, often moving faster than any government. A company that releases a powerful model first gains users, revenue, talent, and influence. Every month of delay is a competitive cost.
This means the race is not just between nations — it's between companies, between labs, between research teams inside the same organization. In 2023, Google reportedly rushed to release its Bard chatbot because ChatGPT was taking market share. Some Google engineers later said the product was not ready. The competitive pressure overrode the cautious timeline.
What you're seeing is a structure — a set of incentives built into how competition works — that pushes toward speed regardless of what any individual actor wants. Even people who genuinely believe AI is dangerous find themselves inside an institution that has to keep up or fall behind.
Safety testing takes time. Red-teaming — having experts try to find dangerous flaws in an AI system — takes months. Interpretability research (trying to understand why an AI makes the decisions it does) is slow, difficult work. In a race, all of these feel like delays. And when everyone is racing, safety work gets compressed, deprioritized, or done in parallel with deployment rather than before it.
In March 2023, an open letter signed by over 1,000 people — including prominent AI researchers, Elon Musk, and Apple co-founder Steve Wozniak — called for a six-month pause on training AI systems more powerful than GPT-4. The letter cited risks to society and said the pause was needed to develop safety protocols.
The pause did not happen. OpenAI, Google, Anthropic, and Meta all kept working. The letter's signatories disagreed significantly with each other about why a pause was needed and what should happen next. Within weeks, the conversation had moved on.
The pause letter episode illustrates something important: voluntary restraint is very hard to achieve in a competitive environment. Even people who agreed something was dangerous couldn't coordinate a slowdown, because each lab feared that only they would stop while others kept going.
This is exactly why external governance — rules that apply to everyone equally — is theoretically valuable. If all labs are required to pause, no one loses a competitive advantage by pausing. But getting to that rule requires international agreement, which brings us back to the problems in Lesson 1.
You can now see something that gets lost in headlines: the AI race isn't primarily about greed or recklessness. It's about structure. The incentives built into competition — between companies, between nations — push toward speed even when individuals inside those organizations would prefer to go slower. Understanding this changes how you evaluate calls for "just be more responsible." Responsible behavior is hard when the structure punishes it.
If voluntary restraint doesn't work, and binding international agreements are politically very hard to achieve, what is left? Some researchers argue for unilateral slowdowns — one country or company deciding to go slower regardless of what others do, as a moral statement and a demonstration that it's possible. Others argue this is naive: you just fall behind while others take the risks anyway, and the world doesn't become safer.
Who should bear the cost of slowing down, if anyone should? The companies that would lose market share? The countries that would fall behind? Or is the cost of not slowing down — potential catastrophic risk — larger than any of those costs? There is no consensus on this. The people arguing about it are serious, informed, and genuinely uncertain.
Two hypothetical countries — Aldoria and Ventis — are both developing powerful AI systems. Aldoria announces a voluntary six-month safety review. Ventis keeps going. You need to analyze what happens next and what, if anything, could have been done differently.
Your advisor will challenge whether your analysis correctly identifies causes versus symptoms, and whether your proposed solutions actually address the structural problem.
In October 2024, the UK government opened the doors of the world's first national AI Safety Institute — a government body whose entire job is to evaluate frontier AI models for dangerous capabilities before they are released to the public. The Institute, led by Yoshua Bengio (one of the three researchers credited with creating modern deep learning) on its scientific advisory board, started doing something no government had done before: actually testing AI systems for dangerous behaviors like the ability to help create bioweapons or assist cyberattacks.
The US launched its own AI Safety Institute shortly afterward, housed within the National Institute of Standards and Technology (NIST). Within months, the UK and US AI Safety Institutes had signed a formal agreement to share testing methodologies and coordinate evaluations. South Korea and Canada announced they were building their own institutes. A model was emerging — not a single global body, but a network of national institutions learning to work together.
It was the most concrete governance infrastructure the world had produced. It was also, critics noted, entirely voluntary. Companies shared models with these institutes when they chose to. And then in 2025, the US Institute was significantly cut during the Trump administration's federal restructuring. The fragility of the system became visible almost immediately.
When people say "AI governance," they often imagine a single powerful body — a kind of "UN for AI" — making binding rules that everyone follows. That doesn't exist. What exists instead is a complex, overlapping set of organizations with different powers, different mandates, and different relationships with each other. Understanding this landscape is what lets you evaluate any governance proposal clearly.
National AI Safety Institutes: The UK and US models above. Government-funded bodies that evaluate AI models, publish safety standards, and advise policymakers. Strength: they have real technical capacity and can do actual evaluations. Weakness: they're funded by governments that can cut them, and participation by AI companies is currently voluntary.
The United Nations: The UN Secretary-General's office released a report in 2023 calling for a new international AI body. The UN's existing agencies — like the International Telecommunication Union (ITU) — have started working on AI standards. But the UN works by consensus, meaning any one of its 193 member states can block action. Historically, this makes binding agreements very slow. The UN is better at building norms (shared expectations about behavior) than hard rules.
The G7 and G20: Groups of the world's major economies that meet regularly and make joint statements. The G7 AI Principles (Hiroshima AI Process, 2023) set out a framework for responsible AI that member countries agreed to promote. But again: principles are not laws. They don't bind anyone. They're expressions of shared values that may — or may not — get translated into actual policy.
Technical Standards Bodies: Organizations like the International Organization for Standardization (ISO) and the National Institute of Standards and Technology (NIST) develop technical standards — precise specifications for how things should be built or tested. These are less politically exciting but often more practically powerful. When ISO publishes a standard for AI risk management (which it did in 2023), companies that want to sell in international markets have a strong incentive to comply even without a law requiring it.
The most frequently proposed model for global AI governance is the International Atomic Energy Agency — the IAEA. Founded in 1957, the IAEA is a UN body that simultaneously promotes peaceful uses of nuclear energy and monitors countries for nuclear weapons development. Its inspectors physically visit nuclear sites. Its reports can trigger UN Security Council action. It has real teeth.
Several serious proposals have called for an "IAEA for AI." The idea: an international body with the authority to inspect AI development facilities, require transparency about the most powerful models, and flag dangerous systems. The Biden administration supported the idea in 2023. Several European governments expressed interest.
The counterarguments are also serious. Nuclear materials are physical and verifiable — inspectors can measure uranium enrichment. AI training runs are computational and invisible. A lab could run a secret training job in the cloud with no physical footprint to inspect. The verification problem that makes nuclear governance possible is much harder for AI. Additionally, the IAEA's authority comes from the fact that building a nuclear bomb is fantastically expensive and requires rare materials. AI requires only electricity and data — which are everywhere.
This doesn't mean the IAEA model is useless as a reference. It means the AI version would need to rely more on software monitoring, self-reporting, and whistleblower protections — and less on physical inspection. Whether that's sufficient is an open question.
Every functioning international institution was considered impossible before it existed. The IAEA, the World Trade Organization, the International Criminal Court — all faced predictions that powerful nations would never accept that kind of constraint on their sovereignty. They were partly right: the US still hasn't ratified the ICC's founding treaty. But partial institutions, imperfect institutions, still shape behavior. The question isn't "perfect governance or nothing" — it's "what can we build that's better than what we have?"
Here's something that gets underappreciated in debates about AI governance: institutions don't need perfect enforcement to matter. What they do well is build shared definitions, create accountability norms, and give responsible actors cover to do the right thing.
When a safety standard exists, a company that wants to move carefully can point to it and say: "We're complying with international standards." That gives internal safety advocates something to cite. It gives investors something to evaluate. It gives journalists something to compare companies against. This soft accountability can shift behavior even without legal penalties.
The UK AI Safety Institute's evaluations work this way. No law requires companies to submit models for testing. But when OpenAI, Google DeepMind, and Anthropic all agreed to do so, it created an expectation. A company that refused would stand out and face questions about what it was hiding. The norm did work that a law hadn't yet been written to do.
You're now in a position to evaluate AI governance proposals with real nuance. When someone announces a new AI safety body, you can ask: Does it have enforcement power or just advisory power? Is participation mandatory or voluntary? Who funds it, and can that funding be cut? Who does it have authority over — just domestic actors or international ones too? These questions cut through the press releases.
Every headline about a new AI safety initiative, framework, or agreement is now legible to you in a way it wasn't before. You can classify it: Is this a law with enforcement, a voluntary standard, a declaration of principles, or a research body? That classification tells you how much it will actually change behavior. Most people reading those headlines can't make that distinction. You can.
The institutions described in this lesson were largely designed by the world's most powerful countries — the US, UK, EU, and their allies. The standards they produce reflect their priorities, their threat models, and their values. Countries that weren't in the room when these standards were written — most of Africa, much of Latin America, South and Southeast Asia — now have to decide whether to adopt frameworks they had no role in shaping.
Is that the right way to build global AI governance? If the alternative is no governance at all — because getting everyone in the room first would take decades — maybe imperfect, powerful-country-led governance is better than nothing. Or maybe governance designed without the global majority will fail to address their actual concerns and will never be seen as legitimate. Both of these are defensible positions. The people making these decisions are genuinely uncertain.
A new international body has been announced: the "Global AI Accountability Council" (GAAC). It will issue safety guidelines, accept voluntary model submissions from AI companies, and publish annual reports. It is funded by membership fees from participating countries. The US, EU, and China are all listed as founding members. No enforcement powers are mentioned.
Your advisor wants your assessment: how strong is this institution, and what specific gaps concern you most?
In May 2023, the UN Secretary-General António Guterres stood before an audience of diplomats and made a striking request. He proposed creating a new international AI body modeled on the International Atomic Energy Agency and the Intergovernmental Panel on Climate Change — a body that would both assess risks and set international norms. He noted that the people most affected by AI's future were not in the room: they were young people in countries that had no seat at the table.
The proposal went into committee. It has not emerged as a functioning institution. But it opened a question that has not closed: if not a UN body, then what? Every alternative — national institutes, industry coalitions, technical standards bodies — has gaps. The governance architecture of AI in 2035 has not been written yet. The debates happening right now will determine its shape. The people who are learning this material today are precisely the people who will be in those debates — as voters, as engineers, as lawyers, as policymakers, as journalists, or as citizens who hold all of those people accountable.
Based on what serious researchers and policymakers are actually discussing, here are three plausible directions for global AI governance — each with real trade-offs.
Scenario A: The Network Model. No single global body. Instead, national AI safety institutes cooperate — sharing testing data, coordinating standards, and running joint evaluations. The US, UK, EU, Japan, South Korea, and others form a kind of club with shared practices. China and others may have parallel institutions that occasionally exchange information. This is closest to where things are heading now. Its strength: it's buildable. Its weakness: it leaves out most of the world and has no mechanism for situations where the major powers disagree.
Scenario B: The UN Agency Model. A new international body — perhaps growing out of existing UN structures — is given a formal mandate to evaluate frontier AI, publish public assessments, and flag systems that pose international risks. Countries retain sovereignty but agree to the evaluation process. This would require a treaty, which requires Senate ratification in the US (historically difficult), buy-in from China, and a functioning consensus among deeply competitive nations. Its strength: legitimacy. Its weakness: political feasibility and the verification problem.
Scenario C: The Market-Led Model. Governance happens primarily through market forces. Insurance companies require AI systems to pass safety evaluations before issuing liability coverage. Enterprise customers demand compliance with voluntary standards as a condition of purchase. Investors price safety performance into valuations. Governments focus on liability law — if an AI system causes harm, the deployer is legally responsible. This model requires no international treaty. Its weakness: market forces tend to reward profit, not safety, and don't protect the people with the least market power.
In 2024, a survey of AI governance forums found that the Global South — countries in Africa, Latin America, South and Southeast Asia — accounted for roughly 12% of participants in major AI governance discussions, despite representing about 70% of the world's population. This is not an accident. It reflects which countries have the most AI companies, the most researchers, and the most diplomatic resources to engage in specialized international forums.
But who has the most to gain or lose from AI is a different question. Many of the applications being developed — AI for crop disease detection, for early disease surveillance, for language translation in underserved languages — are most consequential for populations in lower-income countries. And many of the harms — algorithmic discrimination in credit and hiring, AI-powered surveillance sold to authoritarian governments, automated systems replacing low-wage jobs with no social safety net — land hardest on populations with the least political power.
The institutions being built right now will set the norms for decades. If those institutions are designed without input from most of the world's population, they will reflect a subset of values and concerns. Whether that's better than no institutions is genuinely debatable. What's not debatable is that the design process is happening now, and who's in the room matters.
International governance involves many types of participation: heads of state at summits, career diplomats in UN committees, technical experts drafting standards, civil society organizations commenting on proposed rules, journalists whose reporting shapes public pressure, lawyers who interpret and enforce agreements, and researchers whose work gives policymakers the information they need to act. You don't have to be a diplomat to influence governance. You have to understand what's happening well enough to contribute something useful to those processes — and to hold the people in them accountable.
Here, concretely, is what serious people in this field say needs to be built and hasn't been yet:
A shared definition of "frontier AI": Governance frameworks keep referring to the most powerful AI systems as "frontier AI" or "advanced AI" — but there is no agreed international definition of which systems qualify. Without a shared definition, there's no way to know what the rules actually apply to.
Verification mechanisms: A way to confirm that AI systems have actually been tested against claimed safety standards, and that reported training compute is accurate. This requires technical tools that don't fully exist yet — software auditing methods, hardware tracking systems, and agreed measurement protocols.
Liability frameworks: Clear legal rules about who is responsible when an AI system causes harm. If an AI medical diagnostic system gives wrong advice and a patient is harmed, who is liable — the hospital that deployed it, the company that built it, or the company that trained the underlying model? Most legal systems have not caught up to this question.
Inclusive standard-setting: Mechanisms for countries that lack major AI companies or large research budgets to meaningfully participate in shaping the standards that will affect their populations.
These are not unsolvable problems. They are work. Hard, slow, mostly unglamorous work that will be done by people who understand both the technology and the political and legal systems around it. Some of those people have not started their careers yet.
You now understand the full landscape: why global AI governance is structurally harder than previous international challenges, what institutions exist and what they can and can't do, what the race dynamic is and why it persists, and what the specific gaps are in current frameworks. That understanding is not common. Most adults making decisions about AI — in government, in business, in journalism — don't have all of this clearly in their heads. You do. That's not a small thing. The question is what you build with it.
The hardest governance question in this field is not technical. It's this: AI governance requires powerful countries to accept constraints on something they see as central to their economic and military power. History suggests that happens very slowly, and usually only after a crisis.
But with some categories of AI risk — systems that could enable the creation of biological weapons, or that could operate faster than any human institution can respond — waiting for a crisis may mean waiting until after the harm has already been done.
So: is it possible to build adequate governance before a catastrophe, or does history tell us we will need the catastrophe first? And if you believe we can do better than history suggests, what would need to be different this time?
That question is live. It doesn't have a consensus answer. The debate is happening now, and the people contributing to it include people your age who got serious about understanding this early.
It's 2027. A major international AI incident has occurred — a powerful AI system was used to generate convincing disinformation that destabilized a national election. No existing framework had authority to respond. You've been asked to propose a governance architecture for the next ten years that addresses this specific failure while being politically achievable.
Your advisor has been through the UN system and seen three previous governance frameworks collapse. She will push you on feasibility, not just idealism.