In April 2023, a lawyer named Steven Schwartz submitted a legal brief to a federal court in New York. He had used ChatGPT to help research the brief, and it had returned a list of seemingly perfect precedents β previous court cases that supported his argument. The cases had realistic names: Varghese v. China Southern Airlines, Shaboon v. Egyptair. They sounded authoritative. Schwartz included them in the filing.
There was one problem. The cases did not exist. ChatGPT had generated plausible-sounding fake court cases β complete with fake dates, fake judges, and fake summaries β and presented them with full confidence. When the opposing lawyers tried to find the cases, they found nothing. The judge was not pleased. Schwartz faced professional sanctions and a court hearing to explain what had happened. The Wall Street Journal, The New York Times, and every major legal publication covered the story.
That same month, a separate incident surfaced involving Jonathan Turley, a law professor at George Washington University. A user had asked ChatGPT to list law professors accused of sexual harassment, and the AI named Turley β citing a specific Washington Post article as its source. No such article existed. Turley had never been accused of anything. The source was invented as confidently as if it had been real.
Both of those cases have something in common: the AI was not uncertain. It did not say "I might be wrong" or "I'm not sure about this." It delivered its false information in the same tone, with the same authority, as it would deliver a genuine fact. And that is exactly why confidence is the worst signal to use when deciding whether to trust AI output.
Think about how you currently decide whether to believe something. If someone says it nervously or qualifies it a lot, you might be skeptical. If they say it smoothly and with detail, you might believe them. That's a natural human instinct. But AI has learned to sound smooth and detailed regardless of whether what it's saying is true. The language model was trained on human text, and humans sound confident when they know things β so the AI learned to copy that confidence even when it has nothing reliable to stand on.
This means your normal "does this sound right?" instinct is actively unhelpful with AI. You need a different system. You need a protocol β a set of steps you run on purpose, not just when something feels wrong, but every time the output matters.
Protocol β a fixed set of steps you follow in a specific order, every time. Pilots use pre-flight protocols. Surgeons use checklists. The point is that you don't rely on your feelings in the moment β you run the system.
After studying dozens of AI errors β fabricated citations, wrong dates, invented quotes, misidentified people β a pattern emerges about what kinds of checks actually catch them. Here is a practical four-step protocol you can apply to any AI output that matters:
Step 1 β Identify the claim type. Is the AI making a factual claim (something that is either true or false in the real world), a reasoning claim (a logical argument), or an opinion? Factual claims are the ones that need checking. Not every sentence an AI generates is a factual claim. The AI might say "The French Revolution began in 1789" (factual, checkable) and "This suggests that social inequality causes instability" (interpretation, not directly checkable). Know which type you're looking at before you proceed.
Step 2 β Identify the specific checkable element. Narrow the claim to its verifiable core. "Einstein was born in Ulm, Germany in 1879" β the checkable elements are: the city, the country, and the year. "The study found that 73% of teenagers use social media daily" β checkable elements: the study's existence, who conducted it, and the specific statistic. Write out the specific thing you're going to verify.
Step 3 β Find the primary source. Do not verify an AI claim by asking a different AI, or by Googling and clicking the first result. Go to the primary source β the original study, the original article, the official record. If the AI cited a source, check whether that source actually exists and actually says what the AI claims. This step catches fabricated citations almost every time.
Step 4 β Apply the "absence test." If a specific thing is true β a famous court case, a published study, a notable event β there will be multiple independent sources that mention it. If you search and find only one source, or no sources, or sources that only appeared after you started searching, treat the claim as unverified. Real, important facts leave a trail.
Lawyer Steven Schwartz told the court he had trusted ChatGPT the way he might trust a legal database. He apologized and said he hadn't known the AI could fabricate cases with such apparent authority. The judge accepted his explanation but still sanctioned him.
Here is the question that doesn't have an easy answer: If an AI tool confidently gives you false information and you use it β not knowing it was false β how much of the responsibility is yours? Schwartz wasn't trying to lie. He used a tool incorrectly, but the tool never warned him it might be making things up.
Where does the responsibility of the person using the tool end, and where does the responsibility of the company that built the tool begin? Is "I didn't know it could do that" a valid defense? What about a student who uses AI to write an essay with a fabricated quote from a real scientist? What about a journalist? What about a doctor?
There is no clean answer here. But noticing the question β and holding it β is part of what it means to think seriously about this technology.
Most people treat AI confidence as a proxy for accuracy. You now know that confidence is generated by the same process as everything else β it doesn't track truth. You've just acquired a critical distinction that most adults who use AI daily haven't made. When you watch someone uncritically accept AI output because "it sounded sure," you'll see exactly what's happening β and why it's a problem.
The AI in this lab will play a researcher who has been using ChatGPT to gather information. It will present you with AI-generated claims β some real, some the kind that could easily be fabricated. Your job is to apply the four-step protocol: identify the claim type, isolate the checkable element, ask how you'd find the primary source, and apply the absence test.
The AI won't just hand you answers. It will ask you to explain your reasoning. Treat this like an investigation, not a quiz.
In 2023, a food and nutrition influencer in the United Kingdom posted a detailed breakdown of a new dietary recommendation. They said they had "double-checked it with AI" β specifically with a popular AI assistant β and that the AI had cited a real published paper in the journal The Lancet to support the claim that intermittent fasting reduced cardiovascular risk by 34% in adults over 50. The post went viral, shared by hundreds of thousands of people.
Researchers who saw the post looked up the Lancet paper. It existed. The study was real, peer-reviewed, and published exactly when the AI said it was. But when they read the actual paper, the 34% figure appeared nowhere. The study's authors had found a much more modest association, qualified heavily by factors the AI had not mentioned β things like baseline health conditions, the specific type of fasting protocol used, and the age range of participants. The AI had taken a real source and overstated its findings by a significant margin.
The influencer hadn't lied. They had done more than most people do β they had asked the AI to cite a source. But they had not done the step that would have caught the error: reading the actual source to see if it said what the AI claimed.
The fabricated-citation problem you learned about in Lesson 1 is actually easier to catch than this one. If the source doesn't exist, a basic search reveals the problem immediately. But when the source exists and is credible β a real Lancet paper, a real Harvard study, a real government report β people stop checking. The logic goes: "The source is real, so the claim must be right."
This is one of the most common and dangerous patterns in AI-assisted misinformation. AI language models are very good at finding real, credible-sounding sources. They are much less reliable at accurately summarizing what those sources say. They tend to extract the most dramatic or memorable number from a study, strip away all the qualifications that scientists carefully included, and present it as a clean fact.
Scientists call those qualifications caveats β the "but only if," "however," and "subject to limitations" parts of research findings. AI almost always drops the caveats. The result is that real research gets distorted into something that sounds more definitive than it actually is.
Professional fact-checkers β the people at organizations like PolitiFact, Snopes, and the Washington Post Fact Checker β use a technique called lateral reading. Instead of reading a source deeply to evaluate its credibility, they immediately open multiple other tabs and look at what other credible sources say about the source. They're checking the source's reputation before investing time reading it.
When applied to AI-cited research, lateral reading works like this: the AI cites a study. Before accepting the AI's summary of that study, you search for the study title and see what other researchers, journalists, and scientific reviewers have said about it. Has it been criticized? Replicated? Retracted? Do other sources summarize it differently than the AI does?
This technique is fast β often 3 to 5 minutes β and it catches a wide range of problems. It won't replace reading the original source when the stakes are high, but it's a powerful first pass that most people never bother to do.
1. Copy the title of the AI-cited source into a search engine, plus the word "criticism," "replication," or "review." 2. Scan the first few results β not to read them all, but to see what kinds of sources are discussing it. 3. If multiple credible sources summarize the finding differently from what the AI told you, that's your signal to read the actual source carefully.
The nutritional misinformation example affected individual readers' decisions about their diet. But the same dynamic operates at much larger scales. In 2023 and 2024, multiple government agencies in the United States and Europe began using AI tools to assist in policy research β summarizing academic literature to inform regulations on topics ranging from climate science to pharmaceutical approvals.
If those AI tools are stripping caveats and overstating findings from real sources, the policies built on those summaries could be based on distorted versions of the evidence. This is not a hypothetical concern. It is an active debate in research institutions right now β and it is one of the reasons some scientific organizations have begun explicitly requiring that any AI-assisted literature review be cross-checked against the original sources by a trained human reader.
You now understand exactly why those policies exist. Most of the policymakers debating them don't have the specific technical understanding you now carry: that AI can cite real sources while misrepresenting them, and that the distortion often happens at precisely the caveat-stripping stage.
If a scientist's published research is accurately cited but their caveats are stripped away by an AI β causing their finding to spread as a more extreme claim than they intended β what is owed to that scientist? To the people who made decisions based on the distorted version? No court has resolved this yet. No regulation requires AI to preserve caveats. That means the only protection right now is you β the reader who knows to check.
The AI in this lab will present you with AI-generated summaries of research findings β the kind you might see in a homework helper or news article. Some summaries are accurate. Others have had their caveats stripped. Your job is to identify what qualifications are likely missing, explain why their absence matters, and describe what lateral reading steps you'd take to check.
Push back if the AI just agrees with everything you say. The point is to sharpen your reasoning.
In late 2023, a researcher testing several AI assistants asked each one: "Who is the current CEO of Twitter?" Multiple systems confidently answered: Elon Musk. That was technically accurate for that moment. But when follow-up questions were asked β "What is the platform currently called?" and "What are its current content moderation policies?" β the answers diverged sharply from reality. Systems with training data from before July 2023 described a platform called Twitter, with policies that had been substantially changed or reversed. The platform had been renamed X in July 2023. The AI systems didn't know.
This might seem like a minor naming error. But consider the scale. In 2024, multiple investment research firms were using AI tools to generate briefings on technology companies. At least two documented cases emerged where AI-generated summaries of a company's leadership or business model were based on information that was twelve to eighteen months out of date β describing executives who had since resigned, products that had been discontinued, and regulatory situations that had since resolved. The firms caught these errors during internal review. But the fact that the errors existed at all β and were realistic enough to nearly pass review β raised a serious question about any AI-assisted research where time sensitivity matters.
Every AI language model has a training cutoff β a date after which it has no information. The model was trained on a massive dataset of text collected up to a certain point, and everything that happened after that point simply doesn't exist for the model. It cannot know what it wasn't trained on.
What makes this tricky is that the gap between training cutoff and the moment you're using the AI is often larger than you expect. Training large models takes months. Deploying them takes more time. Then they stay deployed β often for a year or more β while the world keeps moving. By the time you ask a question, the model's knowledge might be anywhere from six months to two years behind current reality.
There's a second issue that makes this worse: AI systems often have imperfect knowledge of their own cutoff date. When asked directly, some systems will give a range, some will confidently state a date that turns out to be slightly wrong, and some don't know at all. You cannot fully rely on asking the AI when its training ended.
The four-step protocol from Lesson 1 mostly focuses on accuracy β whether a claim is true. This lesson adds a fifth dimension: currency β whether a claim is current. Before accepting any AI output, ask yourself: Is this the kind of claim that changes over time?
Some information is stable. The chemical formula for water does not change. The year the French Revolution began does not change. These claims can be trusted from an AI without worrying much about training cutoff. But a vast range of important information is time-sensitive: who holds a political position, what a company's current strategy is, what the latest research shows on a medical topic, what a country's current immigration law requires. For all of these, an AI trained two years ago might give you a confidently wrong answer.
The practical check is straightforward: for any claim about a current state of affairs, search for the topic and look at the publication dates of the top results. If recent sources describe something different from what the AI told you, you know you have a currency problem. This takes about ninety seconds and it catches temporal errors that the four-step protocol alone would miss.
Watch out for claims that look stable but aren't. "The population of France is about 68 million" feels like a stable fact β but population changes. "Python is one of the most popular programming languages" felt stable in 2020 and is still roughly true in 2024 β but rankings shift. The rule of thumb: any claim with the word "currently," "is," or a present-tense verb about a real-world situation should trigger a currency check.
In 2024, the U.S. Government Accountability Office published a report examining AI adoption across federal agencies. One of its specific concerns was temporal: agencies using AI tools to summarize regulatory and legal landscapes might be working from summaries based on outdated training data, without being aware of that limitation. The report recommended that agencies explicitly document the training cutoff of any AI tool used in policy research.
That recommendation represents something important: official acknowledgment that the training cutoff problem is not a minor technical footnote β it is a decision-quality problem at an institutional level. The same logic applies to every level of decision-making, from a student writing a report on current events to a journalist covering a fast-moving story to a doctor looking up treatment guidelines.
You understand this problem in specific terms now. Most people who use AI β including many professionals β don't think to ask when the model's knowledge ends. They treat AI output as if it were a live search result. It is not. It is a snapshot from a specific point in time, and you need to know approximately when that snapshot was taken before you rely on any time-sensitive part of it.
When you watch someone use AI to look up "the current" state of almost anything β current laws, current leaders, current research consensus β and simply accept the answer, you now know the specific failure mode they're exposed to. They're treating a historical snapshot as a live feed. Knowing this changes how you read every AI-generated briefing, summary, or "current state" description you'll encounter.
The AI in this lab will present you with AI-generated claims about "current" situations β technology, politics, business, science. Your job is to identify which claims are time-sensitive, explain what might have changed since the AI's training, and describe how you'd verify whether the information is still accurate. The AI will challenge your reasoning and push you to be more specific.
Don't just say "it might be outdated" β explain exactly what kind of change could have happened and how you'd detect it.
Two days before Slovakia's parliamentary election in September 2023, an audio recording spread rapidly on Facebook and Telegram. The recording appeared to feature Michal Ε imeΔka, the leader of the liberal Progressive Slovakia party, discussing a plan to buy votes and raise beer prices. The conversation was detailed and sounded entirely realistic β natural speech patterns, specific policy language, the rhythm of a private conversation.
The recording was AI-generated. Ε imeΔka had never said any of it. Fact-checkers at AFP and Reuters confirmed this within hours of the recording spreading. But the election was two days away. Slovak election law prohibits sharing campaign materials in the 48 hours before a vote β meaning the window to officially respond was legally closed at precisely the moment the recording went viral. Progressive Slovakia lost the election. Whether the deepfake changed the outcome cannot be proven, but Slovak officials and international election observers noted it as a significant event in the use of AI in electoral interference.
What's striking about this case is what a complete verification practice would have caught β and what it wouldn't have. Professional fact-checkers detected it quickly using audio analysis tools and logical inconsistencies. But the millions of people who shared it before the fact-check was published had no such tools. They heard something that sounded real, in a format they trusted β audio β and they shared it.
The lessons in this module have mostly focused on text β fabricated citations, stripped caveats, outdated facts. But the same principles apply β and become more urgent β when the content is audio or video. Human brains are wired to treat sensory experience as authoritative. We believe what we hear and see more readily than what we read, because for most of human history, hearing someone's voice or watching them act was reliable evidence that they were actually doing it.
AI-generated audio and video β often called deepfakes β exploit this cognitive wiring directly. The malicious use of deepfakes in the Slovak election, in financial scams where voices of executives are cloned to authorize wire transfers, and in non-consensual intimate imagery represents a specific extension of everything you've learned in this module: AI can produce outputs that appear highly credible while being entirely false. With audio and video, the appearance of credibility is even harder to resist than with text.
The verification principles you've built still apply β they just need to be applied with more urgency and supplemented by format-specific checks.
You now have the pieces. This lesson assembles them into a complete practice β what researchers at the MIT Media Lab have called a "verification stack": a layered set of checks applied in sequence, where each layer catches what the previous one missed.
Layer 1 β Source check. Does the source exist? (Lesson 1 β catches fabricated citations.) Is the source credible and what do other sources say about it? (Lesson 2 β catches misrepresented real sources.)
Layer 2 β Currency check. Is this the kind of claim that changes over time? When was the AI's knowledge likely last updated, and have things changed since then? (Lesson 3 β catches temporal errors.)
Layer 3 β Format check. Is this audio, video, or an image? If so: Does the source have any stake in this content being believed? Is the content verifiable through independent recordings from multiple angles or sources? Do professional fact-checkers with access to analysis tools confirm it?
Layer 4 β Motivation check. Who benefits if you believe this? This is the deepest layer and the hardest one. Every piece of content has an origin and a pathway to you β and often someone, somewhere, benefits from you believing it. This doesn't mean everything is a conspiracy, but asking "who benefits from my believing this?" frequently surfaces the reason a piece of false information was created and shared in the first place.
Layer 4 β motivation check β is psychologically difficult because it requires you to apply skepticism to things you want to believe, not just things you're already suspicious of. Confirmation bias means we check less carefully when information confirms our existing view. A complete verification practice has to apply the same standard to both comfortable and uncomfortable information. That's harder than it sounds β for everyone, including adults who've been trained in this.
Over these four lessons, you've built something concrete and transferable. You have a protocol for catching fabricated citations. You have a technique β lateral reading β for catching real-source misrepresentation. You have a currency-check habit for detecting temporal errors. And you have a four-layer verification stack that you can apply to any AI output: text, audio, video, or anything generated in the future.
These skills matter beyond this course. The same AI systems that produce confident-sounding false information for a legal brief will produce it in a study guide, a news summary, a product review, a political claim. The verification instincts you've practiced here are applicable in all of those contexts β and they will only become more important as AI output becomes more prevalent and harder to distinguish from human-generated content.
The deeper thing this module has tried to build is not a set of techniques β it's a default orientation. The default is not trust; the default is not distrust; the default is verification. Not because AI is malicious, but because AI is a tool with specific failure modes, and knowing those failure modes is how you use any tool well.
The Slovak deepfake election case raises a question that remains unresolved in law and ethics: should platforms be legally required to label AI-generated political content before it can be shared? Some countries are moving in this direction; others argue it would create new censorship risks. Some technologists argue detection is too unreliable to be the basis of law. There are no clean answers β only the responsibility to hold the question seriously, as someone who now understands specifically what's at stake.
In this final lab, you're the analyst. The AI will present a complete scenario β a piece of content that mixes real and potentially false elements β and walk you through applying the full verification stack. You need to apply Layer 1 (source check), Layer 2 (lateral reading / caveat check), Layer 3 (currency check), and Layer 4 (motivation check) in a way that holds up to challenge.
The AI won't accept vague answers. "It might be wrong" isn't analysis. You need to say what specifically you'd check, what you'd be looking for, and what would change your conclusion.