In February 2023, Steven Schwartz, a lawyer in New York City with thirty years of experience, faced a tight deadline. His client had sued an airline, and Schwartz needed to find legal cases — prior court decisions — that supported his argument. He turned to ChatGPT and asked it to find relevant cases.
ChatGPT delivered. It produced six detailed case citations, complete with court names, dates, docket numbers, and summaries of the rulings. The language was precise, authoritative, professional. Schwartz filed the document in federal court.
The problem: every single case was fake. Not slightly wrong. Not paraphrased. Completely invented. Mata v. Avianca. Varghese v. China Southern Airlines. Zicherman v. Korean Air Lines. None of them existed anywhere in any legal database on Earth.
The judge, P. Kevin Castel, was bewildered. When he ordered Schwartz to produce copies of the cases, Schwartz went back to ChatGPT — and the AI confirmed the cases were real. It even provided fake quotes from fake judges. Schwartz filed an affidavit explaining what had happened. Judge Castel fined him $5,000. The case became front-page news worldwide.
What made this catastrophic wasn't just that ChatGPT was wrong. It was that ChatGPT was confidently, fluently, elaborately wrong — and when asked to double-check, it doubled down.
Here is the first thing you need to understand, and it will change how you read every AI output for the rest of your life: an AI language model does not know what it doesn't know.
When ChatGPT, or any similar system, generates text, it is not looking up facts in a database. It is predicting which words are most likely to come next, based on patterns it learned from billions of documents. It learned that legal briefs contain case names, docket numbers, and citations — so when asked for legal cases, it produces text that looks like legal cases. The pattern is correct. The content is invented.
Researchers who study this problem have a name for it. They call it hallucination — a word borrowed from psychology, where it means perceiving something that isn't there. When an AI hallucinates, it generates text that is fluent, grammatically perfect, internally consistent, and completely fabricated.
The reason this is so dangerous is that the writing style gives you no clue that something is wrong. A human expert who doesn't know something will usually say "I'm not sure," or their answer will seem hesitant. An AI trained on millions of confident, polished documents produces confident, polished text even when it has no idea what it's talking about.
Think of it like this: imagine a student who has read thousands of history essays, but never actually studied history. If you asked them "What was the main cause of World War I?" they might produce a paragraph that sounds exactly like a history essay — with dates, names, and connecting arguments — even if they just assembled the pattern without knowing whether any of it is accurate. That's the closest human analogy to how this works.
Detective work means reading outputs for clues that something might be fabricated. There are three specific patterns to watch for — not because they prove an output is wrong, but because they should trigger your verification instinct.
Signal One: Specific but unverifiable details. Hallucinated facts often come with very precise-sounding details — exact dates, statistics with decimal places, full names of obscure people. Precision sounds credible, but fabricated precision is one of the most common hallucination patterns. When an AI gives you a statistic like "studies show that 67.3% of users report..." and there's no source attached, that specificity is a red flag, not a reassurance.
Signal Two: The double-down on challenge. Steven Schwartz asked ChatGPT to confirm the cases were real. It confirmed them. This is a known and documented failure mode. Current AI systems are trained partly on human feedback that rewards agreeable, helpful-sounding responses. When you push back on an AI and it confidently restates its original claim rather than expressing uncertainty, that is not evidence the claim is correct — it is evidence the system is optimizing for sounding helpful.
Signal Three: No acknowledgment of the limits of training. All current AI language models have a training cutoff — a date after which they have no information. Events, publications, and changes that happened after that cutoff don't exist in the model's world. If you ask about something recent and the AI answers without noting that its information might be outdated, that silence is a warning sign.
A good detective doesn't disbelieve everything. They hold belief in suspension until evidence supports it. You don't reject AI outputs by default — you verify before you rely on them, especially when the stakes are high. Schwartz's mistake wasn't using AI. It was trusting AI without verification in a context where being wrong had serious consequences.
Here is a question with no clean answer. After the Schwartz case broke, some legal experts said the lesson was simple: lawyers should never use AI for legal research without verification. But others pointed out that the legal system already has a documented problem with access. Hiring a full research team to verify every AI output costs money that most defendants cannot afford. If AI tools make legal research faster and cheaper, banning them from courtrooms could end up hurting the people who most need affordable legal help.
So: who should bear the cost of AI verification? The lawyer, who might pass it to the client? The AI company, through some kind of liability system? The court system, by creating verified legal AI databases? Or should AI never be used in high-stakes professional contexts at all?
There is no consensus. Legal scholars, AI researchers, and courts are still arguing about it. The point isn't to solve it here — the point is that you can see why the question matters, and why it doesn't have a neat answer.
You now understand something that most adults who use AI daily have not actually internalized: confident tone is not evidence of accuracy. AI outputs are not more reliable when they sound more certain. The system has no internal flag that says "I'm making this up" — it generates the same fluent prose whether it's accurately summarizing Shakespeare or inventing a court case that never existed.
This doesn't mean AI is useless. It means AI requires a specific kind of reader — one who treats outputs as a starting point for verification, not an endpoint. You are now that kind of reader. Most people who click "copy" on an AI answer and paste it into a document are not.
Every time you see someone cite an AI-generated statistic without a source, or quote an AI without checking it, you will recognize something they missed. That's not superiority — it's a skill. And skills can be shared.
Below you'll find a paragraph that was supposedly generated by an AI assistant. Your job is to identify the warning signs that something might be hallucinated — then explain your reasoning to the AI partner. They won't let you off easy with vague answers.
Tell the AI: Which specific signals in this paragraph would make a detective suspicious? What would you do to verify or debunk it?
In June 2023, a graduate student at University of Southern California named Anna Sokol was working on a thesis about social media and mental health. She used an AI writing assistant to help find supporting literature. The assistant returned eleven academic citations — journal names, volume numbers, page ranges, DOIs (those digital object identifiers that look like doi.org/10.xxxx).
Sokol submitted her draft to her faculty advisor, Dr. Patricia Chen. Chen, a researcher with decades of experience navigating academic databases, did something Sokol hadn't: she tried to look up three of the citations in PubMed and Google Scholar. None existed. The DOI numbers were formatted correctly but led nowhere. The journal names were real — the papers were fake.
What made this case particularly instructive was that the AI had not used obviously invented journal names. It used real journals — Journal of Adolescent Health, Computers in Human Behavior — paired with invented authors, invented volume numbers, and invented titles that sounded exactly like the kind of research that would appear in those journals. If you searched the journal name, the journal existed. Only if you searched the specific paper did you discover the absence.
Sokol told a reporter: "It wasn't like it gave me a fake journal. It gave me a real journal with a fake article in it. I had no reason to doubt it." Her thesis was delayed by three months while she found legitimate sources. The advisor's instinct to verify saved what could have been an academic misconduct finding.
AI hallucinations about sources follow a consistent pattern, and once you know the pattern, you'll spot it faster. Here's what a convincingly fake citation tends to look like:
Real container, fake content. The AI uses a legitimate journal or publisher name — one that has genuinely published work in that area — but invents the specific article. This is more dangerous than a fully invented source because your initial check (does this journal exist?) passes.
Plausible but nonexistent authors. AI models often generate names that fit the demographic and specialty patterns of researchers in a field. A paper about machine learning might be attributed to "Dr. Wei Zhang" or "Dr. Sarah Kowalski" — both are names that plausibly belong to real researchers, but the specific person cited may not exist, or may exist but never wrote that paper.
Correctly formatted but dead DOIs. DOIs (Digital Object Identifiers) follow a specific format: 10.followed-by-numbers/and-a-suffix. AI models learned that format from millions of academic documents. They can generate a string that looks exactly like a valid DOI but resolves to nothing when you type it into a browser.
The verification move is simple but must be habitual: never cite a source you haven't actually opened. If a citation exists, you can find the actual document. If you can't find the document — not the journal, the document — the citation may be fake.
The real-container-fake-content pattern isn't limited to academic papers. It appears in virtually every domain where AI generates referenced information.
In journalism, AI tools have generated stories that quote real news organizations with fabricated headlines. A reporter verifying a story might check: does CNN exist? Yes. Did CNN report this? That second question is the one that exposes the hallucination, and it's the one that doesn't get asked.
In legal contexts — as in the Schwartz case — the AI used real court names (real jurisdiction, real procedural format) paired with invented case names and docket numbers. The Federal Rules of Civil Procedure existed. The cases did not.
In historical research, AI systems have generated accurate-sounding quotes attributed to real historical figures. Abraham Lincoln, Marie Curie, and Winston Churchill have all been given fabricated quotes by AI systems — quotes that fit their documented style and worldview closely enough to fool casual readers.
Step 1: Does the container exist? (Does this journal, news org, court, or person exist?) Step 2: Does this specific content exist within that container? Most people only do Step 1. The hallucination hides in Step 2.
The reason this matters beyond academic work: news articles, political arguments, and public health claims are increasingly generated or assisted by AI. When someone shares a post saying "according to the CDC, X..." the real question isn't whether the CDC exists — it's whether the CDC actually said X. Knowing to ask Step 2 is what separates an informed reader from someone being misled.
Here is the uncomfortable question the Sokol case raises. The AI that generated those fake citations was a product — a tool a company built and sold. Sokol was a graduate student doing academic work in good faith. When the fake citations almost made it into a published thesis, whose failure was that?
You might say: Sokol should have verified. But she was using the tool in the way it was designed to be used — as a research aid. The tool did not warn her that its citations might be invented. Most AI writing tools, as of 2023 and 2024, do not include prominent warnings that their citations require verification. Some include fine print. Most users don't read it.
Is it sufficient for AI companies to include a disclaimer in their terms of service? Or do they have an obligation to make the limitations unmissably clear — not buried in legal text, but visible at the moment the hallucinated output appears? How prominent would a warning need to be before responsibility shifts from user to platform?
These questions are being debated right now by regulators in the European Union, the United States, and the United Kingdom. The EU's AI Act, which passed in 2024, begins to address some of them. But "begins to address" is not the same as resolves.
You can now read a citation differently from most people. When someone shows you a source — whether it's a paper, a news article, or a quote from a historical figure — your first question is no longer just "does this source generally exist?" but "does this specific thing exist within that source?"
That's the two-step check. It takes thirty seconds with a search engine or a DOI lookup tool like doi.org. Thirty seconds is all that stood between Anna Sokol's thesis and academic misconduct. Most people skip it because the source looks real enough, and looking real enough has always been sufficient.
In a world where AI can generate convincing-looking sources in seconds, "looking real enough" is no longer sufficient. You know that now. That knowledge shapes how you read everything that cites a source — which is almost everything that matters.
You've been handed three citations from an AI-generated research report. Your job is to decide which ones you trust enough to use without verification, which ones need Step 2 verification, and what that verification would actually look like in practice.
Tell the AI: Rank these from most to least suspicious. Explain what specific Step 2 action you'd take for each. Defend your ranking.
In November 2022, a major consumer electronics company — CNET, one of the oldest and most-read technology news sites in the United States — quietly began publishing financial explainer articles written by an AI. The articles had bylines that read "CNET Money Staff" and appeared alongside human-written content with no obvious distinction.
In January 2023, the tech publication Futurism broke the story. Reporters examined the AI-written articles and found a pattern: many contained small but concrete numerical errors. One article stated that if you invest $10,000 at 3% annual interest for a year, you'd earn $10,300 in total — which is correct — but described this as "an increase of 300 dollars, or 3.3%." The error is subtle: 300 divided by 10,000 is 3%, not 3.3%.
Readers who weren't checking the arithmetic would never notice. But across dozens of articles, CNET's editors found more than 41 errors in content the AI had produced. Some were minor rounding errors. Others were more substantial misstatements about tax rules, interest calculations, and compound growth formulas.
CNET issued corrections and suspended the program. A senior editor publicly stated that the errors were "not acceptable for a publication built on trust." What made the story significant wasn't just the errors themselves — it was that the AI had been producing content that looked exactly like financial journalism, numbers included, and had been doing so for two months before anyone noticed. The numbers had the right format, the right units, the right order of magnitude. They were just subtly wrong.
There is an important distinction to make here, because it changes how you read AI-generated numbers. AI language models don't do arithmetic the way a calculator does. When a calculator adds 3% to $10,000, it computes. When an AI language model produces the result, it is predicting what a plausible-looking answer would be in this context — based on patterns in documents where similar calculations appeared.
Most of the time, that prediction is correct, because the training data was mostly correct. But for calculations involving specific percentages, compound interest, or unit conversions, the AI is not running the numbers — it is generating text that looks like someone ran the numbers. This distinction is subtle and consequential.
The reason CNET's errors were subtle — 3.3% instead of 3% — is that both numbers are plausible in the context of financial writing. The AI generated a number that felt right for the sentence. If the actual answer had been 47.3%, an error of 0.3 percentage points might never be noticed by most readers. But if the error compounds — as errors in financial calculations often do — the downstream effect on someone's actual financial decision could be significant.
There is also a related problem: invented statistics that sound like research findings. When an AI writes "studies show that X% of people..." it has often learned this phrasing pattern from real research documents — but it may be generating the number from pattern-matching rather than citing any real study. The phrase pattern is real. The statistic may be fabricated.
Once you know this, you can sort the numbers you encounter into three categories that require different levels of scrutiny.
Category 1: Arithmetic from stated inputs. If an AI does a calculation based on numbers you gave it, the risk is arithmetic hallucination. The inputs are known; the operation should be straightforward. Check the math manually or with a calculator. This is the CNET problem — the inputs were correct, but the calculation was subtly wrong.
Category 2: Statistics attributed to named sources. If an AI cites "a 2023 CDC report" for a statistic, apply the two-step check from Lesson 2. Does the CDC exist? Yes. Does the CDC report say this specific number? That requires the actual report. Don't skip Step 2 just because the number sounds plausible.
Category 3: Unattributed statistical claims. "Research suggests," "studies show," "experts estimate" — these phrases followed by a specific percentage are a high-risk pattern. The AI has learned that this phrasing precedes statistics. It may have generated the statistic to fit the sentence. Without a specific source, these numbers are nearly impossible to verify and should be treated as uncorroborated until you find the actual study.
Counter-intuitively, more precise numbers are sometimes more suspicious, not less. A statistic of "about 70%" might come from actual research. A statistic of "72.4%" — with a decimal — sounds more authoritative, but that precision is often a hallucination artifact. Real survey data has decimal precision because of sample sizes; AI generates decimal precision because it looks credible.
The CNET story wasn't just about one publication making errors. It exposed something about what happens when AI-generated content enters information ecosystems at volume. CNET is read by millions of people making real financial decisions — about savings accounts, loans, investment products. A subtle error in a widely-read financial explainer can ripple outward in ways that are nearly impossible to trace.
By 2024, multiple studies estimated that somewhere between 15% and 20% of content on major social platforms contained text that was substantially AI-generated. The exact figure varies by study and by platform. But the implication is significant: if AI arithmetic hallucinations appear in even a small fraction of that content, the total number of people encountering subtly wrong numbers is very large.
No regulatory framework in 2023 or 2024 required AI-generated content to be labeled in a way that would alert readers to check the arithmetic. The EU AI Act requires transparency for certain high-risk applications. But an AI-written financial explainer on a news website does not currently fall into a regulated category in most jurisdictions.
The ethical question here is about scale: if one human journalist makes an arithmetic error, the impact is limited. If an AI system makes a consistent type of error and that system produces thousands of articles, the error propagates at a scale no individual journalist could create. Does scale change moral responsibility? Does it change regulatory obligation? These questions don't have finalized answers.
You're reviewing an AI-generated health article before publication. Below are four numerical claims from the draft. Your editor needs your assessment: which ones can publish as-is, which need verification, and which should be cut until sourced?
Tell the AI: Which category (1, 2, or 3) does each claim fall into? What's your publish/verify/cut decision for each? Be specific.
In March 2023, Google launched a product called Bard — its answer to ChatGPT — at a live event streamed to millions of viewers. In a promotional advertisement released beforehand, Bard was shown answering the question: "What new discoveries from the James Webb Space Telescope can I tell my 9 year old about?"
Bard answered fluently, with three bullet points. The third claimed that the Webb telescope had taken "the very first pictures of a planet outside of our own solar system." Astronomers immediately recognized the problem. The first direct images of exoplanets — planets outside our solar system — were taken in 2004, by the Very Large Telescope in Chile, nearly two decades before Webb launched.
The other two bullet points were accurate. The framing of the ad was enthusiastic and positive. The error was specific and verifiable. But here is what made the episode instructive beyond the single factual mistake: Google's stock dropped 9% the next day — erasing roughly $100 billion in market value. A one-sentence factual error in an advertisement caused the largest single-day dollar loss in the company's history up to that point.
What the Bard case illustrates isn't just about hallucination. It's about something subtler: the overall confidence and authority of the AI's presentation made it harder to notice the error. The two accurate bullet points provided cover for the false third one. Readers who trusted the fluent, structured answer as a whole were more vulnerable than readers who interrogated each claim independently.
This lesson is about a more sophisticated reading skill than catching outright hallucinations. It's about recognizing that a collection of true statements can create a false impression — and that AI is particularly capable of producing this kind of output, because AI is very good at generating coherent, well-structured narratives.
Consider a simple example. Suppose you ask an AI: "Is caffeine dangerous?" It might respond: "Caffeine consumption has been linked to elevated heart rate, increased anxiety in sensitive individuals, disrupted sleep patterns, and dependency. In high doses, caffeine can cause cardiac events." Every sentence in that paragraph is technically accurate. But a reader who asked because they wanted to know whether their morning coffee was a health risk would come away with a distorted picture — one that omits the large body of research suggesting moderate caffeine consumption has neutral or positive health effects for most adults.
This happens because AI models, when asked a question, often generate text that is most consistent with a particular framing. If the question implies a negative answer is expected, the model may generate a collection of true-but-negatively-framed facts. If the question implies a positive answer, the opposite can occur. The selection process is driven by what "fits" the narrative context — not by what would give the most balanced picture.
Researchers call this framing bias in AI outputs. It is harder to detect than hallucination because each individual claim survives fact-checking. You need to ask a different question: What has this output left out?
A detective reading an AI output doesn't just check the facts — they also ask whether the facts have been assembled honestly. Here are four specific moves that catch selective truth and framing bias.
Move 1: Ask what's missing. After reading an AI output, ask: what would the other side of this argument say? What evidence would someone use to argue the opposite conclusion? If you can't think of any — if the output seems to have pre-emptively closed off every alternative — that's a signal the framing may be selective.
Move 2: Flip the question. Ask the AI the opposite question and compare outputs. If you asked "is this technology dangerous?" ask "what are the benefits of this technology?" Compare what appears in one answer but not the other. The omissions are informative.
Move 3: Check the weight of emphasis. AI outputs often bury important caveats in subordinate clauses or final sentences, while leading with dramatic or attention-grabbing claims. Read the final sentence of any AI response — it often contains the most important qualifier, placed where it will have the least impact on how readers remember the content.
Move 4: Identify whose perspective is centered. AI models trained on English-language internet text have absorbed perspectives that are disproportionately from certain demographic and geographic groups. Ask whether the framing you're reading reflects the full range of people affected by the issue — or mainly the perspective of those who write most on the internet.
In the Bard/Webb case, Move 1 would have caught the error: "What would an astronomer who disagreed with this answer say?" An astronomer would immediately point out the 2004 discovery. Asking for the critical perspective — even hypothetically — surfaces what the AI left out.
The four moves from this lesson are not just for AI outputs. They are the same reading skills that journalists, researchers, lawyers, and scientists use when evaluating any source. What makes them especially important for AI outputs is that AI produces text that feels complete and authoritative — text designed (by its training) to satisfy the reader's question, not necessarily to give the most accurate picture.
Human experts often signal uncertainty and limitation with their tone, their qualifications, their references to other viewpoints. AI systems are trained to produce helpful, complete-seeming answers. That training creates a specific kind of blind spot: the reader's feeling of "I got what I needed" may arrive before the reading is actually done.
You now have a framework for reading AI outputs that most people — including many professionals who use these tools daily — do not apply systematically. The four moves. The three number categories. The two-step citation check. The confidence-is-not-accuracy principle from Lesson 1. These fit together into a single reading practice: treat every AI output as a starting draft written by a very capable, very confident intern who may not have checked their sources.
Knowing this changes how you read every AI-assisted headline, summary, explainer, or recommendation you encounter — which, by 2024 estimates, is a significant fraction of everything published online. You are not being paranoid. You are being precise. There's a difference, and you know it now.
One more thing to sit with: the same moves you're applying to AI outputs can be applied to human-written content. The difference is that AI produces these patterns at scale and with consistent fluency. The skill transfers everywhere — but it's most urgently needed here, now, in the period before readers, publishers, and regulators catch up to what these systems actually do.
Below is an AI-generated summary about a controversial topic. Your job is to apply at least two of the four reading moves from Lesson 4 — and identify specific evidence of selective truth or framing bias. The AI partner will push back on shallow analysis.
Apply Move 1 (what's missing?) and Move 2 (flip the question). What has the framing left out? What would the opposite question reveal? Be specific — don't just say "it's biased."