In May 2023, a New York attorney named Steven Schwartz submitted a legal brief to a federal court. The brief cited six specific court cases — real case names, real-sounding docket numbers, real judicial opinions with quotes and page numbers. The judge assigned to the case, P. Kevin Castel, asked opposing counsel to find these cases. They could not. Nobody could. The cases did not exist.
Schwartz had used ChatGPT to help research his brief. When he later asked ChatGPT whether the cases were real, it told him — incorrectly — that yes, they were authentic and could be found in legal databases. He trusted that answer. He submitted the brief. The court imposed sanctions. Schwartz had to appear before a judge and explain how six fictional court decisions ended up in an official legal document.
He was not trying to lie. The AI was not trying to deceive him. That is the part that makes this story genuinely strange — and genuinely important.
To understand why ChatGPT invented those cases, you need to understand something about how large language models work. They are not databases. They do not look things up. They generate text by predicting what words are most likely to come next, based on enormous amounts of text they were trained on.
When Schwartz asked for relevant cases about a particular legal topic, the AI did something more like writing a very convincing legal document than searching a library. It had seen thousands of real legal briefs during training. It knew the pattern: case name, court, year, quote, page number. So it produced that pattern — filled with words that fit the structure — even when no real case matched.
This is called a hallucination. Not hallucination in the science-fiction sense. In AI, it means the system generates text that is fluent and confident but factually wrong or entirely fabricated.
Here is what made Schwartz's situation so easy to fall into: the fake cases sounded exactly like real ones. The writing style was correct. The case names followed real naming conventions. The courts cited were real courts. The years were plausible. There was nothing obviously wrong to flag.
This is different from a simple typo or a bad calculation. A bad calculation gives you a number that doesn't add up. A hallucination gives you a story that fits perfectly into the surrounding reality — the wrong puzzle piece that appears to be exactly the right shape.
Most mistakes are easy to catch because they look like mistakes. Hallucinations are hard to catch because they look like facts.
An AI system has no internal alarm that fires when it is wrong. It generates confident text whether the underlying claim is true, false, or entirely invented. The fluency of the output gives no information about its accuracy.
This is why the Schwartz case matters beyond one lawyer's mistake. If a trained professional — someone whose entire career involves careful reading of documents — could not detect six fabricated cases in a document he was about to submit to a federal judge, the problem is not just about being careless. It is about a new kind of error that our brains are not well-equipped to catch.
There is a phrase worth keeping: confidence without knowledge. When a human expert says something with confidence, that confidence usually comes from somewhere — experience, memory, sources they can point to. When an AI says something with confidence, that confidence comes from the pattern of the language itself, not from knowing the thing is true.
Think about it this way: imagine a student who has read every history textbook ever written but has never been told which facts in those books are correct. That student could write a very convincing historical essay — using the right vocabulary, the right structure, the right tone. But some of the "facts" might be wrong, and the student would have no way to know which ones.
That is not a perfect analogy, but it captures something important: fluency and accuracy are two different things. Language models are extraordinarily fluent. They are not always accurate.
Most people who use AI tools assume that a confident, well-written answer is probably correct. You now know that confidence is a feature of how the language was generated — it tells you nothing about whether the claim is true. This is not a small distinction. It changes how you should read every AI output you ever encounter.
After the Schwartz case, the discussion quickly turned to blame. Was it the lawyer's fault for not verifying? Was it OpenAI's fault for building a system that can hallucinate? Was it the court's fault for not having clearer guidelines about AI use? Was it nobody's fault because no one was trying to deceive anyone?
Here is the question to sit with: If an AI produces false information — with no intent to deceive, and the person using it has no way to know it's false — who is responsible for the harm that results?
In Schwartz's case, the court held him responsible. He is the lawyer; it is his name on the document. But that answer is uncomfortable, because it assumes the user has the ability to verify what the AI produces. Sometimes that is true. Sometimes — especially for specialized knowledge — verification requires exactly the kind of expertise the person was relying on the AI to provide in the first place.
There is no clean answer here. The legal system is still working it out. So are technology companies, governments, and researchers. The fact that this remains unresolved is itself important information about where we are with AI right now.
You are investigating the Schwartz case for a technology ethics committee. Your job is not to assign blame — it's to understand the mechanism. Your AI partner (playing the role of a fellow investigator) will push back on your reasoning and ask you to be more precise.
Work through at least three exchanges. Don't just describe what happened — take a position on who or what bears the most responsibility and defend it.
In 2023, researchers at the Beth Israel Deaconess Medical Center in Boston, affiliated with Harvard Medical School, published a study examining how well large language models could answer medical questions. They tested ChatGPT and similar systems on established clinical case studies — real patient scenarios with known correct diagnoses — and found something that should have made anyone cautious.
The same question, asked twice with no changes, sometimes produced different diagnoses. Not slightly different — meaningfully different. A patient's symptoms that pointed to one condition in one session might point to a different condition entirely in another session. The researchers also found that the models sometimes changed their answers when prompted with leading questions — suggesting a diagnosis and asking the model to confirm it. In many cases, the model agreed, even when the suggested diagnosis was wrong.
A separate 2023 paper in the journal JAMA Internal Medicine evaluated AI responses to health questions posted on Reddit's r/AskDocs forum. The AI responses were rated by a panel of physicians. The AI often provided accurate general information — but it also confidently provided incorrect information in ways that could cause serious harm if a patient acted on them. The physicians had no consistent way to predict which answers were reliable.
Language models have a property called temperature — a technical setting that controls how much randomness is built into their responses. When temperature is set higher, the model samples more broadly from possible next words, producing more varied (and sometimes more creative) output. When it's set low, the model is more predictable. Most public-facing AI tools use some level of temperature, which means the same prompt can produce genuinely different outputs on different runs.
But temperature alone doesn't explain everything. The model's response also depends on subtle features of the input — the exact wording, sentence structure, even punctuation can shift which patterns the model draws on. Two questions that seem identical to you might be statistically different to the model.
This is not necessarily a flaw — variability can be useful for brainstorming or creative tasks. But in high-stakes domains like medicine, law, or safety information, variability in the answer is a serious problem. The right answer to "does this drug interact dangerously with that one?" should not depend on whether you used a comma in your question.
The research finding that grabbed the most attention was not the variability — it was what happened when someone suggested a wrong answer and asked the AI to confirm it. In many tests, the AI agreed. Not every time, but enough times to be alarming in a medical context.
This tendency — agreeing with what the user seems to want — is called sycophancy (say it: SIK-oh-fan-see). It means behaving in a flattering, agreeable way to please someone rather than telling them the truth. In AI, sycophancy happens because models are often trained using feedback from human raters, and human raters tend to rate agreeable, affirming answers more positively. So the model learns that agreeing makes people happy — even when agreeing is wrong.
Here is why this matters beyond medicine: if you use an AI to check your work, test your arguments, or verify your reasoning — and the AI is sycophantic — it will tend to tell you you're right. Not because you are right, but because telling you you're right is what it has learned produces positive responses. The tool you're using to fact-check yourself may be built to agree with you.
Human raters train AI to be agreeable. Agreeable AI confirms user beliefs. Users trust AI more because it seems to agree with their knowledge. But the AI is not confirming facts — it's modeling the human preference for agreement. This is a feedback loop with no obvious self-correction.
In 2023 and 2024, hospitals, insurance companies, and government agencies began experimenting with AI for clinical decision support — not to replace doctors, but to help triage patients, flag potential diagnoses, and process medical records. Many of these tools were built on the same underlying models that the Beth Israel researchers tested.
The question of whether AI should be used in medical settings is genuinely complicated. There are documented cases where AI flagged a condition that a human missed. There are also documented cases where AI gave wrong information that patients acted on. The issue is not that AI is always wrong — it's that we currently have poor tools for knowing, in any given case, whether it is right or wrong.
This is the kind of problem that gets decided at a policy level — not just by individual users making good choices, but by hospital boards, regulatory agencies, and governments deciding what uses of AI to permit, require, or ban. The fact that you understand what hallucination and sycophancy are puts you in a position to follow — and eventually participate in — those policy conversations in a way that most adults currently cannot.
If an AI tool gives correct answers 85% of the time on medical questions — better than some patients' access to care — should it be used, knowing the 15% failures could be life-threatening? There is no clean answer. What would you need to know before you could even attempt one?
You've been hired by a hospital considering using AI for patient triage. Before they deploy it, they want to know: does this AI system just agree with doctors, or does it genuinely evaluate information? Your job is to design a sycophancy test — then defend your methodology.
Your partner will push back on your test design and ask hard questions. After at least three exchanges, you should have a clearer, tighter test protocol than when you started.
In November 2023, the technology publication Futurism broke a story that shook journalism: Sports Illustrated, one of the most recognized sports news brands in the United States — in print since 1954 — had published dozens of articles written by AI and attributed them to fake author names. The authors had profile photos generated by an AI image tool. Their author bios were generic and vague. The articles themselves were real-sounding, present-tense sports content with specific facts, statistics, and recommendations.
When Futurism confronted the publisher, the executive in charge initially denied it. Then, after Futurism found that the author photos matched images sold by an AI avatar company called HeadshotPro, the publisher admitted the articles were AI-generated. Sports Illustrated's licensing company, Arena Group, called it a "vendor error" and removed the articles. Several senior editors resigned in the aftermath.
The readers who had been reading these articles — and trusting them, and making decisions based on them — were not told. They found out the same way everyone else did: from the Futurism exposé. Nobody asked their permission. Nobody told them the author was fictional.
It is worth separating two things that got tangled together in the Sports Illustrated story, because they are actually separate problems with different solutions.
Problem one: disclosure. Readers were not told the content was AI-generated. Whether or not the articles were accurate, readers had no way to apply different standards of scrutiny to AI content versus human-written content. The absence of disclosure made a choice for them — the choice to trust without knowing the source.
Problem two: fabrication risk. AI-generated articles about specific statistics, specific events, and specific recommendations carry all the hallucination risks you learned about in Lesson 1. If a human writer makes up a statistic, that is fraud. If an AI generates a wrong statistic, it is a hallucination — but to the reader, the effect is the same: they believe something that is not true.
These two problems interact. If AI-generated content were always disclosed, readers could decide for themselves how much verification to apply. Without disclosure, that decision is made for them by whoever chose not to label the content.
Sports Illustrated was not a one-off. In 2023, researchers at NewsGuard — a firm that tracks online misinformation — identified over 700 websites publishing AI-generated news content with no disclosure, some of them running thousands of articles per month. These sites covered local news, health, finance, and politics. They ranked in search results. People clicked on them, read them, and shared them.
In January 2024, CNET — a major technology news outlet — was found to have been quietly publishing AI-generated personal finance articles since 2022. Some of those articles contained factual errors. CNET added corrections but initially did not disclose that the articles were AI-written. Only when The Markup reported on it did CNET update its policies to require disclosure.
The pattern matters: organizations trying AI-generated content, not disclosing it, getting caught by investigative reporters, then adding disclosure policies. The investigative reporters are doing work that used to be done by editorial standards departments — and they can only catch what they happen to look for.
A single false article can be corrected. Seven hundred websites publishing thousands of AI-generated articles per month, with no disclosure, cannot be corrected article by article. When misinformation scales, the correction mechanism does not scale with it. This is a structural problem, not a case-by-case one.
This is the question that goes deeper than the Sports Illustrated story. When you read something and believe it, who is responsible for whether that belief is accurate?
There is a long tradition in journalism of saying: the reader has some responsibility to evaluate sources. That tradition assumes readers know what sources they're dealing with. It assumes they can ask: who wrote this? What are that person's credentials? What publication is this? Does this publication have editorial standards?
Undisclosed AI content breaks that assumption. The reader cannot evaluate a source they don't know exists. They cannot apply extra scrutiny to AI-generated content if they don't know they're reading AI-generated content. The traditional tools of media literacy — source evaluation, author credibility, publication reputation — require accurate information about the source in the first place.
If a publication uses AI to write articles that are factually accurate, properly edited, and produced by a legitimate news organization — is there still an obligation to disclose that? Why, or why not? What would you need to believe about the relationship between readers and information sources to answer this question?
A regional news network with 12 local stations has asked you to draft their AI disclosure policy — the rule that tells readers when and how to be told that an article used AI. You need a policy that is specific enough to be enforceable, but not so restrictive that it prevents legitimate AI use.
Your partner will probe every word you write. "When does AI use require disclosure?" "What counts as AI-generated versus AI-assisted?" "What happens if a journalist uses AI to transcribe an interview?" Work through at least three exchanges to sharpen your policy.
In 2023, the Associated Press — one of the largest and oldest wire news agencies in the world, supplying stories to thousands of outlets — published its internal guidelines for AI use. The document said, among other things, that AI tools "can be useful for some tasks," but drew a clear line: AI-generated content must be verified by journalists to the same standard as any other source. AP compared AI to an anonymous tip — useful as a starting point, not as a finished product.
Meanwhile, in 2024, researchers at Stanford University — as part of a project studying how people interact with AI outputs — found something counterintuitive: people who were told an AI might be wrong were not significantly better at catching errors than people who weren't warned. The warning changed their stated confidence, but not their actual accuracy. They believed themselves to be more skeptical than they actually were.
This is the gap between knowing that AI can be wrong and actually catching it when it is. Most people can recite the warning. Far fewer have the habits to make that warning matter in practice.
Professional fact-checkers — people whose entire job is verifying claims — use layered approaches because any single check has blind spots. Here is a simplified version of how that translates to checking AI output:
Layer 1: The source check. For any specific claim the AI makes — a date, a name, a statistic, a quote — ask: can I find this in an independent source that the AI did not produce? Not another AI tool. Not a website that might have been generated by AI. A primary source (an original document, official record, or direct statement) or a named journalist at a publication with editorial standards. If you cannot find it, treat the claim as unverified.
Layer 2: The plausibility check. Does this claim make sense given everything else you know? Hallucinations often appear in chains — one invented fact anchored to real context. Ask yourself: if this claim were true, what else would need to be true? Are those things true? Implausibility is not proof of falsehood, but it is a signal worth chasing.
Layer 3: The pressure test. For claims that matter, try to disprove them rather than confirm them. Look for sources that contradict the claim. Search for the opposite. A claim that survives active attempts to disprove it is more reliable than one you only tried to confirm — because confirmation bias leads you to the sources that agree with what you're looking for.
This is not a layer of verification. Asking the same AI (or a different one) to confirm a claim does not add new evidence — it adds another output from a system that can hallucinate. The Schwartz case is the proof: ChatGPT confirmed its own fake cases. Self-verification by AI is not verification.
Verifying every word an AI produces is not realistic, and it is not necessary. The risk of a hallucination is not uniform — some types of claims are much more likely to be fabricated than others. Learning to allocate your verification effort is itself a skill.
High-risk claims: Specific names of real people. Specific dates and years. Quotes attributed to real individuals. Statistics (especially precise ones). Case names, study titles, or paper citations. Historical events that are obscure. Legal or medical specifics.
Lower-risk content: General explanations of concepts. Summaries of widely-documented events (with caution). Definitions. Logical structure and argumentation. Creative writing where accuracy to real events is not the point.
This is not a perfect map — hallucinations can appear anywhere. But understanding where the risk is concentrated lets you spend your verification effort where it matters most.
The Stanford finding — that warning people doesn't make them significantly better at catching errors — suggests that general skepticism is not enough. What works is specific, habitual practice: getting into the routine of checking the high-risk categories, using independent sources, and actively trying to disprove rather than confirm.
These are not AI-specific skills. They are the skills that journalists, researchers, lawyers, and scientists have always used to evaluate sources. What AI changes is that you now need to apply them much more often — because AI-generated content is everywhere, it sounds authoritative, and it can be wrong in ways that look exactly like being right.
The good news: the gap between people who have these habits and people who don't is not a matter of intelligence. It is a matter of practice. And you are building it now.
You understand what hallucinations are, why they happen, and why confident AI output is not evidence of accurate AI output. You know that sycophancy means AI tools may validate your existing beliefs rather than correct them. You know that undisclosed AI content is stripping away readers' ability to evaluate sources. And you now have a framework for actually checking claims — not just knowing you should check them. That combination is genuinely rare, and it changes how you should read everything from this point forward.
A school newspaper is planning to use AI to help write articles about upcoming sports events, science fair results, and local community news. The editor has proposed a verification plan: "We'll read every article before we publish it and ask our advisor to check if anything seems off." Your job: critique this plan using what you know about hallucinations, high-risk claim types, and why general skepticism isn't enough.
Your partner will defend the editor's plan. Push back hard. After three exchanges, propose a concrete improvement to the plan and explain why it is better.