A large language model is, at its core, a compressed statistical summary of the text it was trained on. During pre-training, billions of parameters are adjusted to predict the next token across hundreds of billions of words. Once training ends, those parameters are frozen. The model has no mechanism to update itself when a company files for bankruptcy, a law changes, or a new product ships.

This is not a bug — it is an architectural fact. Training GPT-4, according to public estimates, cost tens of millions of dollars in compute. Retraining to add a week's worth of news is economically absurd. Fine-tuning on new data is cheaper, but it introduces its own hazards: catastrophic forgetting, where updating weights for new facts degrades performance on old ones. Neither path scales to the pace of real-world knowledge change.

The result: every deployed LLM has a knowledge cutoff — the date beyond which it has no direct information. OpenAI's GPT-4 Turbo, released in November 2023, has a training cutoff of April 2023. Claude 3 Opus has a cutoff of August 2023. Any question that depends on facts after those dates is answered from inference and interpolation, not from evidence.

When a model is asked about something beyond its training data, it does not say "I don't know" by default — it generates the most statistically plausible continuation of the prompt. That continuation may be factually wrong. This phenomenon, called hallucination, became a defining concern of enterprise AI deployment in 2022–2023.

In May 2023, two lawyers in New York, Steven Schwartz and Peter LoDuca, submitted a legal brief to federal court containing citations to six cases that did not exist. ChatGPT had fabricated them with confident, realistic-looking formatting. Judge P. Kevin Castel fined the law firm $5,000 and issued a blistering opinion. The incident became one of the most-cited examples of LLM hallucination in a high-stakes professional context.

The structural cause: the model had no access to an actual legal database during generation. It was producing tokens that looked like Westlaw citations because that pattern appeared in training data — not because it had retrieved and verified the underlying documents.

Enterprise AI applications almost universally require facts the base model cannot have: internal policy documents updated last week, customer records, product specs, regulatory filings, support tickets. A model trained on public internet data has none of this. Asking it to answer questions about your company's Q3 pricing strategy or your hospital's current formulary is asking it to invent.

The naive fix — stuffing everything into the system prompt — runs into context window limits. A 128,000-token context window sounds large until you realize that a medium-sized company's internal knowledge base might contain 50 million tokens of documentation. Even with 1-million-token windows (Google's Gemini 1.5 Pro, 2024), loading entire knowledge bases into every request is economically and latency-wise impractical at scale.

RAG solves this by retrieving only the relevant subset of knowledge at query time. Instead of loading everything, the system finds the three or five most relevant passages and includes only those. The model generates from evidence, not from statistical memory.

A transformer model processes all tokens in its context window through self-attention — every token attends to every other token. This means computational cost scales as O(n²) with context length. Doubling the context window roughly quadruples the attention computation. At one million tokens, the cost per inference call is substantial, even with efficient attention variants.

At current API pricing (2024), sending one million tokens to Gemini 1.5 Pro costs approximately $3.50 per query. A customer service system handling 100,000 queries per day, if it loaded its full knowledge base each time, would cost $350,000 per day in input tokens alone. RAG, by retrieving 3–5 relevant passages (perhaps 2,000 tokens), reduces that input cost by 99.8%.

This is not a hypothetical concern. Glean, the enterprise search startup, explicitly built its architecture around retrieval rather than context stuffing precisely because enterprise knowledge bases are orders of magnitude larger than any realistic context window.

Even when information is present in the context window, models do not attend to it uniformly. The 2023 paper "Lost in the Middle: How Language Models Use Long Contexts" by Liu, Lin, et al. at Stanford demonstrated that retrieval performance degrades significantly for documents placed in the middle of a long context. Models are strongly biased toward information at the beginning and end of their context.

This has a practical implication: naively stuffing a knowledge base into a long context does not guarantee the model will find the relevant fact. If the answer happens to be in document 47 of 100 documents, the model may effectively ignore it. A well-designed RAG pipeline that surfaces only the top-3 relevant documents produces more reliable answers than a context dump that buries the answer in the middle.

Beyond staleness and window limits, there is a third structural reason RAG exists: private knowledge. The vast majority of enterprise knowledge was never on the public internet and thus never in any model's training data. Your internal CRM notes, your legal contracts, your engineering runbooks, your customer correspondence — none of this was in the CommonCrawl dataset that trained most LLMs.

No amount of window size or model recency solves this. The only way to make a model reason about your proprietary documents is to provide them at inference time. RAG is the scalable mechanism for doing that selectively and efficiently.

Fine-tuning modifies a model's weights — the billions of numerical parameters that encode its behavior. It is most effective at changing three things: output format (producing structured JSON, following a specific template), tone and style (more terse, more formal, using domain vocabulary naturally), and task specialization (improving performance on a narrow task type like extracting contract clauses or classifying support ticket urgency).

What fine-tuning does poorly: injecting specific facts. When you try to fine-tune a model to "know" that your company's refund policy is 30 days, the model distributes that information across weight adjustments throughout the network. It may recall it correctly most of the time — but it may also confabulate variations, blend it with similar policies it saw in training, or override it when contradicting context appears. Facts stored in weights are lossy and unreliable compared to facts retrieved verbatim from a document.

The research on this is consistent: the 2023 paper "How Do Large Language Models Handle Privacy Sensitive Text?" and related work on knowledge editing (ROME, MEMIT) shows that inserting or editing specific factual knowledge in model weights is technically possible but unreliable at scale. A change to one fact can propagate inconsistently to related assertions.

BloombergGPT was trained on 363 billion tokens of financial text — filings, news articles, analyst reports spanning decades. The result was a model with dramatically better financial domain fluency. On the Financial Phrase Bank sentiment classification task, it outperformed GPT-4 in several evaluations. The fine-tuning was genuinely valuable.

But financial AI applications need current data. A model that knows financial vocabulary but cannot access yesterday's earnings announcement is not useful for most Bloomberg Terminal workflows. Bloomberg's real production systems connect the fine-tuned model to live data feeds and retrieval systems. The two layers solve different problems: fine-tuning for domain competence, RAG for factual currency.

This pattern repeats across industries. In medical AI, fine-tuning on clinical notes improves HIPAA-compliant terminology and clinical reasoning patterns. But a doctor asking about a specific patient's medication history needs retrieval from that patient's actual EHR record — not from statistical patterns in a training corpus. Google's Med-PaLM 2 (2023) was fine-tuned on medical text and achieved expert-level performance on USMLE questions, but real clinical deployment requires retrieval from patient records and current clinical guidelines.

There is a failure mode that appears repeatedly in enterprise AI projects: teams spend months fine-tuning a model on proprietary documents, expecting it to "learn" their knowledge base. The model does learn statistical patterns from the documents — but it does so imprecisely and without attribution. Specific factual answers become blurry. Worse, the model may confidently produce information that blends multiple documents or updates from the fine-tuning data, making it harder to trace errors to their source.

A RAG system, by contrast, returns a retrieved passage verbatim and can cite the source document and even the page or paragraph. When the retrieved content is wrong, you can audit the retrieval pipeline. When a fine-tuned model produces wrong facts, you cannot easily determine whether the error came from the original training data, the fine-tuning corpus, or the model's generalization.

Auditability is increasingly a legal and regulatory requirement. The EU AI Act (2024) mandates traceability for high-risk AI systems. A RAG system's retrieved passages are inherently more auditable than knowledge encoded opaquely in model weights.

Every RAG system, from the simplest prototype to production systems at companies like Notion, Glean, and Perplexity, implements some version of four core stages. Understanding these stages at a conceptual level before diving into implementation is essential for reasoning about where failures occur and where to optimize.

The most important design insight in RAG is that retrieval and generation are separate, independently improvable components. This is unlike a pure LLM where the "retrieval" happens implicitly in attention over training data — opaque and unmodifiable at inference time.

In a RAG system, if answers are wrong, you can diagnose whether the problem is in retrieval (the right passages are not being found) or generation (the right passages are found but the model misuses them). Each stage has its own metrics: retrieval precision and recall, answer faithfulness, answer relevance. Each can be improved independently.

This separation also enables knowledge updates without model retraining. When your company's refund policy changes from 30 to 45 days, you update one document in the index. The LLM is unchanged. The next query about return policies retrieves the updated passage. Compare this to fine-tuning: you would need to fine-tune the model again, verify the update didn't degrade other capabilities, and redeploy — a cycle that takes days to weeks.

The simplest RAG system — embed documents, do cosine similarity search, stuff top results into a prompt — is called naive RAG. It is the starting point for every RAG project and the architecture that most tutorials implement. It also has well-documented failure modes that the rest of this course teaches you to address.

Chunking failures: If a document is split in a way that cuts across the answer, neither chunk alone contains enough context. The retrieved passage is technically relevant but incomplete. Retrieval failures: Embedding similarity finds semantically similar text, but may miss exact-match answers or technical terms not well-represented in the embedding space. Generation failures: Even with correct passages, the model may ignore them in favor of its parametric knowledge, hallucinate additional facts not in the passage, or fail to synthesize across multiple retrieved chunks.

Understanding these failure modes — before you ever write code — is what separates engineers who ship reliable RAG systems from those who ship demos that break in production.