A diffusion model is trained in two phases. The first — the forward process — requires no learning at all. You take a clean training image and add a small, mathematically precise amount of Gaussian noise at each of T timesteps (typically 1,000). At step 1, the image looks nearly identical to the original. At step 500, it is recognizably degraded. At step 1,000, it is statistically indistinguishable from pure random noise. This schedule is fixed before training begins — it is not learned.

The critical property of this schedule is that it is Markovian: each noisy image depends only on the image one step earlier, not on the full history. This simplifies the mathematics dramatically. You can also compute the noise at any arbitrary timestep directly, without running through all prior steps — a shortcut that makes training feasible at scale. The noise added follows a Gaussian distribution with a variance controlled by a fixed schedule (linear, cosine, or sigmoid in different papers), so you always know exactly what was added.

The forward process produces a vast training dataset at no annotation cost: pairs of (noisy image at step t, clean image, timestep t). Every image in your training corpus becomes 1,000 training examples automatically. This is why diffusion models could be trained on LAION-5B — 5.8 billion image-text pairs — without requiring expensive human labeling of noise levels.

The reverse process is what the neural network actually learns. Given a noisy image at timestep t, the network must predict either: (a) the original clean image, or (b) the noise that was added — these two framings are mathematically equivalent but have different training dynamics. Ho et al. found that predicting the noise (called epsilon prediction, or ε-prediction) worked better in practice. The network's job at each step is to estimate what random static was layered on top of the underlying signal.

The architecture used for this prediction is a U-Net — originally developed for biomedical image segmentation in 2015 by Ronneberger et al. at the University of Freiburg. A U-Net has an encoder path that progressively compresses the image into a compact representation, and a decoder path that expands it back to full resolution. Crucially, skip connections link corresponding encoder and decoder layers, preserving fine spatial detail while also incorporating global context. In diffusion models, the timestep t is embedded as a numerical vector and injected into every layer of the U-Net, so the network always knows how noisy the input is and calibrates its prediction accordingly.

Training the U-Net is straightforward: feed in a noisy image and a timestep, have the network predict the noise, compare its prediction to the actual noise that was added (which you know), and backpropagate the mean squared error. Repeat across millions of image-timestep pairs. After training, the network has learned a surprisingly general model of how images look — what textures, edges, lighting, and compositional structures are probable — because recovering signal from noise requires understanding what signal looks like.

At inference time — when you actually want to generate an image — you start with pure Gaussian noise (a tensor of random numbers the same size as your target image) and run the reverse process step by step. At each step, the U-Net predicts the noise component, you subtract it (scaled appropriately), and you obtain a slightly less noisy image. After T steps, you have a clean image.

The original DDPM paper (2020) used T = 1,000 steps, which was too slow for practical use. A year later, Song et al. at Stanford introduced DDIM (Denoising Diffusion Implicit Models), a deterministic sampling algorithm that can produce high-quality images in 20–50 steps by taking larger, more carefully calculated steps through the noise schedule. This reduced generation time by roughly 20× without retraining the underlying model. The insight was that the Markovian constraint is only needed during training, not during sampling — you can skip steps if you account for the skipped noise analytically.

Later work by Lu et al. (2022) introduced DPM-Solver, which reduced steps further to 10–20 for many use cases. Today, schedulers like DPM-Solver++, PNDM, and LCM (Latent Consistency Models, 2023) can produce acceptable images in as few as 4 steps by distilling the multi-step process into a learned shortcut. Speed and quality remain in tension — fewer steps means less refinement — but the trajectory has been consistently toward faster generation at equivalent or better quality.

A Variational Autoencoder (VAE) is trained as a separate component, before and independent of the diffusion model. Its job is to learn a compressed representation of images. The encoder takes a full-resolution image and maps it to a lower-dimensional tensor — in Stable Diffusion's case, 8× spatial compression in each dimension, plus 4 latent channels. The decoder takes that compressed tensor and reconstructs the original image as faithfully as possible.

The "variational" aspect introduces an important regularization: the latent space is trained to be approximately Gaussian — meaning similar images cluster near each other, and the space doesn't have arbitrary holes or discontinuities. This property matters enormously for diffusion, because the forward process starts from Gaussian noise. If the latent space is also Gaussian-shaped, then starting from noise and denoising gives you something that the decoder can meaningfully reconstruct. If the latent space had arbitrary topology, noise initialization would produce garbage.

The VAE is trained on a perceptual loss (does the reconstruction look similar to the original according to a pre-trained image classifier?) plus a reconstruction loss (are the pixel values numerically close?) plus a KL divergence term (is the latent distribution close to Gaussian?). These three objectives in tension produce a latent space that is both compact and semantically organized.

The latent representations learned by a well-trained VAE have an interpretable geometry, even though this structure is not explicitly supervised. Directions in latent space tend to correspond to semantically meaningful variations: moving along certain axes shifts color temperature, others shift spatial composition, others shift object identity. This emergent structure is what makes latent space arithmetic sometimes work — the phenomenon where (latent of "king") minus (latent of "man") plus (latent of "woman") roughly equals (latent of "queen"), familiar from word embeddings, has weak analogues in image latent spaces.

More practically, the compressed latent representation preserves high-frequency structural information (edges, textures) through the skip-connection-like paths in the VAE decoder, while the low-dimensional bottleneck forces the model to capture global composition. When the diffusion U-Net operates on this latent representation, it is effectively learning a prior over compressed image descriptions — not over raw pixel values.

The VAE also introduces an important source of quality limitation: decoder artifacts. Certain fine details — text rendering, fingers, high-frequency textures — are difficult to reconstruct faithfully through the compression bottleneck. When Stable Diffusion generates blurry text or malformed hands, part of the blame belongs to the VAE decoder's limitations at these structure types, not only to the diffusion model's prior. Stable Diffusion XL (2023) and SDXL-VAE attempted to address this with a higher-capacity decoder and training data that included more fine-detail examples.

Not all major diffusion models use the latent approach. DALL·E 2 (OpenAI, 2022) used a pixel-space diffusion model for its final stage, operating at 64×64 then upsampled through a separate diffusion upsampler. Imagen (Google Brain, 2022) similarly used a cascade of pixel-space diffusion models at increasing resolutions (64→256→1024). These approaches traded computational cost for avoiding the VAE reconstruction bottleneck — they never had to decompress from a latent space, so they avoided decoder artifacts entirely.

The latent approach won commercially because the compute savings outweighed the quality tradeoffs for most applications. DALL·E 3 (2023) shifted OpenAI back toward a latent-space approach with a heavily improved VAE. Google's Imagen 2 and the underlying infrastructure of Midjourney v5 and v6 also use latent-space variants. The consensus in the research community by 2023–2024 was that with a sufficiently high-capacity VAE, latent diffusion matches or exceeds pixel diffusion quality at a fraction of the cost.

Before a text prompt can guide a diffusion model, it must be converted into a numerical representation that the U-Net can process. Stable Diffusion 1.x used the CLIP text encoder (ViT-L/14, OpenAI 2021) — a transformer that maps tokenized text to a sequence of 768-dimensional vectors. Stable Diffusion 2.x switched to OpenCLIP ViT-H/14, producing 1024-dimensional embeddings trained on a larger and more carefully filtered dataset. SDXL (2023) took this further by using two text encoders simultaneously: OpenCLIP ViT-bigG (1280-dimensional) and the original CLIP ViT-L (768-dimensional), concatenating their outputs to produce a richer 2048-dimensional conditioning signal.

The key property of CLIP embeddings is that they were trained to align text and image representations in a shared space: text describing an image should be close in embedding space to the image itself. This alignment is what makes CLIP embeddings useful as conditioning signals — they carry visual semantic meaning, not just syntactic structure. A CLIP embedding of "sunset over mountains" contains information about warm colors, horizontal gradients, and jagged silhouettes, encoded implicitly through the contrastive training objective on 400 million image-text pairs.

For DALL·E 3 (2023), OpenAI moved to a different conditioning regime: rather than CLIP, they used T5 (a pure-language transformer from Google) for text encoding. T5 embeddings are better at preserving syntactic and relational structure — understanding prompts like "a red cube on top of a blue sphere" more reliably than CLIP, which tends to represent concepts as unordered bags of features. The tradeoff is that T5 embeddings have no inherent visual grounding, so they require a larger and better-trained diffusion model to interpret them correctly.

The mechanism by which text embeddings actually influence the U-Net is cross-attention. In the U-Net's intermediate layers, the spatial feature maps (representing the current state of the noisy latent) are treated as queries, while the text embedding sequence is treated as keys and values. At each cross-attention operation, every spatial position in the image attends to every token in the text prompt, weighting its influence based on learned relevance scores.

This means the conditioning is not applied once at the beginning but at every cross-attention layer throughout the denoising process. The U-Net is constantly re-querying the text embedding, asking (in effect) "given this prompt, what should this spatial region look like at this noise level?" Different attention heads specialize for different aspects of the conditioning: some heads are primarily responsible for style, others for object identity, others for spatial arrangement.

The cross-attention maps produced during generation are interpretable and have been studied explicitly. In a 2023 paper from Google Research (Prompt-to-Prompt, Hertz et al.), researchers showed that specific text tokens reliably activate specific spatial regions in the cross-attention maps — and that you can edit images by manipulating these maps rather than by changing pixels. This research confirmed that the diffusion model is building a genuine spatial-semantic correspondence between prompt tokens and image regions, not simply pattern-matching at a global level.

Even with cross-attention, early conditioned diffusion models tended to produce images that were only weakly responsive to the prompt — they looked like plausible images, but not necessarily of the thing described. The fix was Classifier-Free Guidance (CFG), introduced by Ho and Salimans at Google Brain in 2021.

CFG works by training the model with both conditioned (with text) and unconditioned (with empty text) examples. At inference time, you run the denoiser twice per step: once with your text prompt and once with an empty prompt. The final noise prediction is: unconditioned prediction + guidance_scale × (conditioned prediction − unconditioned prediction). This extrapolates in the direction that makes the output more consistent with the conditioning. A guidance scale of 1.0 is equivalent to no guidance; 7.5 is the Stable Diffusion default; values above 15 typically produce over-saturated, distorted outputs as the model is pushed far outside its training distribution.

CFG is computationally expensive: it doubles the number of U-Net forward passes per step. Various approximations have been developed — Perturbed Attention Guidance (2024) and Autoguidance (2024) attempt to achieve similar effects with a single forward pass, using perturbations of the attention maps rather than a separate unconditioned pass. At the time of writing, CFG remains the dominant guidance method in production systems, though its compute cost has spurred significant research into single-pass alternatives.

Diffusion models do not have an explicit symbolic reasoning layer. They do not count objects; they learn statistical distributions over what images containing "four coins" tend to look like. In the training data (LAION-5B or similar), images labeled with "four coins" vary enormously — different angles, lighting, arrangements, overlaps. The model learns a blurry average of these visual patterns, not the rule that produces them.

More precisely: the U-Net operates on continuous activations. It has no mechanism to say "I have placed exactly N instances of object X." During each denoising step, it is making a local probabilistic prediction about what the image should look like given the current noisy state and the conditioning. Whether there are 3 or 5 coins emerges from the accumulated momentum of many local decisions — and nothing in the architecture enforces global count consistency.

This is why models like DALL·E 3 made specific improvements to counting by dramatically improving their training data through recaptioning — having GPT-4V write detailed, count-explicit captions for training images. This didn't fix the architectural limitation; it worked around it by ensuring the statistical prior better matched count-explicit descriptions. The improvement is real but fragile: push counts high enough (7 coins, 12 coins) and the failures return.

Prompts specifying spatial relationships — "the red sphere is to the left of the blue cube" — fail at high rates in most diffusion models. The underlying cause relates to how CLIP embeddings encode meaning. CLIP training used contrastive objectives on image-text pairs where the text was typically a caption describing the image's content, not its precise spatial layout. The training signal did not reward precise spatial binding, so the embeddings don't encode it reliably.

Cross-attention can, in principle, bind attributes to spatial positions — and the Prompt-to-Prompt work (Hertz et al., 2023) showed that attention maps do develop loose spatial specialization. But this specialization is learned statistically, not architecturally guaranteed. A text token for "left" will activate attention patterns associated with the left half of images in the training set, but only as a probabilistic tendency. It does not force the object to appear on the left.

Research efforts to fix this include structure-guided generation (providing bounding boxes as additional conditioning, as in GLIGEN, 2023), multi-object diffusion (segmenting the generation into object-level passes), and training on datasets with explicit spatial annotation. These help but each introduces new constraints: bounding box conditioning requires users to specify layouts, which defeats the convenience of natural language generation.

Diffusion models trained on the standard LAION-5B dataset are notoriously bad at rendering legible text within images. The structural reason is that text rendering requires precise spatial arrangement of strokes into recognizable symbols — a task that requires understanding discrete character shapes and their sequential arrangement, not just the statistical distribution of pixel patterns near text-like regions.

The U-Net's denoising process has no mechanism that corresponds to "write these specific characters in this sequence at this position." Text tokens from the CLIP encoder describe what the text says, not how the specific glyphs look spatially. The model learned that images with prompts mentioning a word tend to have text-like regions — but learned this distribution too coarsely to reproduce specific characters reliably.

DALL·E 3 made substantial improvements in text rendering, partly through training on synthetic images with clean, diverse text and partly through the T5 encoder's better preservation of the specific character sequence. Imagen 2 (Google, 2023) and Stable Diffusion 3 (2024) also made significant progress using improved text encoders and architecture changes. SD3 specifically introduced a transformer-based architecture (DiT, Diffusion Transformer) rather than a U-Net, and the attention mechanism in DiT handles long-range dependencies — including character sequences — more directly.

The "six-fingered hand" failure is one of the most widely observed artifacts in early diffusion models and merits specific explanation. Human hands are topologically complex — five distinct articulated appendages emerging from a palm in consistent spatial arrangement — and are photographically variable. In the training data, hands are often partially occluded, blurred, unusual in gesture, or seen from ambiguous angles. The model's prior over hand appearance is therefore wide and uncertain.

More importantly, the diffusion process operates locally: at each denoising step, the U-Net makes predictions based on local receptive fields. Whether a specific region looks like a finger is evaluated locally. The constraint that exactly five non-overlapping fingers must emerge from a palm in biologically consistent arrangement is a global topological constraint. The diffusion process has no mechanism to enforce global topological consistency; it optimizes for local plausibility at each step. The result is that individual regions each look finger-like, but their aggregate arrangement violates the global constraint.

By 2024, models like Midjourney v6, DALL·E 3, and Stable Diffusion 3 substantially reduced hand artifacts through larger training datasets that over-represented clear hand photographs, better VAE decoders, and architecture improvements that increased effective receptive field size. The improvement demonstrates that the problem is addressable through scale and data, but the architectural analysis explains why it exists in the first place.