Diffusion models: from noise to images, and now to text

Language models generate one token at a time, left to right. Diffusion models generate everything at once, by starting from pure noise and repeatedly removing a little of it. They took over image and video generation, and they’re now being tried on text. Here’s the math, without the hand-waving.

The forward process: destroy data on a schedule

Take an image x₀ and add a little Gaussian noise, over and over, for T steps, until nothing but noise remains. With a noise schedule β_t, each step is:

q(x_t | x_{t−1}) = N( √(1 − β_t) · x_{t−1} ,  β_t · I )

The useful property is that you can jump to any step directly. With ᾱ_t = Π (1 − β_s):

x_t = √ᾱ_t · x₀  +  √(1 − ᾱ_t) · ε          ε ~ N(0, I)

Any noise level is one line of arithmetic away from the clean image. DDPM (Ho, Jain and Abbeel, 2020) used T = 1,000 steps.

Training: predict the noise

The generative model has to learn the reverse: given a noisy x_t and the step t, estimate the noise that was added. Training is almost embarrassingly simple:

1. pick an image and a random step t;
2. sample noise ε and build x_t with the formula above;
3. train a network ε_θ(x_t, t) to predict ε, with a plain mean-squared error.

DDPM showed that this objective is a simplified form of a proper likelihood bound, and that it works better in practice. An equivalent view, from score-based models (Song and Ermon, 2019; Song et al., 2021), is that the network learns the gradient of the data’s log-density at every noise level, which points toward more likely images.

Sampling: from 1,000 steps to a few

To generate, start from pure noise x_T and apply the learned denoiser step by step, back to x₀. With DDPM’s 1,000 steps, that’s 1,000 network evaluations per image, which made the first diffusion models slow.

The step count came down fast:

• DDIM (Song, Meng and Ermon, 2020): a deterministic sampler over the same trained model that skips most steps; tens of steps instead of a thousand.
• Better ODE solvers treat sampling as solving a differential equation and need only a few dozen evaluations.
• Distillation into few-step models trains a student to jump from noise to image in very few steps: progressive distillation, and consistency models (Song et al., 2023).

Guidance: following the prompt

A text-to-image model is conditioned on a prompt c. Classifier-free guidance (Ho and Salimans, 2022) trains the same network both with and without the prompt, by randomly dropping the condition. At sampling time it extrapolates away from the unconditioned prediction:

ε̂ = ε_θ(x_t, ∅) + w · ( ε_θ(x_t, c) − ε_θ(x_t, ∅) )

The guidance weight w trades diversity for prompt adherence. It’s the “CFG scale” slider in every image tool, and it costs two network evaluations per step.

Latent diffusion: don’t denoise pixels

Running a network over every pixel at every step is expensive. Latent diffusion (Rombach et al., 2022), the basis of Stable Diffusion, first trains an autoencoder that compresses images into a much smaller latent space, then runs diffusion there. A 512 × 512 RGB image has 786,432 values; its 64 × 64 × 4 latent has 16,384, which is 48× fewer. The denoiser does its work at 48× less cost, and the decoder turns the final latent back into pixels once.

The backbone became a transformer

Early diffusion models used convolutional U-Nets. Diffusion Transformers (DiT, Peebles and Xie, 2023) replaced them with a transformer over latent patches, and found it scaled the way transformers do: more compute, predictably better images. Stable Diffusion 3 uses a transformer backbone. Video models treat clips as sequences of spacetime patches; OpenAI described Sora as a diffusion transformer.

The training objective has also been generalised. Flow matching and rectified flow (Lipman et al., 2023; Liu et al., 2023) learn a velocity field that transports noise to data along straighter paths. Straighter paths need fewer sampling steps, and Stable Diffusion 3 trains with rectified flow.

Diffusion for text

Text is discrete, so you can’t add Gaussian noise to a word. Masked diffusion language models replace noise with masking instead: corrupt by masking a growing fraction of tokens, and train the model to fill them back in. Generation starts from a fully masked sequence and unmasks over several rounds, many tokens at a time. LLaDA (Nie et al., 2025) showed the approach working at the 8B scale, and commercial systems such as Inception’s Mercury and Google’s Gemini Diffusion followed in 2025.

The appeal is parallelism. Many tokens are produced per step instead of one, which attacks the memory-bound one-token-per-pass decode that limits autoregressive models (the decode note explains that ceiling). Whether diffusion matches autoregressive quality on reasoning and long-form text is still an open question. It’s the most interesting challenge to left-to-right generation so far.

Rules I use

1. Steps are the cost knob. Most of a diffusion system’s latency is the number of network evaluations times the cost of each one; count both.
2. Work in latent space unless you have a reason not to. It’s the single biggest efficiency win.
3. Guidance doubles the work per step. Budget for two evaluations per step when CFG is on.

Sources: Ho, Jain and Abbeel, DDPM (2020). Song, Meng and Ermon, DDIM (2020). Song et al., Score-Based Generative Modeling through SDEs (2021). Ho and Salimans, Classifier-Free Diffusion Guidance (2022). Rombach et al., Latent Diffusion Models (2022). Peebles and Xie, DiT (2023). Lipman et al., Flow Matching (2023). Song et al., Consistency Models (2023). Nie et al., LLaDA (2025). The latent compression ratio is computed for a 512 × 512 image with an 8× downsampling, 4-channel autoencoder.