ModelRefs / Diffusion Model — AI Glossary

Diffusion Model — AI Glossary

A generative model that learns to reverse a gradual noising process to produce images, audio, or video from noise.

Overview

Diffusion models power Stable Diffusion, DALL-E, Midjourney, Sora, and Veo. Modern variants use flow matching and rectified flows for fewer denoising steps. For images, latent diffusion models (LDMs) operate in a compressed latent space for efficiency.

Reference details

Topic	multimodal
Last reviewed	2026-06-24

Multimodal AI

How it works

Training corrupts real data with noise at many levels and teaches a network to predict the noise that was added, so the model learns the gradient of the data distribution rather than the data itself.
Generation starts from pure noise and applies that network repeatedly, removing a little noise each step, so a sample is produced by a trajectory rather than by a single forward pass.
The step count is therefore the dominant cost: each step is one full network evaluation, and a sample costs as many evaluations as the schedule has steps.
Latent diffusion moves the process into a compressed space learned by an autoencoder, so the trajectory runs over far fewer values and the cost per step falls accordingly.
Conditioning — a text prompt, an image, a depth map — is injected at every step through cross-attention, which is why the prompt influences the whole trajectory rather than just its start.
Distillation and reformulated trajectories reduce the step count by training a student to take larger jumps, which is where most recent speed improvement has come from.

Example: Two multipliers, and why the latent one came first

A 50-step schedule means 50 full network evaluations per image; a distilled 4-step model means 4, a factor of 50 / 4 = 12.5. That is the headline speedup, and it came after the larger one. Operating in pixel space at 512 x 512 with 3 colour channels means the network processes 512 x 512 x 3 = 786,432 values at every one of those steps. A latent space compressing by a factor of 8 per spatial dimension with 4 channels holds 64 x 64 x 4 = 16,384 values, so each step processes 786,432 / 16,384 = 48 times less data. The two multiply: 50 pixel-space steps against 4 latent steps is 50 x 786,432 against 4 x 16,384, a ratio of 39,321,600 / 65,536 = 600. That is why latent diffusion rather than step reduction was the change that made text-to-image generation a consumer product — a 48x reduction applied to every step of every sample, before anyone shortened the trajectory. It also sets the quality trade: the autoencoder's reconstruction is a ceiling on detail, so fine texture lost in compression cannot be recovered by any number of denoising steps.

Commonly confused with

A diffusion model is not an autoregressive model, and the difference is where the sequential cost lives. An autoregressive model produces output one element at a time, so cost scales with output length and each element is final once emitted. A diffusion model produces the whole output at once and refines it over steps, so cost scales with the step count and every part of the output changes at every step. This is why diffusion suits fixed-size outputs such as images and why autoregression suits variable-length ones such as text — and why applying diffusion to text requires deciding the length up front.

When to use it

Reach for it when:

Fixed-size continuous outputs — images, audio, video — where the whole-output refinement structure fits the data.
Where sample diversity matters, since the stochastic trajectory produces genuine variety from one prompt.
With latent-space operation, which is the larger of the two available cost reductions.
With a distilled few-step variant when latency matters and some quality can be traded.

Reach for something else when:

Variable-length sequential output such as text, where the length must be fixed up front and autoregression is the natural fit.
Hard real-time generation, where even a few-step trajectory is several network evaluations.
Where fine detail below the autoencoder's reconstruction quality is required, which no step count recovers.
Expecting step reduction to deliver the gains latent-space operation already delivered, which is the larger factor.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Diffusion Model — AI Glossary.

Frequently asked questions

What is Diffusion Model?