Diffusion Models from the Ground Up — Add Noise, Then Subtract It

A reading of the DDPM paper (Ho et al., 2020) grounded strictly in its own text: the forward process that adds noise and the reverse process that removes it, why the objective settles into 'predict the noise', how a simplified loss hurt likelihood while improving samples, and what the paper does and does not say about step count.

Denoising Diffusion Probabilistic Models

Retracing the path that destroyed the picture

A diffusion model is a parameterized Markov chain trained using variational inference to produce samples matching the data after finite time (§1). Its transitions are learned to reverse a diffusion process — a Markov chain that gradually adds noise to the data until the signal is destroyed.

The design hinge sits right there. As the paper puts it, when the diffusion consists of small amounts of Gaussian noise, it is sufficient to set the sampling chain transitions to conditional Gaussians too, which allows a particularly simple neural network parameterization (§1). Break the destruction into fine enough steps and the reconstruction becomes writable in a simple form.

The forward process: shrink, then add

The forward process is not learned. It is fixed to a variance schedule $\beta_1,\dots,\beta_T$ (Eq. 2):

Swapping the symbols for plain words: $q(\mathbf{x}_{t}|\mathbf{x}_{t-1})$ is the rule that makes the next, slightly dirtier picture out of the one you are holding, and $\mathcal{N}$ is the instruction which says draw from a bell curve — the two slots after the semicolon being where that bell curve sits and how wide it is. In words: at each step, scale the image down by $\sqrt{1-\beta_{t}}$ first, then add noise of variance $\beta_{t}$. Fade the picture a little, sprinkle grit over it — that is the whole step. The scaling is there so that variance does not grow as noise accumulates (Appendix C).

The process has a convenient property. Writing $\alpha_{t}=1-\beta_{t}$ and $\bar{\alpha}_{t}=\prod_{s=1}^{t}\alpha_{s}$, the state at any timestep has a closed form (Eq. 4):

Here $\bar{\alpha}_{t}$ is a single number standing for how much of the original picture is still present after $t$ steps, and $1-\bar{\alpha}_{t}$ is how much of it has been replaced by noise. Read in words, the formula says the bell curve is centred on the original faded to $\sqrt{\bar{\alpha}_{t}}$ of its strength, and its width is exactly the part that has been given over to noise. That is, you can jump from the original image to any step in one shot — no thousand-iteration loop — so training can sample $t$ at random (§2). The experiments use $T=1000$ with $\beta$ increasing linearly from $10^{-4}$ to $0.02$. Those constants were chosen to be small relative to data scaled to $[-1,1]$, keeping the forward and reverse processes approximately the same functional form while pushing the signal-to-noise ratio at $\mathbf{x}_{T}$ as low as possible (§4).

The reverse step looks like a small step downhill

The reverse process is a chain of learned Gaussian transitions starting at $p(\mathbf{x}_{T})=\mathcal{N}(\mathbf{0},\mathbf{I})$ (Eq. 1), trained on the variational bound on negative log likelihood (Eq. 3).

A correspondence the paper points out matters here. Under the $\boldsymbol{\epsilon}$-prediction parameterization introduced below, the sampling procedure (Algorithm 2) resembles Langevin dynamics, with $\boldsymbol{\epsilon}_{\theta}$ acting as a learned gradient of the data density (§3.2). One step is close to "move a little along a gradient, then add noise".

Unfolding the variational bound

The bound is rewritten for variance reduction (Eq. 5) into three kinds of terms: $L_{T}$, the $L_{t-1}$ terms, and $L_{0}$. Because the $\beta_{t}$ are fixed constants, $L_{T}$ is a constant during training and can be ignored (§3.1). The middle terms become tractable once the forward process posterior is conditioned on $\mathbf{x}_{0}$ (Eqs. 6 and 7), so every KL divergence is a comparison between Gaussians and has a closed form (§2).

From there $L_{t-1}$ reduces to a squared error between $\tilde{\boldsymbol{\mu}}_{t}$ and $\boldsymbol{\mu}_{\theta}$ (Eq. 8). Read plainly, the network should just predict the posterior mean. That would be the natural stopping point — but the paper rewrites it once more.

The only unknown left is the noise

Reparameterizing Eq. 4 as $\mathbf{x}_{t}=\sqrt{\bar{\alpha}_{t}}\mathbf{x}_{0}+\sqrt{1-\bar{\alpha}_{t}}\boldsymbol{\epsilon}$ and expanding reveals what $\boldsymbol{\mu}_{\theta}$ has to predict (Eq. 10):

Read as an instruction, the right-hand side says: take the noisy picture you are holding, subtract the noise the network guessed at — shrunk by the fixed factor $\beta_{t}/\sqrt{1-\bar{\alpha}_{t}}$ — and then divide by $\sqrt{\alpha_{t}}$ to undo the shrinking the forward step applied. Which in words is: estimate the grime, wipe off that much, stretch the picture back to size. The point of this equation is that only two things appear on the right: $\mathbf{x}_{t}$ and $\boldsymbol{\epsilon}$. And $\mathbf{x}_{t}$ is already in hand as the network's input, so the only unknown is $\boldsymbol{\epsilon}$ — predict that and the mean falls out (Eq. 11). Substituting turns the objective into a weighted $\|\boldsymbol{\epsilon}-\boldsymbol{\epsilon}_{\theta}\|^{2}$ (Eq. 12), which resembles denoising score matching across multiple noise scales (§3.2).

The paper attaches its caveat: this is merely another parameterization of $p_{\theta}$, so its effectiveness is verified in the §4 ablation. Predicting $\mathbf{x}_{0}$ directly was also possible, but led to worse sample quality in early experiments.

Training and sampling fit in ten lines

Training (Algorithm 1): draw x0 from the data; draw t uniformly from 1..T
  draw ε ~ N(0, I); take a gradient step on
  || ε - ε_θ(√ᾱ_t x0 + √(1-ᾱ_t) ε, t) ||²

Sampling (Algorithm 2): x_T ~ N(0, I)
  for t = T..1
    z ~ N(0, I)  (z = 0 when t = 1)
    x_{t-1} = (1/√α_t)( x_t - ((1-α_t)/√(1-ᾱ_t)) ε_θ(x_t, t) ) + σ_t z

There is no loop on the training side. Thanks to Eq. 4, one random timestep and one noise draw are enough.

The simplified loss is what raised quality

The paper then drops the weight $\frac{\beta_{t}^{2}}{2\sigma_{t}^{2}\alpha_{t}(1-\bar{\alpha}_{t})}$ from Eq. 12 entirely (Eq. 14), leaving plain $\|\boldsymbol{\epsilon}-\boldsymbol{\epsilon}_{\theta}\|^{2}$. This is not the variational bound but a weighted variational bound, and in the §4 setup it down-weights the small-$t$ terms. Those terms train the network to denoise data that is barely noisy, so it is beneficial to down-weight them and let the network concentrate on the harder denoising at larger $t$ (§3.4).

Table 2 shows the effect. Predicting $\tilde{\boldsymbol{\mu}}$ with the true bound and fixed isotropic $\boldsymbol{\Sigma}$ gives IS 8.06 / FID 13.22; predicting $\boldsymbol{\epsilon}$ under the same conditions gives 7.67 / 13.51 — essentially a tie. But $\boldsymbol{\epsilon}$-prediction with $L_{\mathrm{simple}}$ jumps to IS 9.46 / FID 3.17. Meanwhile, training $\tilde{\boldsymbol{\mu}}$ on the simplified objective, and learning a diagonal $\boldsymbol{\Sigma}$ under $\boldsymbol{\epsilon}$-prediction, were unstable enough that no scores are reported. The noise parameterization and the simplified loss only work as a pair.

Step count against quality

$T=1000$ was set so the number of network evaluations during sampling would match prior work, and the paper states it was chosen without a sweep (§4, Appendix B). Sampling calls the network $T$ times exactly as Algorithm 2 says, so cost is directly proportional to step count. Measured: 17 seconds for a batch of 256 CIFAR10 images, 300 seconds for 128 images at $256\times256$ (Appendix B).

On length itself, what the paper says is this: the Gaussian diffusion length is not restricted to the data dimension, and can be made shorter for fast sampling or longer for model expressiveness (§4.3). Predicting $\hat{\mathbf{x}}_{0}$ partway through the reverse process also shows large-scale features appearing first and details last.

The allocation of rate and distortion is measured too. In Table 4, running the reverse process to completion gives a rate of 1.776 bits/dim at RMSE 0.951, while at 900 steps remaining it is 0.120 bits/dim at RMSE 12.02. Most of the bits go to describing imperceptible distortion (§4.3). Note, though, that this is an analysis of a progressive lossy code, not a measurement of image quality when sampling is cut short.

Limits and open questions

Stated by the paper itself:

• The likelihoods are not competitive. CIFAR10 lands at $\le 3.75$ bits/dim against Sparse Transformer's 2.80, and the authors write that their codelengths "are not competitive with other types of likelihood-based generative models" (§4.3, Table 1).
• Learning the reverse variances is unstable. A parameterized diagonal $\boldsymbol{\Sigma}_{\theta}$ trains less stably and gives poorer samples than fixed variances (§4.2, Table 2).
• The progressive compression is a proof of concept. It depends on a procedure like minimal random coding, which is stated to be intractable for high-dimensional data (Extra information).
• Hyperparameters were tuned on CIFAR10 and transferred. $T$ was fixed at 1000 without a sweep, and the EMA decay of 0.9999 was not swept either (Appendix B).
• Final experiments were trained once, with scores reported at the minimum FID over training (Appendix B). There is no information about run-to-run variance.
• Some attributes do not interpolate. Pose, hairstyle, expression and background vary smoothly; eyewear does not (§4.4).

On overfitting, the paper argues from a train-test codelength gap of at most 0.03 bits/dim that the model is not overfitting (§4.3).

Note as of writing (2026) — an author's remark, not a claim of the paper: all numbers are this paper's own from 2020. Few-step samplers and diffusion in latent space are outside its scope and are not included here.

How this shows up on the job

Engineers meet diffusion models when continuing training of a released model on their own data, or training one from scratch on a narrow domain. Rederiving the theory is rare; the values below go straight into a config file.

The knobs. T (the number of diffusion steps), the $\beta$ schedule (linear $10^{-4}\to0.02$), EMA decay 0.9999, dropout (swept on CIFAR10 and set to 0.1), learning rate $2\times10^{-4}$, batch 128 (64 at $256^2$). Self-attention sits between the convolutional blocks at $16\times16$ resolution, and the timestep $t$ enters each residual block as a Transformer sinusoidal position embedding (Appendix B).

Ways to get hurt. (1) Keeping the learning rate at high resolution makes training unstable — the paper drops to $2\times10^{-5}$ at $256^2$. (2) Assuming that learning the reverse variances must help; it breaks (Table 2). (3) EMA and dropout were used at unswept defaults, so behaviour without them is outside what the paper covers. (4) Generation cost scales with step count, one network call per step. (5) FID is 3.17 against the training set and 5.24 against the test set — never compare numbers without matching that reference (§4.1).

Questions that come up in interviews. "Why predict the noise?" — Eq. 10 shows the quantity $\boldsymbol{\mu}_{\theta}$ must produce is a linear combination of $\mathbf{x}_{t}$ and $\boldsymbol{\epsilon}$, and $\mathbf{x}_{t}$ is already the input, so $\boldsymbol{\epsilon}$ is the only unknown (Eq. 11); it also matches the form of denoising score matching (§3.2). "How does $L_{\mathrm{simple}}$ differ from the bound?" — it drops the weight in Eq. 12, so small-$t$ terms count for less, which costs likelihood and buys sample quality (§3.4). The probability background is in probability and statistics for AI, and how to read bits/dim is in information theory for AI.

Summary

• The forward process is fixed, unlearned noise addition, and Eq. 4 lets you jump to any $t$ in one step (§2)
• The reverse process can be written as Gaussians because the $\beta_{t}$ are small — fine steps buy simplicity (§1, §2)
• Predicting $\boldsymbol{\epsilon}$ falls out because it is the only unknown left on the right of Eq. 10 (§3.2)
• Dropping the weight to get $L_{\mathrm{simple}}$ hurt likelihood and substantially improved sample quality (§3.4, Table 2)
• Generation cost is proportional to step count; on length, the paper only says it can go shorter for speed or longer for expressiveness (§4.3)

Original paper: Ho, Jain, Abbeel, "Denoising Diffusion Probabilistic Models" (arXiv:2006.11239, 2020 / v2)