Diffusion Model

Forward (fixed)

$$q({\mathbf{x}}_t\mid {\mathbf{x}}_{t-1})=\mathcal{N}(\sqrt{1-{\beta }_t}{\mathbf{x}}_{t-1},{\beta }_t\mathbf{I}),{\mathbf{x}}_t=\sqrt{{\bar{\alpha }}_t}{\mathbf{x}}_0+\sqrt{1-{\bar{\alpha }}_t}\mathbf{ϵ},{\bar{\alpha }}_t=\prod _{s\le t}(1-{\beta }_s)$$

DDPM (learned reverse)

$$L_{simple}={\mathbb{E}}_{t,{\mathbf{x}}_0,\mathbf{ϵ}}\Vert \mathbf{ϵ}-{\mathbf{ϵ}}_{\theta }(\sqrt{{\bar{\alpha }}_t}{\mathbf{x}}_0+\sqrt{1-{\bar{\alpha }}_t}\mathbf{ϵ},t){\Vert }^2$$ $${\mathbf{x}}_{t-1}=\frac{1}{\sqrt{{\alpha }_t}}\left({\mathbf{x}}_t-\frac{1-{\alpha }_t}{\sqrt{1-{\bar{\alpha }}_t}}{\mathbf{ϵ}}_{\theta }({\mathbf{x}}_t,t)\right)+{\sigma }_t\mathbf{z}$$

	Idea
Training	random $t$, noise with the closed form, predict the noise, L2 loss
Sampling	start from $\mathcal{N}(0,I)$, ~1000 denoising steps
DDIM	predict ${\hat{\mathbf{x}}}_0$ and refine; non-Markovian, ~100 steps, deterministic with ${\sigma }_t=0$
Stable Diffusion	diffusion in the latent space of an autoencoder; cross-attention conditioning
Zero-1-to-3	novel views conditioned on image + $(R,T)$; views are 3D-inconsistent

Intuition

Adding noise is easy and needs no learning. The model only has to learn to undo one small noising step at a time, and the same network is reused for every step.

Appears in

• Lecture 10, Theory
• Lecture 10, DDPM
• Lecture 10, Algorithms
• Lecture 10, DDIM
• Lecture 10, Multi-view diffusion
• Lecture 1, course overview
• Lecture 11, Flow matching vs. diffusion
• Lecture 11, SDS: a frozen 2D diffusion model guides 3D optimization

Related concepts

• Flow Matching
• Attention
• Gaussian Splatting