Understanding the DDPM Abstract

The Big Picture

This abstract introduces Denoising Diffusion Probabilistic Models (DDPMs) as a new approach to generating high-quality images. Rather than explaining the entire methodology here, the abstract is making four key claims:

1. What they're proposing: A new class of generative models inspired by physics
2. How to train them: A specific mathematical approach that connects to existing techniques
3. How to use them: A progressive decoding process with nice properties
4. How well they work: State-of-the-art results on standard benchmarks

Let me break down what each part means:

Part 1: What Are Diffusion Probabilistic Models?

Core Concept

Diffusion probabilistic models are a class of latent variable models. Let me unpack this:

• Latent variable model: A generative model that learns the underlying factors or patterns in data. The "latent variables" are hidden representations that the model learns. Think of them as the "essence" of what makes an image what it is.

• Diffusion: The model is inspired by a process from physics where systems gradually move toward equilibrium (like heat spreading through a room).

The Physics Inspiration: Nonequilibrium Thermodynamics

The paper draws inspiration from nonequilibrium thermodynamics, which studies systems that aren't in equilibrium. Here's the intuition:

• Imagine starting with a clear image (ordered state)
• Gradually add random noise to it (like heat being added to a system, pushing it toward disorder)
• This is the forward process: data → noise

To generate new images, we reverse this:

• Start with pure noise (disordered state)
• Gradually learn to remove noise step-by-step
• This is the reverse process: noise → data

This is conceptually elegant: if we can learn to undo the noise, we can create new images from random noise.

Part 2: The Training Approach

The Weighted Variational Bound

The abstract mentions training on a "weighted variational bound." Let me break down what this means:

Variational bound = A mathematical inequality that provides a lower bound on something we want to maximize (the probability of generating real data)

In generative modeling, we want to maximize:

where $x$ represents our data (an image). This is hard to compute directly, so instead we use a variational bound:

The right-hand side is what we actually optimize during training. The "weighted" part means different terms in this bound are given different importance during training—some terms are multiplied by larger weights than others.

Connection to Denoising Score Matching

The paper makes a novel connection between:

1. Diffusion probabilistic models (what they're proposing)
2. Denoising score matching with Langevin dynamics (an existing technique from physics)

Score matching is a technique from statistical physics. It involves learning the gradient of the log-probability distribution:

where:

• $\nabla_x$ is the gradient with respect to $x$ (points in the direction of increasing probability)
• $\log p(x)$ is the logarithm of the probability density

Langevin dynamics is a mathematical process that uses these gradients to sample from a distribution. The connection the paper discovers is that training diffusion models is mathematically equivalent to learning these score functions.

This connection is powerful because it:

• Provides a principled way to train the model
• Links to well-understood physics concepts
• Suggests better weighting schemes for the training objective

Part 3: The Decoding Process

The abstract mentions a "progressive lossy decompression scheme" that generalizes autoregressive decoding.

What This Means

Progressive: The model generates images in steps, progressively reducing noise

• Step 1: Remove most of the noise from pure noise (rough image)
• Step 2: Refine further (less rough)
• Step 3, 4, ...: Continue until image is clean

Lossy decompression: Like decompressing a compressed image, but with some information loss at each step (we're removing noise, not perfectly reconstructing)

Generalizes autoregressive decoding: Autoregressive models generate data one piece at a time (like predicting next word in a sentence). This approach does something similar but for image generation—predicting progressively refined versions.

The key insight: you can stop the process early to get a rough sample, or continue longer for higher quality. This gives flexibility not present in many other generative models.

Part 4: Quantitative Results

The paper then provides empirical evidence of success:

CIFAR-10 (32×32 images of objects)

• Inception Score = 9.46: Measures if generated images look like real object categories
• FID Score = 3.17: Measures similarity between generated and real image distributions (lower is better)
• Both were state-of-the-art at the time of publication

LSUN (256×256 natural scene images)

• Sample quality similar to ProgressiveGAN: At the time, ProgressiveGAN was one of the best image generation methods
• This shows diffusion models are competitive with the leading approach

Implementation Available

The authors released their code, which is important for reproducibility and adoption by the research community.

Why This Matters

The combination of:

1. Theoretical elegance (connection to physics and score matching)
2. Training stability (the weighted variational bound helps with training)
3. Practical results (competitive or better than existing methods)
4. Flexibility (progressive generation allows quality/speed tradeoffs)

...makes this a significant contribution that would influence generative modeling for years to come.

Understanding the Introduction to Diffusion Probabilistic Models

The Big Picture

This introduction section does several important things:

1. Positions diffusion models in the landscape of generative models — explaining where they fit among existing approaches like GANs, VAEs, and autoregressive models
2. Introduces the core concept — what a diffusion model is and how it works at an intuitive level
3. States the paper's main contributions — showing that diffusion models can generate high-quality images, and revealing a mathematical connection to score matching
4. Sets expectations — acknowledging trade-offs (good samples, but not the best likelihood values)

Let's break this down carefully.

Part 1: The Landscape of Generative Models

"Deep generative models of all kinds have recently exhibited high quality samples in a wide variety of data modalities. Generative adversarial networks (GANs), autoregressive models, flows, and variational autoencoders (VAEs) have synthesized striking image and audio samples..."

What this means: The authors are situating their work among four major families of generative models:

• GANs (Generative Adversarial Networks): Use adversarial training — a generator competes against a discriminator
• Autoregressive models: Generate data sequentially, one element at a time, conditioning on all previous elements
• Flows: Use invertible transformations to map between simple distributions (like Gaussian) and complex data distributions
• VAEs (Variational Autoencoders): Learn a latent representation and reconstruction process

All of these have shown impressive results. The authors are essentially saying: "Here's another approach that also works well."

Part 2: What is a Diffusion Model? (The Core Concept)

This is the crucial conceptual section. Let me break it down carefully:

The Two Directions: Diffusion vs. Reverse

"A diffusion probabilistic model (which we will call a 'diffusion model' for brevity) is a parameterized Markov chain trained using variational inference to produce samples matching the data after finite time. Transitions of this chain are learned to reverse a diffusion process, which is a Markov chain that gradually adds noise to the data in the opposite direction of sampling until signal is destroyed."

This describes two complementary processes:

Process 1: The Diffusion Process (The Forward Direction)

Think of this as corrupting data with noise:

• Start with a clean image from your dataset
• Gradually add Gaussian noise at each timestep
• After many steps, you're left with pure noise (the "signal is destroyed")
• This is a Markov chain — each step only depends on the previous step

Mathematically: If we denote the original data as $x_0$ and the diffusion process at timestep $t$ as $x_t$, then: $x_t = x_{t-1} + \text{small amount of noise}$

After many iterations (say 1000 steps), $x_T$ looks like completely random Gaussian noise.

Process 2: The Reverse Process (What We Learn)

This is the inverse of diffusion — removing noise to recover data:

• Start with pure random noise
• At each timestep, learn to remove some of the noise
• After many steps, you should have a clean, realistic image
• We train a neural network to learn how to do this

Key insight: If we could perfectly learn the reverse of the diffusion process, sampling would work by:

1. Start with random noise $x_T$
2. Apply the learned reverse transitions repeatedly
3. End up with a sample from the data distribution at $x_0$

Part 3: Why Gaussian Noise Makes This Simple

"When the diffusion consists of small amounts of Gaussian noise, it is sufficient to set the sampling chain transitions to conditional Gaussians too, allowing for a particularly simple neural network parameterization."

Why this matters:

If at each step you're adding small amounts of Gaussian noise, then the reverse process can also be parameterized using Gaussian distributions.

The mathematical beauty: For Gaussian distributions, we have nice closed-form formulas. Instead of the neural network learning the full reverse transition, it only needs to predict:

• The mean (center) of the Gaussian at each step
• The variance (spread) is often fixed

This is much simpler than having the network predict an entire probability distribution. It's one of the practical advantages of using Gaussian noise.

Part 4: The Paper's Main Contributions (Three Key Claims)

Contribution 1: Diffusion Models Can Generate High-Quality Images

"Diffusion models are straightforward to define and efficient to train, but to the best of our knowledge, there has been no demonstration that they are capable of generating high quality samples. We show that diffusion models actually are capable of generating high quality samples, sometimes better than the published results on other types of generative models (Section 4)."

Translation: Previously, nobody had shown that diffusion models work well for image generation. This paper demonstrates they do — and sometimes better than GANs or VAEs.

Evidence (from the abstract): They achieve FID score of 3.17 on CIFAR10, which was state-of-the-art at the time.

Contribution 2: Connection to Score Matching and Langevin Dynamics

"In addition, we show that a certain parameterization of diffusion models reveals an equivalence with denoising score matching over multiple noise levels during training and with annealed Langevin dynamics during sampling (Section 3.2)."

This is more technical, but here's the intuition:

• Score matching is another established method in machine learning — it learns the gradient (called the "score") of the log-probability distribution
• Langevin dynamics is a sampling technique that uses these gradients to generate samples
• The authors' discovery: When you set up diffusion models the right way, training them is mathematically equivalent to training a score matching model, and sampling is equivalent to running Langevin dynamics

Why this matters: This connection validates diffusion models theoretically — they're not just an ad-hoc method, but deeply connected to established mathematical frameworks.

Part 5: The Trade-off: Sample Quality vs. Likelihood

"Despite their sample quality, our models do not have competitive log likelihoods compared to other likelihood-based models..."

What this means:

• Sample quality: The images generated look good (this paper's strength)
• Log likelihood: A measure of how well the model assigns probability to real data — mathematically, $\log p(x)$ for actual data $x$. Higher is better.

The authors are being honest: their models generate beautiful images, but if you ask "what probability does this model assign to real data?", the answer isn't as competitive as other methods.

Why? The next sentence explains:

"...the majority of our models' lossless codelengths are consumed to describe imperceptible image details."

Translation: The model is spending its probability budget on tiny details humans can't perceive. If you're measuring likelihood (which treats all details equally), the model looks inefficient. But perceptually, the samples look great.

Part 6: Progressive Decoding Perspective

"...the sampling procedure of diffusion models is a type of progressive decoding that resembles autoregressive decoding along a bit ordering that vastly generalizes what is normally possible with autoregressive models."

What this means:

• Progressive decoding: Refine the result step-by-step, getting better each time
• Unlike autoregressive models that generate data in a fixed sequence (like left-to-right in images), diffusion models refine all positions simultaneously, progressively adding detail

This is a conceptually interesting perspective — it shows diffusion sampling as a generalization of autoregressive sampling.

Summary: What You Should Take Away

1. Conceptually: Diffusion models work by learning to reverse a noise-addition process. You train on the task "remove noise from images," then sample by starting with noise and repeatedly denoising.

2. Practically: They're simple to implement (just Gaussian distributions) and efficient to train.

3. Theoretically: They connect to score matching and Langevin dynamics — grounding them in established mathematical frameworks.

4. Empirically: They generate high-quality images, though they don't optimize for likelihood metrics.

5. Philosophically: The sampling is a progressive refinement that generalizes autoregressive generation.

The next sections will formalize these intuitions with mathematics.

Understanding Section 2: Background on Diffusion Models

Big Picture: What's This Section About?

This section introduces the mathematical framework of diffusion probabilistic models. Think of it as describing a two-way process:

1. Forward process (easy, fixed): Gradually add noise to clean data until it becomes pure random noise
2. Reverse process (hard, learned): Learn to gradually remove noise from random noise to reconstruct clean data

The key insight is that if you can learn to reverse the forward process, you can generate new data by starting with random noise and denoising it. This section lays out the mathematical machinery to make this work.

Part 1: What Are Diffusion Models? (The Overall Framework)

The Core Equation

A diffusion probabilistic model is a latent variable model. This means it works with hidden variables (latents) to model observed data. The key equation is:

$p_\theta(\mathbf{x}_0) := \int p_\theta(\mathbf{x}_{0:T}) \, d\mathbf{x}_{1:T}$

Let me break down the notation:

• $\mathbf{x}_0$ = the observed data (e.g., an image) - this is what we want to generate
• $\mathbf{x}_1, \mathbf{x}_2, \ldots, \mathbf{x}_T$ = hidden variables at different noise levels (latents)
• $\mathbf{x}_{0:T}$ = shorthand meaning "all variables from $\mathbf{x}_0$ to $\mathbf{x}_T$"
• $p_\theta(\mathbf{x}_{0:T})$ = the reverse process: a probability distribution parameterized by neural network weights $\theta$
• The integral $\int d\mathbf{x}_{1:T}$ means we "marginalize out" or sum over all possible latent values

In plain English: To get the probability of clean data $\mathbf{x}_0$, we consider all possible noisy versions $\mathbf{x}_1$ through $\mathbf{x}_T$ and integrate over them.

Part 2: The Reverse Process (What the Model Learns)

Equation (1) defines the reverse process in detail:

$p_\theta(\mathbf{x}_{0:T}) := p(\mathbf{x}_T) \prod_{t=1}^{T} p_\theta(\mathbf{x}_{t-1}|\mathbf{x}_t)$

$p_\theta(\mathbf{x}_{t-1}|\mathbf{x}_t) := \mathcal{N}(\mathbf{x}_{t-1}; \boldsymbol{\mu}_\theta(\mathbf{x}_t, t), \boldsymbol{\Sigma}_\theta(\mathbf{x}_t, t))$

Understanding the Components

The product notation $\prod_{t=1}^{T}$ is like a $\sum$ but for multiplication. It means we're multiplying probabilities together: $p(\mathbf{x}_T) \times p_\theta(\mathbf{x}_{T-1}|\mathbf{x}_T) \times p_\theta(\mathbf{x}_{T-2}|\mathbf{x}_{T-1}) \times \cdots \times p_\theta(\mathbf{x}_0|\mathbf{x}_1)$

This forms a Markov chain - a chain where each step depends only on the previous step (not the whole history).

Key components:

• $p(\mathbf{x}_T) = \mathcal{N}(\mathbf{x}_T; \mathbf{0}, \mathbf{I})$ = starting point is pure Gaussian noise (mean $\mathbf{0}$, identity covariance $\mathbf{I}$)
• $p_\theta(\mathbf{x}_{t-1}|\mathbf{x}_t)$ = the learned conditional: "given noisy version at step $t$, what's the cleaner version at step $t-1$?"
• This is Gaussian with mean $\boldsymbol{\mu}_\theta(\mathbf{x}_t, t)$ and covariance $\boldsymbol{\Sigma}_\theta(\mathbf{x}_t, t)$ - both predicted by neural networks

In plain English: We start with pure noise and repeatedly apply learned denoising steps. Each step takes slightly noisy data and produces slightly cleaner data.

Part 3: The Forward Process (Fixed, No Learning)

Equation (2) defines the forward process - the opposite direction:

$q(\mathbf{x}_{1:T}|\mathbf{x}_0) := \prod_{t=1}^{T} q(\mathbf{x}_t|\mathbf{x}_{t-1})$

$q(\mathbf{x}_t|\mathbf{x}_{t-1}) := \mathcal{N}(\mathbf{x}_t; \sqrt{1-\beta_t}\mathbf{x}_{t-1}, \beta_t \mathbf{I})$

Key Differences from the Reverse Process

• The forward process is fixed - no learning here! It's just a mathematical definition
• It gradually adds noise using a variance schedule $\beta_1, \beta_2, \ldots, \beta_T$
  • These $\beta_t$ values are hyperparameters (set by the researcher, not learned)
  • Small values like $\beta_t = 0.0001$ to $0.02$ are typical

What Does Each Step Do?

At step $t$, we go from $\mathbf{x}_{t-1}$ (slightly noisy) to $\mathbf{x}_t$ (more noisy) using:

$\mathbf{x}_t = \sqrt{1-\beta_t} \cdot \mathbf{x}_{t-1} + \sqrt{\beta_t} \cdot \boldsymbol{\epsilon}$

where $\boldsymbol{\epsilon}$ is a random noise vector.

Breaking this down:

• $\sqrt{1-\beta_t}$ ≈ slightly less than 1 (scales down the signal)
• $\sqrt{\beta_t}$ ≈ small (scales the added noise)
• So each step slightly reduces the signal and adds a bit of noise

If you repeat this $T$ times (like $T=1000$), eventually the signal completely vanishes and you have pure noise.

Part 4: The Key Innovation - Closed-Form Sampling

This is a crucial practical advantage. Instead of applying the forward process $T$ times sequentially, you can jump directly to any timestep $t$ using Equation (4):

$q(\mathbf{x}_t|\mathbf{x}_0) = \mathcal{N}(\mathbf{x}_t; \sqrt{\bar{\alpha}_t}\mathbf{x}_0, (1-\bar{\alpha}_t)\mathbf{I})$

With definitions:

• $\alpha_t := 1 - \beta_t$ (how much signal remains at step $t$)
• $\bar{\alpha}_t := \prod_{s=1}^{t} \alpha_s$ (cumulative product - how much signal remains after $t$ steps)

Why this matters: You can randomly sample a timestep $t$ and directly compute what $\mathbf{x}_t$ looks like, without sequentially applying $t$ noising operations. This dramatically speeds up training!

Part 5: Training the Model (The Variational Bound)

Now here's the challenge: how do we train the reverse process to actually reverse the forward process?

The standard approach uses variational inference - specifically, optimizing a variational lower bound on the log-likelihood:

$\mathbb{E}[-\log p_\theta(\mathbf{x}_0)] \leq \mathbb{E}_q\left[-\log \frac{p_\theta(\mathbf{x}_{0:T})}{q(\mathbf{x}_{1:T}|\mathbf{x}_0)}\right]$

Interpreting This Inequality

• Left side: The negative log-likelihood (how bad our model is at predicting real data)
• Right side: An upper bound that's easier to compute
• The inequality comes from a fundamental principle called Jensen's inequality

The right side expands to:

$L = \mathbb{E}_q\left[-\log p(\mathbf{x}_T) - \sum_{t \geq 1} \log \frac{p_\theta(\mathbf{x}_{t-1}|\mathbf{x}_t)}{q(\mathbf{x}_t|\mathbf{x}_{t-1})}\right]$

Breaking down the sum:

• First term $-\log p(\mathbf{x}_T)$: How good are we at having a distribution that matches pure noise?
• Remaining terms: At each timestep, compare our learned reverse step against the forward process

When we minimize this bound, we're training the neural network to predict the right denoising steps.

Part 6: Variance Reduction - A Better Loss Function

Computing the loss above can have high variance. The paper rewrites it more cleverly in Equation (5):

$\mathbb{E}_q\left[\underbrace{D_{\mathrm{KL}}(q(\mathbf{x}_T|\mathbf{x}_0) \| p(\mathbf{x}_T))}_{L_T} + \sum_{t>1} \underbrace{D_{\mathrm{KL}}(q(\mathbf{x}_{t-1}|\mathbf{x}_t,\mathbf{x}_0) \| p_\theta(\mathbf{x}_{t-1}|\mathbf{x}_t))}_{L_{t-1}} + \underbrace{- \log p_\theta(\mathbf{x}_0|\mathbf{x}_1)}_{L_0}\right]$

What Changed?

Instead of comparing our reverse step directly to the forward process, we compare it to $q(\mathbf{x}_{t-1}|\mathbf{x}_t, \mathbf{x}_0)$ - the forward process conditioned on knowing the original data $\mathbf{x}_0$.

This is a clever trick! When training on actual data, we know $\mathbf{x}_0$. Using this information in the comparison reduces variance dramatically.

The notation $D_{\mathrm{KL}}(A \| B)$ is the Kullback-Leibler divergence - a standard measure of how different two probability distributions are. It's always non-negative and equals zero only when distributions are identical.

Part 7: The Tractable Posterior

Here's why this clever rewriting works - Equation (6) shows the forward process posterior is Gaussian:

$q(\mathbf{x}_{t-1}|\mathbf{x}_t, \mathbf{x}_0) = \mathcal{N}(\mathbf{x}_{t-1}; \tilde{\boldsymbol{\mu}}_t(\mathbf{x}_t, \mathbf{x}_0), \tilde{\beta}_t \mathbf{I})$

With explicit formulas:

$\tilde{\boldsymbol{\mu}}_t(\mathbf{x}_t, \mathbf{x}_0) := \frac{\sqrt{\bar{\alpha}_{t-1}}\beta_t}{1-\bar{\alpha}_t}\mathbf{x}_0 + \frac{\sqrt{\alpha_t}(1-\bar{\alpha}_{t-1})}{1-\bar{\alpha}_t}\mathbf{x}_t$

$\tilde{\beta}_t := \frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_t}\beta_t$

Why This is Amazing

• Both $q(\mathbf{x}_{t-1}|\mathbf{x}_t, \mathbf{x}_0)$ (the target) and $p_\theta(\mathbf{x}_{t-1}|\mathbf{x}_t)$ (our learned model) are Gaussians
• When comparing two Gaussians using KL divergence, there's a closed-form formula - no Monte Carlo sampling needed!
• This dramatically reduces training noise and variance

In plain English: We can compute the exact loss using calculus instead of approximating it through random sampling.

Summary

This section established:

1. Two parallel processes:

   • Forward: gradually add noise (fixed, simple)
   • Reverse: gradually remove noise (learned, complex)

2. Mathematical framework: Both are Markov chains of Gaussians

3. Training trick: Compare learned reverse steps to forward process conditioned on real data

4. Computational efficiency:

   • Jump to any timestep instantly (Eq. 4)
   • Exact loss computation for Gaussian comparisons (Eq. 6-7)

This foundation enables the training procedure described in the later sections, which achieves state-of-the-art image generation results.

Section 3: Diffusion Models and Denoising Autoencoders

Big Picture: What's This Section Trying to Do?

Before diving into the math, let's understand the overarching goal. The previous sections established what diffusion models do (gradually add noise to data, then learn to reverse it) and showed they can be trained using variational inference. But the authors haven't explained how to make the best design choices for these models.

This section answers that question by revealing a surprising connection: diffusion models are mathematically equivalent to denoising score matching, a technique from a different field entirely. This connection gives the authors:

1. Theoretical insight into why their approach works
2. Practical guidance on how to parameterize and weight the training objective
3. A simplified objective function that works better empirically

Think of it like discovering that two seemingly different recipes produce the same dish—once you make that connection, you can borrow techniques from one to improve the other.

The Core Challenge: Too Many Design Degrees of Freedom

The opening paragraph highlights the problem:

"Diffusion models might appear to be a restricted class of latent variable models, but they allow a large number of degrees of freedom in implementation."

What does this mean?

Recall from Section 2 that diffusion models require you to choose:

• The variance schedule $\beta_1, \ldots, \beta_T$ (how much noise to add at each step)
• The neural network architecture and parameterization for the reverse process

These aren't trivial choices—different choices will give different results. The section is asking: Is there a principled way to make these choices, rather than just guessing?

The answer the authors provide: Connect to denoising score matching.

Score Matching: A Brief Introduction

To understand the connection, we need to know what "score matching" means.

The score of a distribution is the gradient of its log-probability:

Here:

• $\nabla_{\mathbf{x}}$ is the gradient operator (the vector of partial derivatives with respect to each component of $\mathbf{x}$)
• $\log p(\mathbf{x})$ is the natural logarithm of the probability density

Intuition: The score points in the direction of increasing probability—it's like a compass that always points toward regions where the data is more likely to appear.

Denoising score matching is a technique that learns to predict this score by training a network $s_\theta(\mathbf{x}, t)$ to match the true score at different noise levels. The key insight is:

This says: "Given a noisy version of data $\mathbf{x}_t$, the score tells us how to remove noise to recover $\mathbf{x}_0$."

The Connection: Why Denoising Networks Learn Scores

This is where the magic happens. The section (specifically 3.2, though you haven't asked me to explain it fully) establishes that predicting the noise in diffusion models is mathematically equivalent to predicting the score.

Here's the intuition:

In diffusion models, the network predicts what noise was added: given a noisy image, predict the noise $\boldsymbol{\epsilon}$.

In denoising score matching, the network predicts the gradient of the log-probability (the score).

These are the same thing (up to a scaling factor). If you predict the score correctly, you implicitly predict the noise correctly, and vice versa.

This connection is powerful because score-based methods have strong theoretical foundations. Once we make this connection, we can adopt their techniques—specifically, Langevin dynamics for sampling, which has better theoretical guarantees.

The Practical Payoff: A Better Training Objective

The section mentions:

"leads to a simplified, weighted variational bound objective for diffusion models (Section 3.4)"

This is referring to the key result: once you embrace the score-matching perspective, you can reweight the variational bound from Equation (5) to focus on what matters.

Recall Equation (5) from Section 2:

where:

• $L_T$ = KL divergence at the final (pure noise) step
• $L_{t-1}$ = KL divergence at intermediate steps
• $L_0$ = reconstruction loss for the data itself

The question: Should all these terms be weighted equally when training? Probably not—$L_T$ is comparing two nearly-identical Gaussian distributions (both nearly pure noise), so it's numerically stable. But $L_0$ (reconstructing from almost-pure-noise to the actual image) is harder and probably deserves more weight.

The weighted bound (which Section 3.4 will detail) gives different weights to different terms based on the signal-to-noise ratio and other properties. This is derived from the score-matching connection, not just ad-hoc design.

How This Section Is Organized

The text tells us the section proceeds by "categorized by the terms of Eq. (5)":

• Section 3.1: Likely discusses $L_T$ (the pure-noise term)
• Section 3.2: Establishes the score-matching connection (the theoretical centerpiece)
• Section 3.3: Likely discusses intermediate terms $L_{t-1}$
• Section 3.4: Proposes the simplified, weighted objective

This organization shows how theory (the score-matching connection) guides practical choices (reweighting).

Key Takeaway

This section is fundamentally about justifying design choices through theory:

Without this section: "We tried various parameterizations and found one that works empirically."

With this section: "These design choices emerge naturally from a principled connection to denoising score matching, which explains why they work."

This kind of theoretical grounding is valuable because:

1. It builds confidence in the approach
2. It suggests which hyperparameters matter most
3. It connects this work to the broader literature on score-based generative models
4. It enables future improvements informed by score-matching theory

The mathematical details in Sections 3.2–3.4 will flesh out this connection and derive the weighted objective, but the conceptual insight is already clear: diffusion models are denoising score matchers in disguise.

Section 3.1: Forward Process and $L_T$ - Detailed Explanation

Big Picture: Why This Section Matters

Before diving into mathematics, let's understand what's happening here and why it's important:

The paper is describing how to train diffusion models by optimizing a variational bound (the loss function). In equation (5) from the background, this loss has three types of terms: $L_T$, $L_{t-1}$ (for various $t$), and $L_0$.

This section is about a crucial simplification: The authors are saying that one of these loss terms—specifically $L_T$—can be completely ignored during training because it's a constant. This is a huge practical advantage because it means less computation and simpler training. Let's understand why this works.

Understanding the Forward Process Setup

Recall from equation (5) in the background:

Let me define the key quantities here:

• $q(\mathbf{x}_T|\mathbf{x}_0)$: This is the distribution of the data after $T$ steps of noise addition, starting from the original data $\mathbf{x}_0$. Remember from equation (4) in the background: $q(\mathbf{x}_T|\mathbf{x}_0) = \mathcal{N}(\mathbf{x}_T; \sqrt{\bar{\alpha}_T}\mathbf{x}_0, (1-\bar{\alpha}_T)\mathbf{I})$

• $p(\mathbf{x}_T)$: This is the target distribution we've chosen—specifically $p(\mathbf{x}_T) = \mathcal{N}(\mathbf{x}_T; \mathbf{0}, \mathbf{I})$ (standard normal distribution)

• $D_{\mathrm{KL}}$: The Kullback-Leibler (KL) divergence, which measures how different two probability distributions are. It's non-negative, equals zero only when the distributions are identical, and higher values mean more different distributions.

The Key Insight: Why $L_T$ is Constant

Here's the crucial reasoning:

Step 1: Identify what has learnable parameters

In the training process, we have two things:

1. The forward process $q(\mathbf{x}_{1:T}|\mathbf{x}_0)$: This is defined by the variance schedule $\beta_1, \ldots, \beta_T$ (see equation 2). According to this section, we fix these variances to constants rather than learning them.

2. The reverse process $p_\theta(\mathbf{x}_{0:T})$: This is what we're training—the neural network learns parameters $\theta$ to specify the mean and covariance of the reverse process transitions.

Step 2: Analyze $L_T$ specifically

Notice what happens here:

• The left side $q(\mathbf{x}_T|\mathbf{x}_0)$ depends only on the forward process, which has no learnable parameters (we fixed $\beta_t$)
• The right side $p(\mathbf{x}_T) = \mathcal{N}(\mathbf{x}_T; \mathbf{0}, \mathbf{I})$ is a fixed standard normal distribution that doesn't depend on any parameters at all

Step 3: Why this means we can ignore it

Since neither side of the KL divergence in $L_T$ depends on the learnable parameters $\theta$:

• As we train and update $\theta$, the value of $L_T$ never changes
• It's a constant—like adding $+5$ to a loss function every training step
• When we take gradients with respect to $\theta$ for optimization, the derivative of a constant is zero: $\frac{\partial L_T}{\partial \theta} = 0$

From an optimization perspective, constants don't affect which direction to move the parameters. Therefore, we can simply drop $L_T$ from the training objective without changing the optimal solution.

Mathematical Formulation

Let me show this more formally. The full loss from equation (5) is:

During training, we optimize by computing gradients:

Since $L_T$ doesn't depend on $\theta$ (it only depends on the fixed forward process):

Therefore, we can train using:

without any loss in optimality.

Why This Design Choice Matters

Practical implications:

1. Simpler training: One fewer term to compute during each training step
2. Cleaner gradient flow: The optimization focuses only on the terms where the model can actually improve
3. Design justification: The authors note that this choice is "justified by simplicity and empirical results"

A philosophical note: The section explicitly states they ignore the possibility that $\beta_t$ could be learned through reparameterization. This is an intentional design choice—it simplifies the method and, as we'll see in Section 4, works empirically very well. Sometimes in machine learning, simpler approaches that fix certain components actually perform better than more complex ones.

Connection to the Bigger Picture

This section represents part of the authors' solution to a fundamental question: How do we design a diffusion model that trains efficiently and produces high-quality samples?

The answer involves:

• Fixing forward process variances (this section)
• Establishing connections to denoising score matching (Section 3.2)
• Using a weighted loss that emphasizes the right noise levels (Section 3.4)

By removing the constant $L_T$ term, we simplify the training objective so the model can focus on learning what actually matters: the reverse process at noise levels where the data still has meaningful signal.

Understanding Section 3.2: Reverse Process and the Mean Parameterization

The Big Picture

This section tackles a critical design decision in diffusion models: How should the neural network learn to predict the reverse process mean? In other words, when we're going backward from noise to images, what should our network actually output?

The section reveals something elegant: there are multiple mathematically equivalent ways to parameterize what the network predicts, and the authors show that predicting noise (denoted $\boldsymbol{\epsilon}_\theta$) is particularly effective. This choice connects diffusion models to a classical machine learning technique called denoising score matching, which provides theoretical justification for the approach.

Part 1: Setting the Variance (the Easy Choice)

Let's start simple. Recall from Equation (1) in the background that the reverse process is:

$p_\theta(\mathbf{x}_{t-1}|\mathbf{x}_t) = \mathcal{N}(\mathbf{x}_{t-1}; \boldsymbol{\mu}_\theta(\mathbf{x}_t, t), \boldsymbol{\Sigma}_\theta(\mathbf{x}_t, t))$

This says: "given noisy image $\mathbf{x}_t$ at step $t$, the next step back (less noisy) is normally distributed with mean $\boldsymbol{\mu}_\theta$ and variance $\boldsymbol{\Sigma}_\theta$."

The first choice: Make the variance fixed and time-dependent.

The authors set: $\boldsymbol{\Sigma}_\theta(\mathbf{x}_t, t) = \sigma_t^2 \mathbf{I}$

where:

• $\sigma_t^2$ is a constant (not learned) that depends only on timestep $t$
• $\mathbf{I}$ is the identity matrix (meaning independent, equal variance in all dimensions)
• The authors explore two options: $\sigma_t^2 = \beta_t$ or $\sigma_t^2 = \tilde{\beta}_t$

Why this makes sense: The variance of the reverse process should stay relatively small and consistent—we're taking small steps backward through the noise. By making it a constant based on the schedule, the network only needs to learn the mean, not worry about variance estimation.

Part 2: The Mean—Finding the Right Parameterization

Now for the interesting part: What should the network output for the mean?

Starting Point: The Direct Approach

From Equation (5), the loss term is $L_{t-1}$, which (after some calculation shown in Equation 8) equals:

$L_{t-1} = \mathbb{E}_q\left[\frac{1}{2\sigma_t^2}\|\tilde{\boldsymbol{\mu}}_t(\mathbf{x}_t, \mathbf{x}_0) - \boldsymbol{\mu}_\theta(\mathbf{x}_t, t)\|^2\right] + C$

What does this mean in words?

• $\tilde{\boldsymbol{\mu}}_t(\mathbf{x}_t, \mathbf{x}_0)$ (defined in Eq. 7) is the true posterior mean—where we should go when reversing the noise
• $\boldsymbol{\mu}_\theta(\mathbf{x}_t, t)$ is what our network predicts
• The loss measures the squared distance between them
• $C$ is just a constant that doesn't affect learning

The naive approach: Train a network to directly predict $\tilde{\boldsymbol{\mu}}_t$. This would work, but there's a better way.

The Key Insight: Reparameterization

The authors show that we can expand Equation (8) further using a clever algebraic trick. Recall from Equation (4) that we can write any noisy sample $\mathbf{x}_t$ as:

$\mathbf{x}_t = \sqrt{\bar{\alpha}_t}\mathbf{x}_0 + \sqrt{1-\bar{\alpha}_t}\boldsymbol{\epsilon}$

where $\boldsymbol{\epsilon} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ is random Gaussian noise.

What does this mean? Any point in the noisy distribution is a weighted combination of:

• The original image scaled by $\sqrt{\bar{\alpha}_t}$ (shrinks as $t$ increases)
• Pure noise scaled by $\sqrt{1-\bar{\alpha}_t}$ (grows as $t$ increases)

After substituting this reparameterization into the loss (Equations 9-10), the authors derive that the network must predict:

$\frac{1}{\sqrt{\alpha_t}}\left(\mathbf{x}_t - \frac{\beta_t}{\sqrt{1-\bar{\alpha}_t}}\boldsymbol{\epsilon}\right)$

The crucial observation: This expression involves the noise $\boldsymbol{\epsilon}$ that was added during the forward process! So instead of having the network predict $\tilde{\boldsymbol{\mu}}_t$ directly, we can have it predict the noise itself.

The Noise Prediction Parameterization (Equation 11)

The authors propose:

$\boldsymbol{\mu}_\theta(\mathbf{x}_t, t) = \frac{1}{\sqrt{\alpha_t}}\left(\mathbf{x}_t - \frac{\beta_t}{\sqrt{1-\bar{\alpha}_t}}\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t)\right)$

where $\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t)$ is a neural network trained to predict the noise.

Why is this genius? Let's break down what happens:

1. The network learns to denoise: $\boldsymbol{\epsilon}_\theta$ takes a noisy image and predicts what noise was added
2. The formula reconstructs the mean: Once we know the predicted noise, the fraction subtracts it out and rescales to get a less-noisy version
3. It connects to prior work: This becomes equivalent to denoising score matching, a classical technique

Sampling with the Noise Parameterization

When actually generating images (during inference), we sample:

$\mathbf{x}_{t-1} = \frac{1}{\sqrt{\alpha_t}}\left(\mathbf{x}_t - \frac{\beta_t}{\sqrt{1-\bar{\alpha}_t}}\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t)\right) + \sigma_t \mathbf{z}$

where $\mathbf{z} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ is fresh random noise.

Intuition:

• The first fraction removes predicted noise from the current step
• The $+ \sigma_t \mathbf{z}$ term adds back a little noise to maintain proper variance (because the process is stochastic)

Part 3: The Connection to Denoising Score Matching

With the noise parameterization in place, the loss simplifies dramatically to Equation (12):

$\mathbb{E}_{\mathbf{x}_0, \boldsymbol{\epsilon}}\left[\frac{\beta_t^2}{2\sigma_t^2 \alpha_t(1-\bar{\alpha}_t)}\left\|\boldsymbol{\epsilon} - \boldsymbol{\epsilon}_\theta(\sqrt{\bar{\alpha}_t}\mathbf{x}_0 + \sqrt{1-\bar{\alpha}_t}\boldsymbol{\epsilon}, t)\right\|^2\right]$

This is the key equation. What does it say?

• The network is trained to predict: actual noise $\boldsymbol{\epsilon}$ vs. predicted noise $\boldsymbol{\epsilon}_\theta$
• We're measuring squared error between them
• We're doing this for multiple noise scales (indexed by $t$)
• The weighting factor out front varies with the schedule

Why "denoising score matching"? Score matching is a classical technique where you learn to denoise data at various noise levels. The fact that diffusion models reduce to this objective is remarkable—it connects the diffusion model framework to decades of prior work.

Part 4: Why This Parameterization?

The section concludes by noting three possible parameterizations:

1. Predict $\tilde{\boldsymbol{\mu}}_t$: The mean directly (straightforward but less elegant)
2. Predict $\boldsymbol{\epsilon}$: The noise (what's discussed extensively)
3. Predict $\mathbf{x}_0$: The original clean image (tried but didn't work as well)

The authors chose option 2 because:

• It has strong theoretical justification (connection to denoising score matching)
• It resembles Langevin dynamics (a classical sampling technique from statistical physics)
• It empirically works better (verified in Section 4 experiments)

Summary: The Key Takeaway

Instead of making a neural network directly learn what the next reverse step should be, we make it learn to predict and remove noise. This is:

• ✓ Mathematically equivalent to the original formulation
• ✓ Connected to classical machine learning theory (denoising score matching)
• ✓ Connected to classical physics (Langevin dynamics)
• ✓ Empirically more effective in practice

The elegance of this section is showing that what might seem like an arbitrary choice (predict noise instead of mean) actually falls out naturally from careful mathematical analysis.

Section 3.3: Data Scaling, Reverse Process Decoder, and $L_0$

The Big Picture

This section addresses a practical but crucial problem: how do we handle the fact that images are discrete (pixel values are integers from 0 to 255) when our diffusion model is built on continuous Gaussian distributions?

Up until this point in the paper, the diffusion process has assumed continuous data. But real image data is discrete—each pixel is an integer. This section explains:

1. How to scale discrete image data appropriately
2. How to design the final step of the reverse process to convert back to discrete pixels
3. Why this approach lets us compute exact log-likelihoods without cheating

Let's work through this step by step.

Part 1: Data Scaling ($[-1, 1]$ normalization)

The Problem and Solution

Problem: Image pixels naturally range from 0 to 255 (integer values). The diffusion model's reverse process starts from $p(\mathbf{x}_T)$, which is a standard normal distribution (mean 0, variance 1). If we fed raw pixel values (0-255) into the neural network, the scales would be completely mismatched.

Solution: Scale all pixel values linearly to the range $[-1, 1]$.

Mathematically, if $x_{\text{pixel}} \in \{0, 1, \ldots, 255\}$, we transform it to:

x_{\text{scaled}} = \frac{2 \cdot x_{\text{pixel}}}{255} - 1 \in [-1, 1]

• Data in $[-1, 1]$ is naturally compatible with a standard normal prior (which also has zero mean and unit variance)
• The neural network $\boldsymbol{\epsilon}_\theta$ (or $\boldsymbol{\mu}_\theta$) now sees consistently scaled inputs throughout training
• This improves numerical stability and learning efficiency

Here's the subtle issue: our continuous Gaussian model describes $p_\theta(\mathbf{x}_0 | \mathbf{x}_1)$ as a continuous distribution. But we want to report a log-likelihood for discrete data—the actual integers from 0-255.

Look at Equation (13). For each pixel coordinate $i$, the probability of observing a discrete pixel value $x_0^i$ is:

The notation is a bit cryptic, so let's unpack it. The functions $\delta_+$ and $\delta_-$ define the boundaries of the integration interval:

Interpretation:

Since we scaled pixels from $\{0, 1, \ldots, 255\}$ to $[-1, 1]$, each discrete pixel value now corresponds to a small interval of width $\frac{2}{255}$ in the continuous space.

• For pixel value 0 (scaled to $-1$): the interval is $[-\infty, -1 + \frac{1}{255}]$ (all values from $-\infty$ up to just past $-1$)
• For pixel value 127 (scaled to $\approx -0.002$): the interval is $[-0.002 - \frac{1}{255}, -0.002 + \frac{1}{255}]$ (centered at $-0.002$ with width $\frac{2}{255}$)
• For pixel value 255 (scaled to $1$): the interval is $[1 - \frac{1}{255}, \infty)$ (all values from just below $1$ to $\infty$)

Why use $\infty$ at the boundaries? This ensures that probability mass beyond the valid pixel range gets assigned to the extreme pixel values. For example, any predicted value below $-1$ contributes to the probability of pixel 0.

Why This Approach is Clever

This discrete decoder has three important properties:

1. Lossless codelength: The integral represents the exact probability of observing a discrete pixel value, given the continuous Gaussian model. The variational bound computed using these probabilities is a true lower bound on the log-likelihood of discrete data.

2. No added noise: Unlike some VAE approaches that add noise to discrete data before processing, we don't corrupt the data. The discretization is in the model, not the data.

3. No Jacobian correction: The scaling from $\{0, 1, \ldots, 255\}$ to $[-1, 1]$ is linear and fixed. We don't need to worry about how this transformation affects probability densities (which would require computing Jacobians) because our final decoder directly models probabilities over the discrete values.

Part 3: Sampling and Reporting

During Sampling

When we're actually sampling images from the model:

1. We sample $\mathbf{x}_1$ from $p(\mathbf{x}_T)$ and run the reverse process (Algorithm 2) to get to $\mathbf{x}_1$
2. For the final step, instead of sampling from the discretized distribution in Equation (13), the authors simply output $\boldsymbol{\mu}_\theta(\mathbf{x}_1, 1)$ without adding noise

This is written as: "At the end of sampling, we display $\boldsymbol{\mu}_\theta(\mathbf{x}_1, 1)$ noiselessly."

Why no noise at the final step?

• The mean prediction $\boldsymbol{\mu}_\theta(\mathbf{x}_1, 1)$ is already optimized to be a good reconstruction of the original data
• Adding additional Gaussian noise would only blur the final image
• Since $\sigma_1$ is very small (remember, we're very close to the data in the reverse process), the Gaussian in Equation (13) is already quite peaked around the mean anyway

Part 4: Connection to Other Methods

The authors note that this discrete decoder strategy is similar to approaches used in:

• Variational Autoencoders (VAEs): VAEs also face the problem of modeling discrete data with continuous latent variables
• Autoregressive models: Models like PixelCNN also need to convert continuous models to discrete predictions

The reference to "more powerful decoders like conditional autoregressive models" hints that one could replace the simple Gaussian integral with something more sophisticated, but the current approach works well and is simpler.

Summary: Why All This Matters

To compute an exact log-likelihood on discrete image data using a continuous diffusion model, we need to:

1. Scale the data appropriately ($[-1, 1]$) so it's compatible with the continuous model
2. Design the final decoder to integrate the Gaussian over discrete pixel regions, giving us proper probabilities for integer values
3. Avoid shortcuts like rounding or adding fake noise, which would make our log-likelihood computation invalid

This section ensures that when the paper reports quantitative results (like log-likelihood or FID scores), they're computing real, valid metrics on the actual discrete data distribution, not approximations.

Section 3.4: Simplified Training Objective

Big Picture: Why This Section Matters

The authors are introducing a clever practical simplification to the theoretical training objective they derived in previous sections. Here's the key insight: The mathematically rigorous variational bound from Section 3.2 (Equation 12) is theoretically perfect but computationally cumbersome with complicated weighting factors. The authors propose dropping these weights to create a simpler objective that, surprisingly, works even better in practice.

This is an important moment in machine learning research—sometimes theory and practice diverge, and empirical evidence wins. The authors are being transparent about this trade-off.

Understanding the Simplified Objective (Equation 14)

The Basic Form

$L_{\text{simple}}(\theta) := \mathbb{E}_{t, \mathbf{x}_0, \boldsymbol{\epsilon}}\left[\left\|\boldsymbol{\epsilon} - \boldsymbol{\epsilon}_\theta(\sqrt{\bar{\alpha}_t}\mathbf{x}_0 + \sqrt{1-\bar{\alpha}_t}\boldsymbol{\epsilon}, t)\right\|^2\right]$

Let me break down what each component means:

What we're computing:

• A loss function (denoted by $L_{\text{simple}}$) that measures how well the neural network learns to denoise images
• The loss is the expected squared error between:
  • The true noise $\boldsymbol{\epsilon}$ (the actual random noise added to an image)
  • The predicted noise $\boldsymbol{\epsilon}_\theta(\cdot)$ (what the neural network thinks the noise is)

The expectation notation $\mathbb{E}_{t, \mathbf{x}_0, \boldsymbol{\epsilon}}[\cdot]$ means we're averaging over three random things:

• $t$: which timestep in the diffusion process (ranging from 1 to $T$)
• $\mathbf{x}_0$: which original clean image we start with
• $\boldsymbol{\epsilon}$: which specific noise was added

What Gets Plugged Into the Network

The network $\boldsymbol{\epsilon}_\theta(\cdot)$ receives:

1. A noisy image: $\sqrt{\bar{\alpha}_t}\mathbf{x}_0 + \sqrt{1-\bar{\alpha}_t}\boldsymbol{\epsilon}$

   • This is the forward process equation from Section 3.1
   • $\sqrt{\bar{\alpha}_t}\mathbf{x}_0$ is the original image scaled down by a factor that decreases with $t$
   • $\sqrt{1-\bar{\alpha}_t}\boldsymbol{\epsilon}$ is noise scaled up by a factor that increases with $t$
   • At small $t$, mostly signal; at large $t$, mostly noise

2. The timestep: $t$

   • The network needs to know which step it's at, because the noise characteristics change at each timestep

The Key Simplification: Dropping the Weights

To understand what was simplified away, compare Equation 14 to Equation 12:

Original (Equation 12) — with weighting factors: $\mathbb{E}_{\mathbf{x}_0, \boldsymbol{\epsilon}}\left[\frac{\beta_t^2}{2\sigma_t^2 \alpha_t(1-\bar{\alpha}_t)}\left\|\boldsymbol{\epsilon} - \boldsymbol{\epsilon}_\theta(\cdots)\right\|^2\right]$

Simplified (Equation 14) — without weighting factors: $\mathbb{E}_{t, \mathbf{x}_0, \boldsymbol{\epsilon}}\left[\left\|\boldsymbol{\epsilon} - \boldsymbol{\epsilon}_\theta(\cdots)\right\|^2\right]$

Notice that all the complicated coefficients in front (like $\frac{\beta_t^2}{2\sigma_t^2 \alpha_t(1-\bar{\alpha}_t)}$) have been removed. This makes training simpler and faster because:

• No need to compute or store these complex weighting factors
• Numerically more stable (no division by tiny numbers at early timesteps)
• Implementation is cleaner

Why This Simpler Loss Actually Works Better

The Loss Weighting Story

The authors explain that by discarding the theoretical weights, their simplified objective creates an implicit new weighting across different timesteps. Here's what happens:

At small $t$ (early denoising, small amounts of noise):

• The theoretical weight $\frac{\beta_t^2}{2\sigma_t^2 \alpha_t(1-\bar{\alpha}_t)}$ is very small
• This weight DOWN-WEIGHTS these terms in the original loss
• By dropping the weight entirely and using a uniform distribution over $t$, we DOWN-WEIGHT these terms even more
• Result: The network doesn't waste capacity learning to denoise almost-clean images

At large $t$ (late denoising, lots of noise):

• The theoretical weight is larger
• These hard denoising tasks get emphasized
• Result: The network focuses computational effort on the harder tasks

The Intuition

Think of it like learning to remove noise from photos:

• If a photo is 99% clear with 1% noise, it's easy—don't spend much time on this
• If a photo is 30% signal and 70% noise, it's hard—spend more time learning to do this well
• By down-weighting easy tasks, the network becomes better at hard tasks overall

This is counterintuitive but makes sense: a good denoiser should excel at difficult noise levels, because those determine the final sample quality.

The Three Cases: How $t$ Relates to the Losses

The authors clarify how their simplified objective connects back to the theoretical framework:

Case 1: $t = 1$ (First denoising step)

• This corresponds to $L_0$ from the variational bound
• Technically, Equation 13 (the discrete decoder) involves an integral
• The simplification approximates this integral by:
  • Treating the Gaussian like it's flat across each bin (using density times bin width)
  • Ignoring the variance $\sigma_1^2$ (treating it as small)
  • Ignoring edge effects at $-1$ and $+1$ boundaries

Cases 2 and beyond: $t > 1$

• These correspond to the denoising objectives from Equation 12
• The simplification uses an "unweighted version"—meaning we drop the weighting factor
• This is analogous to how NCSN (Noise Conditional Score Networks) weights their losses

Why $L_T$ is absent:

• Recall from Section 3.1: the forward process variances $\beta_t$ are fixed constants (not learned)
• Therefore $L_T$ becomes a constant that doesn't affect training
• No need to include it in the loss function

Sampling with the Simplified Objective

An important point: using a simpler training objective doesn't change how you generate samples. Once the network is trained on Equation 14, you still use Algorithm 2 from Section 3.2 to sample:

$\mathbf{x}_{t-1} = \frac{1}{\sqrt{\alpha_t}}\left(\mathbf{x}_t - \frac{\beta_t}{\sqrt{1-\bar{\alpha}_t}}\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t)\right) + \sigma_t \mathbf{z}$

The sampling procedure is unchanged; only the training loss changes.

Summary: The Core Contribution

The key takeaway: Sometimes dropping theoretical baggage and using a simpler loss with implicit reweighting produces better results. This is why empirical validation matters in machine learning.

Section 4: Experiments - A Detailed Explanation

Big Picture: What Are We Doing Here?

This section describes the practical implementation choices the authors made when training and using their diffusion model. Think of it as the "engineering manual" for the theory developed in earlier sections. The authors need to make concrete decisions about:

1. How many diffusion steps to use ($T$)
2. How much noise to add at each step (the $\beta_t$ values)
3. What neural network architecture to train

These aren't arbitrary choices—they're carefully calibrated based on the theory and earlier findings from related work. This section justifies each choice and explains the resulting design.

Part 1: Setting the Number of Steps and Noise Schedule

The Key Parameters: $T$ and $\{\beta_t\}$

The Number of Steps ($T = 1000$):

The authors set $T = 1000$, meaning the diffusion process has 1000 steps. This number was chosen to match previous diffusion work [53, 55]. Why does this matter? Recall from earlier sections that at each step, the model must make a neural network prediction (the $\epsilon_\theta$ function from Eq. (11)). More steps means more predictions during sampling, which affects computational cost. By matching previous work, they ensure fair comparison.

The Variance Schedule: Linear Increase from $\beta_1$ to $\beta_T$

Now for the noise levels. Remember from the forward process (mentioned in previous sections) that $\beta_t$ controls how much noise is added at step $t$. The authors set:

$\beta_1 = 10^{-4}, \quad \beta_T = 0.02, \quad \text{increasing linearly}$

What does this mean geometrically? Think of the diffusion process as a journey through noise space:

• At step 1: Add a tiny amount of noise ($\beta_1 = 0.0001$)
• At step 1000: Add substantially more noise ($\beta_T = 0.02$)
• In between: Increase smoothly and linearly

These values were "chosen to be small relative to data scaled to $[-1, 1]$" (as mentioned in Section 3.3). Since image data was scaled to the range $[-1, 1]$, these noise levels are appropriately small—they don't immediately destroy the image structure.

Why These Specific Values? Three Justifications

1. Maintaining Process Symmetry:

The authors state that "reverse and forward processes have approximately the same functional form." What does this mean mathematically?

In the forward process, at step $t$, we have: $\mathbf{x}_t = \sqrt{\bar{\alpha}_t}\mathbf{x}_0 + \sqrt{1-\bar{\alpha}_t}\boldsymbol{\epsilon}$

where $\bar{\alpha}_t = \prod_{s=1}^{t}(1-\beta_s)$ (this accumulates all the noise up to step $t$).