What Does a Discrete Diffusion Model Learn?

TL;DR

The paper establishes a precise mathematical framework for discrete diffusion models by deriving the continuous-time Markov chain ELBO with exact boundary terms. It shows that the optimal model minimizes the divergence between the learned reverse process and the true reverse process, with the best possible performance limited only by information loss during noising. The analysis reveals that denoisers, cavity functions, and scores are equivalent representations of the same underlying object under different parameterizations.

Why It Matters

This work clarifies the fundamental objective of discrete diffusion models, resolving long-standing confusion about what different training losses actually optimize. It enables more reliable model design by showing which parameterization avoids numerical instabilities and ensures consistent training dynamics. The findings provide a foundation for developing better diffusion-based models across language and sequence modeling.

Key Findings

• Denoiser, cavity, and score functions are mathematically equivalent under token-factorizing noise, with exact conversion formulas between them.
• The bridge plug-in parameterization remains stable at initialization, unlike the denoiser, which causes the uniform ELBO to diverge.
• All known diffusion losses (MDM, UDM, SEDD, GIDD) are shown to optimize specific forms of the reverse process under different coordinate systems.