A Better Critic Matters for Distribution Matching Distillation

TL;DR

This paper shows that critic quality is critical for Distribution Matching Distillation (DMD), with joint scaling of critic and teacher capacity preventing training collapse. Introducing a step-aware critic (STAC) that conditions on generation step improves distillation performance across image, text-to-image, and text-to-video tasks.

Why It Matters

Reliable critic design enables more efficient and stable training of small, fast student models using large teachers—key for deploying high-quality generative models in resource-constrained environments. The findings offer practical guidelines for scaling DMD in multimodal generation systems.

Key Findings

• Fixed-size critics cause student quality degradation and training collapse when paired with larger teachers.
• Joint scaling of critic and teacher capacity prevents collapse and enables student benefits from larger teachers.
• Step-aware critic (STAC) improves distillation by estimating step-specific scores, outperforming step-agnostic critics.