LLaCO: Continuous Diffusion for Large Language Models with Competition-Aware Semantic Scheduling

TL;DR

LLaCO introduces a competition-aware semantic scheduling framework for continuous diffusion in large language models, addressing temporal compression caused by embedding norm decay. It proposes the Competition Threshold Theory and Semantic Retention Scheduling to align time allocation with identity decay, using a spherical noising process to preserve token competition and improve generation quality.

Why It Matters

This work advances non-autoregressive text generation by solving fundamental geometric misalignments in continuous diffusion, enabling more accurate and efficient language modeling without sacrificing quality. It bridges theoretical insights with practical improvements, offering a scalable alternative to discrete diffusion models.

Key Findings

• Embedding norm decay causes spurious overtaking in Euclidean space, misaligning temporal progression with semantic trajectory.
• Competition Threshold Theory formalizes directional token competition, showing high-dimensional overtaking requires extreme noise levels.
• Semantic Retention Scheduling and spherical noising enable proportional denoising along semantic paths, improving identity preservation and generation quality.