Metacognitive Decoding: Efficient Inference for Diffusion Language Agents

TL;DR

The paper presents a metacognitive controller that optimizes diffusion-based language agents by dynamically deciding when to finalize tokens and how far to generate, reducing computational cost during inference. This lightweight module is trained on agent interaction data to balance response quality and efficiency, achieving lower FLOPs without sacrificing accuracy.

Why It Matters

Efficient inference is critical for deploying language agents in real-time applications. By reducing computational demands while maintaining performance, this method enables faster, more scalable agent deployment, especially in resource-constrained environments.

Key Findings

• The metacognitive controller reduces average inference FLOPs by 20.6% compared to the most efficient baseline across five benchmarks.
• The controller learns from agent trajectories to adapt generation length and termination timing based on task context and interaction needs.
• Ablation studies confirm that both dynamic termination and adaptive generation extent contribute significantly to efficiency gains.