Pretraining Data Scale Reverses the Muon-AdamW Fine-Tuning Gap

TL;DR

This paper reveals that Muon, while faster than AdamW during pretraining, underperforms AdamW in downstream fine-tuning when pretraining data is sufficiently large. The performance reversal is explained by the margin-to-spread ratio, which favors AdamW at scale due to better output calibration. The effect intensifies with model size, with AdamW overtaking Muon at smaller data-to-parameter ratios.

Why It Matters

The findings challenge the assumption that faster pretraining optimizers generalize better, guiding practitioners to choose optimizers based on data scale and model size. This insight is crucial for large-scale language model training pipelines where efficiency and downstream performance must be balanced.

Key Findings

• Muon outperforms AdamW in pretraining but lags in fine-tuning at large data budgets.
• The margin-to-spread ratio favors AdamW over Muon as pretraining data increases.
• In linear associative memory, gradient descent (AdamW-like behavior) surpasses Muon in margin-to-spread ratio with growing data, confirmed via simulations.