UE5M3 FP4 Block Scaling for Stable Language Model Pretraining

TL;DR

The paper proposes a simpler recipe for stable 4-bit floating-point pretraining using unsigned E5M3 block scales. This approach achieves lower training and validation loss compared to NVIDIA's Transformer Engine. The results demonstrate the effectiveness of end-to-end software-emulated pretraining with the proposed recipe.

Why It Matters

The proposed recipe has practical significance as it enables more efficient and accurate pretraining of large language models. This can lead to improved performance in various natural language processing tasks. The simpler recipe also motivates native support for unsigned E5M3 block scaling, which can further accelerate pretraining.

Key Findings

• Lower final-window training loss
• Lower validation loss
• Higher quantized-inference downstream point estimates