Low-Discrepancy Dither for Quantized Recurrent State Caches

TL;DR

This paper investigates rounding rules for quantized recurrent state caches in language models. It finds that deterministic golden-ratio Weyl dither consistently outperforms stochastic rounding. This is significant for memory bandwidth savings in production systems.

Why It Matters

The research has practical implications for improving the efficiency of language models. It can help reduce memory usage and increase decoding speed. This is particularly important for long-generation tasks and production environments.

Key Findings

• Deterministic golden-ratio Weyl dither outperforms stochastic rounding
• Round-to-nearest discards small updates, leading to growing error
• Discrepancy analysis explains the ordering of rounding rules