STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization

TL;DR

STEPQuant is a precision-aware quantization method for recurrent states in linear attention models, dynamically allocating bit budgets based on error sensitivity and memory persistence. It improves accuracy under low-bit constraints by modeling spatial and temporal error impacts and jointly optimizing scaling factors. The approach achieves over 5x compression and up to 68.7% memory reduction in serving without sacrificing accuracy.

Why It Matters

Reducing memory usage in large language model serving is critical for scalability and cost efficiency. STEPQuant enables high compression of recurrent states without accuracy loss, making real-time inference on large models more feasible on limited hardware.

Key Findings

• Quantization errors in recurrent states propagate over time and vary by spatial location, with long-lived states and certain key rows having higher impact on output quality.
• STEPQuant adapts precision allocation across time and space by analyzing error sensitivity and state magnitude distributions, leading to better accuracy than uniform quantization.
• Integrated into SGLang with optimized kernels, 6-bit STEPQuant achieves over 5x compression and up to 68.7% reduction in total serving memory while matching FP32 accuracy.