Prefix Sliding for efficient test-time scaling

Niklas Muennighoff, Zhengyang Wang, Zeyi Chen, Weijia Shi, Binyuan Hui, John Yang, Dapeng Jiang, Mika Senghaas, Fares Obeid, Johannes Hagemann, Sami Jaghouar, Ludwig Schmidt, Percy Liang, Jason Wei, Andrew Y. Ng, Luke Zettlemoyer, Yejin Choi, Mike Lewis

arXiv: 2608.26070Published: August 26, 20264 upvotes

View on arXiv Read PDF

AI summary

TL;DR

The paper proposes Prefix Sliding, a method to efficiently scale language models at test-time by discarding intermediate tokens. This approach enables long-horizon reasoning without increasing memory requirements. It achieves 3x speedup without training and better performance with reinforcement learning.

Why It Matters

Prefix Sliding improves the efficiency of test-time scaling, allowing for more complex and longer reasoning processes. This has practical significance for applications requiring lengthy reasoning, such as problem-solving and text generation. It enables the use of more powerful models in resource-constrained environments.

Key Findings

most intermediate reasoning tokens lose importance over time
prefix sliding outperforms summarizing intermediate tokens and vanilla sliding window
training with prefix sliding achieves better performance

Topics

language models

test-time scaling

reinforcement learning

efficient inference

transformer

Thank you to arXiv for use of its open access interoperability. arxiv.org. aibrief.ai is not affiliated with or endorsed by arXiv.