KV-streams for Efficient Compaction in Agentic Reinforcement Learning

Emiliano Penaloza, Dane Malenfant, Dheeraj Vattikonda, Roger Creus Castanyer, Siddarth Venkatraman, Abhay Puri, Jonathan Light, Matthew James Sargent, Augustine N. Mavor-Parker, Massimo Caccia, Lucas Caccia, Glen Berseth, Esmeralda S. Whitammer, Alessandro Sordoni, Minseon Kim, Marc-Alexandre Côté, Laurent Charlin, Guillaume Lajoie

arXiv: 2609.35750Published: September 28, 2026

View on arXiv Read PDF

AI summary

TL;DR

The paper proposes KV-streams, a strategy to improve context compaction in agentic reinforcement learning, enabling scalable compaction and increasing training throughput. KV-streams achieve a 2.6 to 5x wall-clock speedup in training without hindering performance. This approach allows for efficient trainable compaction.

Why It Matters

KV-streams address the bottleneck of fitting long context traces in GPU memory, enabling the scaling of agentic LLMs. This has practical significance for applications requiring long-term memory and efficient training. The approach can be applied to various compaction strategies, making it a valuable contribution to the field.

Key Findings

KV-streams enable scalable compaction
2.6 to 5x wall-clock speedup in training
streamed KV cache acts as a recurrent state

Topics

reinforcement learning

llms

compaction

gpu memory

agentic

Thank you to arXiv for use of its open access interoperability. arxiv.org. aibrief.ai is not affiliated with or endorsed by arXiv.