Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention

TL;DR

Latent Sparse Attention (LSA) enables efficient ultra-long context modeling by hierarchically clustering keys in a learned latent space, reducing attention complexity from quadratic to subquadratic. D-LSA and N-LSA variants achieve significant speedups in prefill and decode phases while maintaining accuracy on tasks with up to 10M tokens. LSA enables practical deployment of long-context models without sacrificing performance.

Why It Matters

Ultra-long context is critical for complex agentic workflows like scientific reasoning, but full attention becomes infeasible beyond 1M tokens. LSA provides a scalable, accurate solution that drastically reduces computational cost, making long-horizon AI tasks viable on existing hardware.

Key Findings

• LSA reduces attention complexity to subquadratic by scoring hierarchical clusters instead of individual keys.
• D-LSA achieves 72.4x speedup in prefill and 42.5x in decode over full attention at 10M tokens with minimal accuracy loss.
• N-LSA matches full attention accuracy with 1.6x training speedup per layer at 128k tokens, enabling native training on long contexts.