Scaled Idempotence in Transformer Attention: Paired OV Geometry and Shared-Value Algebras

TL;DR

The study reveals that a small fraction of attention heads in Transformers exhibit near-idempotent behavior under composition, with their output-value operators nearly satisfying T² ≈ αT. This property emerges from specific geometric orientations in key-value spaces and is significantly disrupted when key orientations are randomized, indicating trained alignment rather than inherent structure. Value sharing enables a local algebraic structure across heads, extending individual idempotence to a collective operator system.

Why It Matters

Understanding the geometric and algebraic regularities in attention mechanisms provides insight into how Transformers learn and represent information. This could inform more efficient architectures, better initialization strategies, and improved interpretability by identifying which structural properties are actively learned during training.

Key Findings

• A sparse subset of attention heads across multiple large-scale models exhibit strong squared closure, indicating near-idempotent behavior in their output-value operators.
• The geometric orientation of the key matrix is critical for achieving closure; randomizing orientation destroys the property, even when singular values and norms are preserved.
• When value matrices are shared across heads, the idempotent relation extends to a right-action algebra, forming a structured operator system with shared kernel and oblique projections.