Towards interpreting any residual stream direction by scaling activation inversion

TL;DR

The paper introduces Maximally-Activating Example Metamodels (MAEMs), a method that generates text exemplars which strongly activate specific directions in a language model's residual stream. MAEMs use a label-free reinforcement learning objective to maximize cosine similarity between a target direction and the generated text's activations, enabling scalable interpretation across diverse directions.

Why It Matters

MAEMs offer a practical way to interpret model internals without relying on pre-existing corpora, enabling discovery of concepts in unseen domains and improving the interpretability of model behaviors such as steering vectors and backdoor payloads. The ability to verify exemplars via forward pass on the original model makes the approach more reliable than prior metamodels.

Key Findings

• MAEMs outperform corpus search and natural-language autoencoders in inverting held-out activations across multiple languages and domains.
• MAEMs activate 89% of held-out sparse autoencoder features, demonstrating broad coverage of model representations.
• MAEMs successfully recover LoRA backdoor payloads more frequently than corpus search and can generate text that reflects internal model reasoning.