When Pruning Meets Interpretability: Preserving Sparse Autoencoder Robustness in LLMs

TL;DR

This paper investigates the impact of pruning on sparse autoencoders in large language models, revealing that activation-aware methods preserve functionality better than magnitude pruning. The study proposes a layer-wise sparsity allocation strategy to achieve lower perplexity. Experiments validate the theoretical findings across four model architectures.

Why It Matters

Understanding how pruning affects sparse autoencoders is crucial for reliable model compression and interpretability in large language models. This research has practical implications for developing more efficient and robust language models. The findings can inform the design of more effective model compression techniques.

Key Findings

• pruning affects SAE behavior
• activation-aware methods are more robust
• middle layers are more sensitive to pruning