Is Weight Tying Still Beneficial for Decoder-Only LLMs in Private Settings Under DP-SGD?

TL;DR

This paper investigates the impact of weight tying in decoder-only large language models under differentially private training. The results show that untied embeddings outperform weight-tied models, achieving gains in accuracy and enabling memory-efficient ghost clipping. This challenges the standard design choice of weight tying in language models.

Why It Matters

The findings have practical significance as they can improve the performance and scalability of privacy-preserving language models. This is crucial for applications where data privacy is a concern, such as in healthcare or finance. The results can also inform the design of more effective and efficient language models.

Key Findings

• Untied embeddings outperform weight-tied models under DP-SGD
• Untying embeddings enables memory-efficient ghost clipping
• Weight tying complicates standard ghost norm computation