How to Loop MoE: Flatten the Experts, Untie the Attention

TL;DR

This paper introduces Foil, a method to improve the performance of looped mixture-of-experts (MoE) models by flattening the experts and untying the attention. Foil outperforms the unflattened looped baseline in pretraining loss and downstream accuracy. The method achieves better expert usage and more balanced routing.

Why It Matters

The proposed method has practical significance as it can lead to more efficient and effective use of model parameters, resulting in better performance and reduced computational costs. This can be particularly important for large-scale language models and other applications where computational resources are limited. Improved expert usage can also lead to more specialized and accurate models.

Key Findings

• Foil outperforms the unflattened looped baseline in pretraining loss and downstream accuracy
• Flattening the experts and untying the attention leads to more balanced and confident routing
• The returns of looping and widening the expert layers amplify each other