Six Layers Less: Encoder Pruning for Whisper with Label-Free Recovery

TL;DR

This paper proposes an approach to prune encoder layers in Whisper, a transformer-based ASR model, by ranking layers based on their impact on Word Error Rate. The pruned model achieves competitive performance after distillation using unlabeled monolingual speech data. This approach simplifies inference without requiring custom code.

Why It Matters

The proposed method enables efficient deployment of large ASR models on resource-constrained devices. It also demonstrates the potential for label-free recovery of performance degradation caused by model pruning. This can lead to significant speedups in end-to-end transcription tasks.

Key Findings

• Encoder layer pruning using leave-one-layer-out WER change
• Removal of 18.5% of encoder layers without custom inference code
• Label-free recovery of performance degradation using unlabeled monolingual speech data