Telescopic Language Models

TL;DR

Telescopic Language Models (TLMs) are trained to function as a continuous family of models across different capacity levels using a single training run. They achieve this through stochastic prefix supervision, enabling valid language models at every layer depth without architectural changes. TLMs reduce the quality-budget curve area by 43-44% compared to fixed-exit models while using 12% fewer GPU resources per run.

Why It Matters

Deploying language models across diverse hardware with varying compute limits typically requires separate training or compression for each target size. TLMs eliminate this overhead by enabling a single model to serve multiple budgets efficiently, reducing training costs and infrastructure complexity.

Key Findings

• A single TLM training run produces valid language models at every layer prefix without architectural modifications.
• Stochastic prefix supervision with full-capacity targets enables consistent performance across all depths, outperforming fixed-exit models in both perplexity and downstream tasks.
• The prefix sampling strategy acts as a tunable control, allowing trade-offs between continuum coverage and performance at specific depths.