Language Models Learn to Improve Language Models

TL;DR

This paper proposes LM2LM, a method that trains large language models to generate effective proposals for improving language models using reinforcement learning. LM2LM addresses the challenges of expensive training and unreliable surrogate feedback. It enables a model to outperform others with lower data-generation cost.

Why It Matters

The proposed method has practical significance as it can automate AI research, including language model development, and lead to recursive self-improvement. This can reduce the cost and time required for training large language models. It also has the potential to improve the performance of language models in various applications.

Key Findings

• LM2LM trains large language models with reinforcement learning
• it uses a research world model to provide scalable feedback
• it generalizes to unseen model families