Base Models Can Reason By Taking a Cue From Training Data

Sophie L. Wang, Amil Dravid, Rulin Shao, Kevin Farhat, Sewon Min, Alexei A. Efros

arXiv: 2610.06851Published: October 5, 20267 upvotes

AI summary

TL;DR

This paper shows that base models can reason competitively with RL-trained models by using specific token cues. The cues are learned from training data and can be edited to elicit different reasoning behaviors. Fixing these cues can recover much of the performance gain of RL-trained models.

Why It Matters

The findings have practical significance for improving language model performance and safety. By understanding how token cues affect reasoning, developers can design more effective prompts and training data. This can lead to more reliable and trustworthy AI systems.

Key Findings

token cues improve base model performance
RL-trained models learn to use effective cues
cues correlate with specific document types in training data

Topics

language models

reinforcement learning

natural language processing

transformer

Thank you to arXiv for use of its open access interoperability. arxiv.org. aibrief.ai is not affiliated with or endorsed by arXiv.