LongZero: Enhancing Long-Context Reasoning in LLMs from Zero Data

TL;DR

LongZero enhances long-context reasoning in LLMs without external data. It synthesizes questions, answers, and evidence, then expands into multi-document contexts. LongZero outperforms baselines on five benchmarks.

Why It Matters

Improving long-context reasoning in LLMs can lead to better performance on complex tasks. LongZero's approach reduces reliance on expensive human annotation and scarce natural data. This can accelerate progress in natural language processing.

Key Findings

• LongZero improves long-context reasoning without external data
• Outperforms RL on synthesized data from external corpora
• Generalizes to real tasks on five benchmarks