Aligning Language Models with General Preferences: An Empirical Study of Direct Nash Optimization

TL;DR

This paper presents DNO-Prct, a practical implementation of Direct Nash Optimization for aligning language models to general preferences using on-policy sampling, LLM-based preference rating, and contrastive updates. It achieves significant performance gains on AlpacaEval 2.0 with minimal reliance on proprietary models, showing that open-weight annotators can retain most of the benefit of GPT-4o.

Why It Matters

The work demonstrates that high-quality alignment can be achieved without relying on expensive proprietary models, making preference-based training more accessible and scalable. This advances the practicality of RLHF alternatives and supports broader adoption of open-weight models in real-world applications.

Key Findings

• DNO-Prct improves length-controlled win rates by up to 13.6 percentage points on AlpacaEval 2.0 using only open-weight models for annotation.
• Removing gold responses, data look-back, or teacher-preferred pairs reduces performance by 6–10 LC points, highlighting their critical role.
• The method is robust to hyperparameter variations in sample count and margin size, with changes yielding ≤2.4 LC point shifts.