Aligning Thoughts with Answers: Probability Rewards to Tame Thinking Drift

TL;DR

This paper proposes Rita, a novel RL paradigm to improve thinking-answer consistency in vision-language models. Rita introduces two reasoning-label-free RL rewards and a difficulty-aware data filtering strategy. This approach outperforms supervised finetuning and vanilla RL-finetuned frameworks.

Why It Matters

The proposed method has practical significance as it enhances the reliability of vision-language models in real-world applications. Improved thinking-answer consistency can lead to more accurate and trustworthy results. This can be particularly important in applications where incorrect reasoning can have significant consequences.

Key Findings

• thinking drift in IoU-based RL frameworks
• effectiveness of Rita's reasoning-label-free RL rewards
• importance of difficulty-aware data filtering