Reinforcement Learning with Visually Attributable On-Policy Self-Distillation

TL;DR

The paper introduces a method that improves reinforcement learning in multimodal tasks by ensuring token-level corrections from a teacher model are grounded in visual evidence. It uses a gating mechanism to filter out corrections influenced by non-visual factors, preserving only those aligned with actual visual changes. This leads to more reliable and accurate learning across multiple visual reasoning benchmarks.

Why It Matters

By ensuring corrections are tied to actual visual evidence, the approach reduces the risk of learning from misleading signals, which is critical for building trustworthy AI systems in real-world applications like robotics and visual question answering.

Key Findings

• Privileged-correction contamination occurs when token updates are influenced by non-visual factors, leading to unreliable learning signals.
• The proposed Visual-Attribution Gating mechanism effectively identifies and suppresses corrections not supported by visual evidence.
• The method achieves state-of-the-art results on six visual reasoning benchmarks, outperforming existing on-policy self-distillation and GRPO-based approaches.