Density-Aware Reward Scaling: A Reinforcement Learning Framework for Reward-Proportional Sampling

TL;DR

DARS introduces a lightweight, model-free reward scaling method that promotes diverse sampling in reinforcement learning by inversely scaling rewards based on local outcome density. It enables reward-proportional sampling in continuous spaces without modifying policy gradients or requiring auxiliary models, improving diversity in multimodal navigation and text-to-image generation.

Why It Matters

Many RL applications require diverse high-quality outputs—such as varied image generations or navigation paths—but standard methods often collapse to a single mode. DARS offers a plug-and-play solution that enhances diversity without compromising performance or increasing complexity.

Key Findings

• DARS prevents single-mode collapse in multimodal navigation by promoting reward-proportional sampling.
• DARS-GRPO improves image and prompt diversity across datasets and reward models while preserving semantic quality.
• The method requires no learned density model, critic, or auxiliary objective, making it easy to integrate into existing RL pipelines.