Where and Why is Grounded Reasoning Useful?

TL;DR

The study evaluates grounded reasoning in visual tasks by analyzing how models use spatial coordinates in their reasoning traces. It finds that grounding helps only when tasks involve visual elements that cannot be described in words, which are rare in current benchmarks. To address this, the authors introduce BrickVision, a new benchmark requiring such undescribable references.

Why It Matters

Many existing visual reasoning benchmarks rely on describable references, making grounded reasoning appear unnecessary. BrickVision creates a more realistic testbed for models that must reason about unverbalizable visual details, pushing the development of more robust multimodal systems.

Key Findings

• Grounded reasoning improves performance on counting tasks but offers no consistent advantage across general spatial reasoning and perception benchmarks.
• The benefit of grounding is limited to tasks involving undescribable visual references—concepts that cannot be effectively communicated through language.
• Most current benchmarks use only describable references, creating a mismatch between model capabilities and evaluation tasks, which BrickVision aims to correct.