Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering

TL;DR

Imagine3D-LLM enhances multimodal large language models by enabling them to generate a compact 3D scene representation before answering spatial reasoning questions. It uses learnable summary tokens to reconstruct a 3D Gaussian Splatting model via photometric loss, improving cross-view understanding without relying on detailed geometric supervision. This approach boosts performance across 3D reasoning benchmarks by fostering stronger inter-view feature alignment.

Why It Matters

The method offers a practical way to improve 3D understanding in MLLMs without requiring complex 3D data or fine-grained geometric annotations. By mimicking human-like scene imagination, it enables more robust reasoning in real-world applications like robotics, AR/VR, and autonomous navigation.

Key Findings

• The model generates a compact 3D scene representation using learnable summary tokens that are trained to reconstruct a 3D Gaussian Splatting model.
• Photometric reconstruction loss on summary tokens improves cross-frame feature correspondence in the underlying image encoder, even without direct supervision on image tokens.
• Imagine3D-LLM achieves state-of-the-art performance on multiple 3D spatial reasoning benchmarks, outperforming methods relying on explicit 3D geometry or pixel-level alignment.