RoGe: Novel View Synthesis via End-to-End Implicit Reconstruction and Generation

TL;DR

RoGe introduces an end-to-end framework for novel view synthesis that unifies implicit scene reconstruction and video generation without relying on explicit 3D intermediates. It uses ray-queried geometric features from a feed-forward reconstruction model as conditioning for a video diffusion model, enabling joint training that improves both geometry and temporal coherence.

Why It Matters

This approach enables high-quality, temporally consistent video generation from sparse input views without error-prone intermediate representations, advancing applications in immersive media, robotics, and virtual environments where accurate scene modeling is critical.

Key Findings

• Ray-queried implicit features outperform rendered RGB and raw reconstruction tokens as conditioning for video generation.
• Joint training of reconstruction and generation modules leads to better geometric accuracy and temporal consistency compared to separate training.
• RoGe achieves state-of-the-art results on DL3DV in both image quality and video coherence metrics across multiple baselines.