GAE: Learning a Geometry-Native Latent Space for 3D-Consistent World Generation

TL;DR

The paper introduces a geometry-native latent space via a geometry-aware autoencoder (GAE) that enables consistent 3D generation by encoding appearance, depth, camera poses, and point maps in a shared compact representation. This latent space improves both visual fidelity and 3D coherence when used with a conditional flow model, outperforming appearance-centric latents in objective metrics.

Why It Matters

By aligning generation and perception through a unified geometry-native latent space, the approach enables more reliable 3D-consistent video and image synthesis, which is crucial for applications like virtual environments, robotics, and immersive media. It establishes a shared foundation that bridges perception and generation without requiring explicit 3D modeling.

Key Findings

• GAE's latent space jointly encodes appearance, depth, camera parameters, and point maps, enabling geometry-consistent generation.
• Replacing standard latents with GAE reduces FVD by 12.7% and 23.1% on RealEstate10K and DL3DV, respectively, indicating improved visual quality.
• Camera trajectory error is reduced by 50% on RealEstate10K, demonstrating stronger 3D coherence in generated sequences.