Less Decoder is More Encoder: Geometric Representation Learning from Novel View Synthesis

TL;DR

This paper introduces SNAP, a self-supervised encoder-decoder transformer for geometric representation learning from novel view synthesis. SNAP addresses limitations of existing methods by using a pose-conditioned local decoder and latent-space reconstruction objective. This approach yields competitive results with special-purpose geometry-supervised methods.

Why It Matters

The proposed method has practical significance in various computer vision tasks, such as visual localization, pose estimation, and robot manipulation. It also demonstrates viewpoint invariance and robustness to camera shifts, making it useful for real-world applications. This can lead to improved performance in tasks that require understanding 3D scene structure.

Key Findings

• SNAP achieves competitive results with geometry-supervised methods
• Emergent viewpoint invariance in patch features
• Restricting decoder expressivity prevents suppression of transferable geometric structure