MoSE3: Learning World-Space SE(3) at Every Pixel

TL;DR

MoSE3 introduces a feed-forward neural network that predicts full 6-DoF rigid motion (SE(3)) for every pixel in monocular video, using learned 3D point tracks and rigidity embeddings to recover transforms via differentiable clustering. It leverages a new synthetic dataset, Art-Kubric, with dense SE(3) and rigidity annotations to enable end-to-end training without real-world supervision. The model achieves top performance on both rigid and articulated motion benchmarks and generalizes well to real videos despite training on synthetic data.

Why It Matters

This work enables detailed, per-pixel 3D motion understanding in dynamic scenes without requiring complex multi-view setups or manual annotations. It opens new possibilities for robotics, AR/VR, and autonomous systems by providing rich motion semantics at pixel-level granularity. The ability to generalize from synthetic data to real-world videos reduces reliance on costly real-world labeling.

Key Findings

• MoSE3 is the first model to predict dense SE(3) motion directly from monocular video in a feed-forward manner.
• It uses differentiable clustering of soft rigid groups to recover 6-DoF transforms from learned 3D tracks and rigidity embeddings.
• The model outperforms prior methods on SE(3) estimation and 3D point tracking across multiple benchmarks, including real-world video generalization.