GoDeep: Annotation-Free Open-Vocabulary 3D Scene Understanding via Language-Space Lifting

TL;DR

The GoDeep method achieves open-vocabulary 3D scene understanding without annotations by using a vision-language model as a translator. It produces entity-level descriptions of images and aggregates them in a language-only embedding space. This approach is competitive with strong baselines and localizes out-of-vocabulary objects.

Why It Matters

GoDeep's annotation-free approach enables efficient 3D scene understanding, which is crucial for applications like robotics, autonomous vehicles, and cultural heritage preservation. The method's explainability and ability to localize out-of-vocabulary objects make it a significant contribution to the field. This technology can be applied to various domains, improving the accuracy and efficiency of 3D scene understanding tasks.

Key Findings

• competitive performance with annotation-free baselines
• localization of out-of-vocabulary objects
• explainability of predictions at the point level