φ-RIE: From Photorealistic Reconstruction to Interactive Environments

TL;DR

φ-RIE introduces a method to convert photorealistic 3D scene reconstructions into interactive environments by extracting movable object assets while preserving the background. It couples asset creation with scene completion, ensuring visual and physical consistency during simulation. The approach improves object matching accuracy and enables realistic manipulation in simulated settings.

Why It Matters

This enables robots to interact with real-world scenes in simulation using photorealistic, physically consistent environments, bridging the gap between perception and action. It supports more accurate training and testing of robotic systems in complex, dynamic scenarios.

Key Findings

• Coupling asset creation with scene completion improves object matching F1 by 14.6% at 20 mm threshold.
• The method preserves unedited scene content while enabling object manipulation and occlusion handling.
• Converted scenes support executable interactions, outperforming single-generator baselines in manipulation tasks.