Pixel-UMI: Learning Environment-Aware Whole-Body Manipulation from Sensorized Human Demonstrations

TL;DR

Pixel-UMI learns environment-aware whole-body manipulation policies from sensorized human demonstrations by introducing Environment-Aware Retargeting (EAR) and Retargeting-Error-Aware Masking (REAM) to handle embodiment differences and viewpoint mismatches. It further improves performance via on-policy human corrections using DAgger, achieving 87.0% average task progress across diverse manipulation tasks.

Why It Matters

This work enables scalable, robust learning of complex whole-body manipulation from human data, overcoming key challenges like embodiment differences and contact-rich interactions—critical for real-world robotic deployment in unstructured environments.

Key Findings

• EAR effectively maps human head and gripper trajectories to collision-aware robot joint and base targets using scene geometry.
• REAM reduces reliance on mismatched visual observations during training by leveraging retargeting errors, improving deployment robustness.
• On-policy human corrections via DAgger significantly enhance performance in contact-rich and bimanual tasks.