ViT-H: Generative Video-To-Humanoid Control with Motion Tracking Feedback

TL;DR

ViT-H is a video-to-humanoid control system that generates reference motions directly from monocular videos using a diffusion model, then improves motion executability via post-training with motion-tracking feedback. It supports real-time teleoperation with low latency and outperforms prior reconstruction-retargeting-tracking methods in both motion quality and execution fidelity.

Why It Matters

This approach enables scalable acquisition of diverse humanoid behaviors from video without intermediate reconstruction steps, reducing pipeline complexity and improving real-world motion execution. The integration of tracking feedback during training enhances motion feasibility, making it more suitable for deployment on physical robots.

Key Findings

• Direct generation of humanoid reference motions from video using a diffusion model eliminates the need for separate motion reconstruction and retargeting stages.
• Post-training with motion-tracking feedback via direct preference optimization significantly improves motion executability across different motion trackers.
• An autoregressive variant of ViT-H achieves real-time teleoperation with sub-400 ms latency, demonstrating strong practical viability.