SimRetract: Zero-Shot Sim-to-Real Robotic Assistance for Retractor Take-Over in Open Surgery

TL;DR

SimRetract enables zero-shot sim-to-real transfer for robotic retractor take-over in open surgery using only simulated demonstrations. The framework improves transferability by training policies on masked instrument images and grasp geometry, enabling robustness to execution drift and off-nominal states.

Why It Matters

This work eliminates the need for costly real-world demonstrations and fine-tuning, making robotic surgical assistance more accessible and scalable. It demonstrates a viable path toward deploying AI-driven surgical robots in real operating rooms without extensive real-world data collection.

Key Findings

• The policy achieves 90% grasp success and 78% retraction preservation in zero-shot real-world deployment.
• Removing the instrument mask in real-world trials drops grasp success to 22%, highlighting the importance of simulation design for transfer.
• Performance degradation is localized to visual and geometric sensing components, suggesting targeted improvements can close the sim-to-real gap.