Cortex: A Bidirectionally Aligned Embodied Agent Framework for Long-horizon Manipulation

TL;DR

Cortex introduces a bidirectionally aligned framework that bridges high-level planning and low-level execution in long-horizon manipulation tasks. It uses 32 standardized skill primitives and tractability principles to generate large-scale training data, enabling zero-shot generalization to unseen real-world tasks through a generalist vision-language model and fine-tuned vision-language-action model.

Why It Matters

The framework enables robots to perform complex, multi-step real-world tasks without task-specific retraining, significantly advancing autonomous manipulation in unstructured environments. Its ability to generalize across domains, such as chemistry experiments, demonstrates practical viability for real-world deployment.

Key Findings

• Cortex achieves 3.1% higher success rate than monolithic baselines on Libero-long and 4.1% on RoboTwin in long-horizon manipulation tasks.
• The use of standardized skill primitives and tractability principles enables automatic annotation of over 4,000 hours of open-source video and 30 hours of simulation data.
• Zero-shot execution of unseen real-world tasks is possible by combining a generalist VLM with a fine-tuned VLA, a capability not achievable through VLA fine-tuning alone.