CMA-OT: Hierarchical Expert Supervision for Dance-to-Music Generation

TL;DR

CMA-OT introduces a hierarchical supervision framework for dance-to-music generation that uses an external music expert to guide the generator's latent representations across multiple scales. It employs curriculum learning and scale-aware optimal transport to align semantic and structural features, improving rhythmic accuracy and musical coherence. The method outperforms prior approaches on benchmark datasets.

Why It Matters

This approach enables more musically expressive and structurally consistent music generation from dance inputs, advancing applications in creative AI, choreography assistance, and interactive performance systems. It addresses a core challenge in multimodal generation: aligning sparse motion cues with dense audio content.

Key Findings

• Hierarchical supervision using an external music expert improves latent representation learning in dance-to-music generation.
• Curriculum-guided multi-scale learning enables stable and effective transfer of musical knowledge from expert to generator.
• Scale-aware optimal transport achieves fine-grained alignment between expert and generator representations despite temporal misalignment.