All modalities are equal, but video is more equal: Closing the Cross-Attention Gap in Joint Video Generation

TL;DR

This paper introduces RecCAR, a regularization technique to improve cross-modal correspondence in joint video generation. RecCAR aligns weaker modality-to-video correspondences with well-established video-to-modality correspondences. This improves video generation quality and synchronization.

Why It Matters

The proposed method enhances video generation by leveraging multimodal information, which is crucial for applications like video editing, generation, and analysis. Improved cross-modal correspondence can lead to more realistic and coherent video outputs. This has significant implications for computer vision and multimedia research.

Key Findings

• joint multimodal diffusion transformers exhibit asymmetry in cross-modal correspondence
• RecCAR improves human anatomy score and reduces audio-video desynchronization
• reciprocal cross-modal attention regularization enhances video generation quality