Architecture-Dependent Fusion Pathways in MLLMs

TL;DR

The study compares how two types of multimodal large language models fuse visual and textual data across layers. It finds that concatenation-based models process text first and integrate vision later, while native multimodal models enable earlier joint adaptation and reorganization of feature spaces. These insights are validated through causal interventions and alignment analysis.

Why It Matters

Understanding fusion mechanisms enables better design and debugging of multimodal systems. This work provides diagnostic tools to assess how different architectures handle cross-modal information, supporting more reliable and interpretable AI models in real-world applications.

Key Findings

• Concatenation-based MLLMs exhibit a sequential fusion pattern where text features are processed before visual features are integrated.
• Native multimodal architectures show earlier co-adaptation between visual and textual representations, with more dynamic feature-space reorganization.
• Causal interventions confirm that modality-specific changes and attention routing are key to distinguishing fusion pathways across architectures.