Look Less, Think Faster: Joint Token-Compute Adaptation for Multimodal LLMs

TL;DR

SmartVL is a unified framework for multimodal LLMs that jointly adapts visual token selection and language model computation during inference. It dynamically allocates resources based on input content and compute budgets using coordinated controllers trained with a shared budget encoding and differentiable latency estimation.

Why It Matters

This approach enables more efficient deployment of multimodal models in real-world applications where computational resources are limited. By optimizing both vision and language components together, it achieves better performance per unit of compute compared to methods that optimize each component in isolation.

Key Findings

• SmartVL jointly controls visual token count and LLM compute usage through coordinated controllers trained end-to-end.
• The framework uses a shared budget encoding and differentiable latency estimator to enable cross-stage resource allocation.
• Experiments show SmartVL achieves superior accuracy-efficiency trade-offs across multiple multimodal benchmarks.