CatchUp: Depth-Aware Re-Admission of Visual Tokens for Efficient Large Vision-Language Models

TL;DR

CatchUp is a training-free method that improves efficiency in large vision-language models by dynamically re-admitting visual tokens at deeper decoder layers using depth-aware relevance assessment. It enhances re-admitted token representations by approximating their evolution through transitions from semantically and spatially similar tokens, leading to better grounding performance. The approach achieves high performance retention with significantly fewer active tokens across multiple benchmarks.

Why It Matters

Efficient inference in large vision-language models is critical for real-world deployment, especially on resource-constrained devices. CatchUp enables substantial reductions in computation and latency without requiring retraining, making advanced multimodal models more practical and scalable.

Key Findings

• Re-admitting visual tokens with stale representations harms grounding performance, but using current-depth oracle states from a full-reference trajectory significantly improves results.
• CatchUp's depth-aware re-admission strategy, which uses transition signals from related tokens to approximate missing representation evolution, outperforms existing recovery methods.