CodeShrink: Adaptive Visual Compression for Efficient Multimodal Code Understanding

TL;DR

CodeShrink introduces a dynamic approach to compressing code visualizations for multimodal models by eliminating redundant whitespace, adapting compression settings per input via a reinforcement learning agent, and removing irrelevant visual tokens during inference. It achieves up to 71.2% reduction in visual tokens while maintaining or improving performance over baseline methods. The framework enhances efficiency without sacrificing task accuracy across multiple code understanding tasks.

Why It Matters

Efficient multimodal code processing is critical for scaling large models in software engineering applications. By reducing visual token overhead without compromising performance, CodeShrink enables faster inference and lower computational costs, making multimodal code analysis more practical for real-world deployment.

Key Findings

• Blank-Free Rendering eliminates layout-induced visual tokens by replacing traditional indentation and line breaks with compact representations and structural markers.
• Adaptive Compression Configuration uses a lightweight reinforcement learning agent to select optimal compression settings per input, improving token efficiency across diverse tasks.
• Dominant Token Selection prunes visually redundant content during inference by analyzing both the instruction and code image, further reducing token usage while preserving task-relevant information.