Studying Image Tokenizers as Visual Languages in Unified Multimodal Models

TL;DR

The paper introduces a controlled autoregressive testbed to evaluate image tokenizers within unified multimodal models by tracking task-specific losses during continual pretraining. It reveals that loss behaviors vary significantly across tasks and tokenizer designs, with I2T loss providing a more stable predictor of downstream performance than T2I loss. The study demonstrates that joint modeling of visual and textual tokens is sensitive to tokenizer choice and that reconstruction quality alone does not determine task performance.

Why It Matters

Understanding how image tokenizers function as visual languages within multimodal systems enables better design of unified models. This work provides a practical framework for evaluating and comparing tokenizers beyond isolated metrics, leading to improved model performance in both generation and understanding tasks.

Key Findings

• Task-specific validation losses exhibit distinct scaling patterns and rank tokenizers differently, necessitating task-by-task analysis.
• I2T loss over a shared text vocabulary consistently correlates with downstream performance across tokenizers, unlike T2I loss which varies with image token space.
• Image tokenizer choice influences text modeling during joint optimization, and higher reconstruction quality does not guarantee better task performance.