Can Discrete Tokens Serve as Semantically Lossless Representations of Visual Signals?

TL;DR

The paper challenges the assumption that quantization error inherently limits semantic fidelity in discrete visual tokens, showing that effective bit budget and representation design matter more than error magnitude. It introduces Factorized Vector Quantization (FVQ), which uses symmetric parallel codebooks to avoid saturation and match continuous representations in multimodal understanding. FVQ achieves state-of-the-art results among discrete models and rivals strong continuous models like Qwen3-VL.

Why It Matters

This work demonstrates that discrete visual tokens can be semantically lossless with proper design, enabling efficient, scalable multimodal foundation models with native language-like interfaces. It opens the door to more efficient training and inference in large-scale multimodal systems, reducing reliance on continuous representations.

Key Findings

• Quantization error is not a reliable proxy for semantic loss; effective bit budget is more predictive of performance.
• Factorized Vector Quantization avoids performance saturation by using complementary, fixed-dimensional views with symmetric parallel codebooks.
• FVQ matches continuous representations in multimodal understanding, outperforming prior discrete models and rivaling Qwen3-VL on fine-grained OCR.