OmniKVQuant: KV Cache Quantization for Omni-LLMs

TL;DR

OmniKVQuant is a training-free framework that enables 2-bit KV cache quantization for Omni-modal large language models, preserving performance across audio-visual benchmarks. It addresses temporal key drift and heterogeneous value geometry issues. This allows for substantial memory cost reduction.

Why It Matters

The proposed method reduces memory costs for Omni-LLMs, making them more feasible for practical applications. This is significant because Omni-LLMs have growing memory requirements due to their multimodal input. The approach enables more efficient deployment of these models.

Key Findings

• temporal key drift
• heterogeneous value geometry
• 2-bit KV cache quantization