When Text Misleads: Inconsistent-Aware Reasoning for Audio-Grounded Dialogue

TL;DR

The paper introduces ContraTalk, a benchmark that exposes models' reliance on misleading text by contrasting transcript-based interpretations with acoustic cues in spoken dialogue. It proposes an Audio Twin framework to make speech features accessible to reasoning models, improving performance on conflicting cases. Experiments reveal that strong LLMs and AudioLLMs often fail in conflict scenarios due to text bias, even when acoustic evidence contradicts the transcript.

Why It Matters

Many current models in spoken dialogue understanding rely on text shortcuts rather than genuine audio grounding, leading to unreliable real-world performance. This work provides a diagnostic tool and a method to improve robustness by making acoustic evidence explicit during reasoning, which is critical for applications like virtual assistants and accessibility tools.

Key Findings

• Text-only LLMs achieve over 90% accuracy on consistent cases but drop to 33-48% on conflict cases, indicating strong reliance on transcripts.
• Direct AudioLLMs show partial grounding, still selecting transcript-biased answers in 30-40% of conflict cases.
• The Audio Twin framework improves conflict-case accuracy and reduces trap selection, though consistent-case performance depends on the underlying model.