MiD-VLM: Detecting and Grounding Global Semantic Misinformation in Image-Caption Pairs

TL;DR

This paper introduces MiD-VLM, a multimodal model that detects global semantic misinformation in image-caption pairs by combining SigLIP and DINOv2 features with LoRA-adapted Qwen2-VL-7B. It achieves high F1 (0.9679) in authenticity classification and enables token-level grounding of misleading captions, addressing limitations of pixel-level detection methods.

Why It Matters

Semantic misinformation in image-caption pairs can mislead users in critical applications like news and education. The ability to detect and localize such distortions without relying on visual artifacts is crucial for trustworthy AI systems.

Key Findings

• MiD-VLM achieves state-of-the-art F1 score of 0.9679 on DGM4+ for detecting semantic inconsistencies.
• The fusion of SigLIP, DINOv2, and LoRA-adapted Qwen2-VL enables effective joint prediction of authenticity, manipulation type, and token-level grounding.
• Parameter-efficient adaptation proves effective for multimodal misinformation detection, enabling precise localization of misleading caption content.