Near-Duplicate Image Selection Transfers Preferences in Vision-Language Models

TL;DR

Selecting near-duplicate images based on caption likelihood transfers implicit preferences to vision-language models, even when source images and captions remain unchanged. This transfer occurs without explicit target prompts and can propagate across model generations, with measurable shifts in model behavior toward the selected target.

Why It Matters

This reveals a hidden mechanism for preference leakage during data curation, which could unintentionally bias downstream models. It highlights risks in automated data filtering pipelines and underscores the need for transparency in training data selection processes.

Key Findings

• Curating near-duplicate images by caption likelihood induces a directional shift in student models' preferences toward the selected target, even with fixed source images and captions.
• The preference transfer is detectable via linear probes and persists across model generations, demonstrating intergenerational propagation of implicit biases.