Purifying Backdoored Large Vision-Language Models by Removing Hijacked Directions

TL;DR

OrthoPurify is a one-step method that removes backdoor vulnerabilities from large vision-language models by identifying and eliminating hijacked weight update directions. It uses a pseudo-benign model derived from minimal clean data fine-tuning to approximate a clean reference, enabling efficient purification without retraining or inference overhead.

Why It Matters

Backdoor attacks threaten the reliability of safety-critical AI systems, and current defenses are either computationally expensive or slow at inference. OrthoPurify offers a fast, scalable solution that restores model integrity with minimal resource use, making it practical for real-world deployment.

Key Findings

• Backdoor attacks manipulate a small number of weight update directions, diverting them from task learning to shortcut encoding.
• A pseudo-benign model created via minimal fine-tuning on clean data effectively approximates a true benign reference.
• OrthoPurify achieves near-zero attack success rate while maintaining original model performance across benchmarks using a single orthogonal projection.