WildFakeToolBench: Benchmarking AI-Generated Image Detectors with Web-Sourced Image Processing

TL;DR

WildFakeToolBench introduces a benchmark of 156 executable image-processing tools sourced from public web domains to evaluate AI-generated image detectors under realistic evasion conditions. The tools, validated for correctness and diversity, demonstrate significant evasion success rates against state-of-the-art detectors, revealing vulnerabilities in current detection systems.

Why It Matters

The study reveals how easily malicious actors can leverage publicly available code and LLM-assisted tools to bypass image detectors without specialized expertise or hardware. This highlights the urgent need for robust evaluation frameworks that incorporate real-world adversarial techniques to improve detector reliability.

Key Findings

• The benchmark includes 156 executable tools derived from 220 public sources across 142 domains, with each tool validated for syntax, execution, and output integrity.
• Individual tools achieve evasion success rates between 25.25% and 100% across six detectors, outperforming traditional image-processing baselines.
• High coverage across a small tool subset (e.g., eight tools) exceeds 80% for all detectors, but effectiveness is highly detector-specific and not reliably predictable from coverage alone.