FinancialAuditBench: Benchmark Construction under Differential Privacy Using Real-World Priors

TL;DR

FinancialAuditBench introduces a privacy-preserving benchmark for evaluating AI agents on financial audit tasks using synthetic data generated from real-world audit statistics and expert knowledge. The framework produces 90 complex tasks across six audit engagements, each with hundreds of files, enabling rigorous evaluation without exposing sensitive data.

Why It Matters

It enables trustworthy evaluation of AI in high-stakes financial domains where data privacy is critical, offering a scalable method to generate realistic, privacy-compliant training and testing data. This supports safer deployment of AI agents in regulated industries like auditing.

Key Findings

• Synthetic audit tasks generated using differentially private statistics maintain fidelity to real-world audit workflows while protecting sensitive data.
• Frontier AI models successfully complete many staff-level audit tasks but occasionally perform incorrect or inappropriate procedures.
• The benchmark framework is generalizable and applicable to other privacy-sensitive domains beyond finance.