Reversible Pseudonymisation of Documents Sent to External LLMs: an Open Bilingual Benchmark and an Engine

TL;DR

This paper introduces an open bilingual benchmark and a new reversible pseudonymisation engine for protecting personal data in documents sent to external LLMs. The engine, built from the two strongest existing tools, significantly outperforms them on synthetic datasets in both Russian and UK English, achieving near-perfect placeholder masking and restoration while meeting all detection thresholds.

Why It Matters

Reversible pseudonymisation is critical for privacy-preserving use of external LLMs in professional settings. This work provides a rigorous, transparent evaluation framework and a high-performing engine, advancing practical privacy safeguards in real-world AI workflows.

Key Findings

• No existing tool meets all predefined privacy thresholds on external datasets, with high rates of personal data leakage (up to 61% in Russian).
• The proposed engine reduces visible data leakage to 0.3% (Russian) and 0.2% (UK) on held-out synthetic documents, meeting all detection thresholds.
• Placeholder restoration is high in UK English (≥99%) but slightly below threshold in Russian (94.6%) due to issues with surname inflection from a dependency.