RegExtract: A Benchmark for PFAS Regulatory Information Extraction with LLMs

TL;DR

RegExtract is a benchmark for extracting structured regulatory data from PFAS-related documents, featuring 488 annotated field instances across 66 US state and federal regulations. It includes provenance tracking and a typed schema for key regulatory elements, enabling evaluation of LLMs with traceable, accurate extraction.

Why It Matters

Regulatory compliance for hazardous substances like PFAS is complex and time-consuming. This benchmark enables development of AI tools that automate accurate information extraction, improving efficiency for manufacturers, regulators, and environmental safety assessments.

Key Findings

• Document identification fields are particularly difficult for LLMs to extract accurately.
• Using field-specific few-shot examples improves extraction performance across different LLMs.
• Provenance-aware evaluation metrics effectively measure traceability and reliability of extracted facts.