Kierston & KierstonBench: Simulation of Allentown, PA and Evaluation of Language Models in Crime Investigation and Surveillance Tasks

TL;DR

The paper introduces Kierston, a city-scale simulation of Allentown, PA with 125,000 residents, and KierstonBench, a benchmark for evaluating language models in crime investigation and surveillance tasks. The benchmark tests models on record linkage, routine reconstruction, and deception detection using simulated data derived from behavioral timelines across multiple data sources.

Why It Matters

This work addresses a critical gap in evaluating AI for law enforcement and surveillance by providing a controlled, reproducible environment with ground truth. It enables fair comparison of models' reasoning, evidence handling, and susceptibility to deception—essential for responsible deployment in high-stakes domains.

Key Findings

• Most language models failed to perform reliable records investigation and made unsupported identifications.
• Frontier models solved most tasks and abstained when evidence was insufficient, showing improved reliability.
• No model fully replaced human investigators, and even top models were misled by common deceptions and assumptions.