Molecular Déjà Vu: Digit-Level Retrieval of Published Values in Frontier Language Models

TL;DR

This paper audits 22 large language models on molecular property benchmarks and finds widespread verbatim retrieval of published values. The study reveals that retrieval is benchmark-specific and changes with reasoning levels. It also tests a method to interrupt retrieval, showing that predictive capability is not solely determined by memorized values.

Why It Matters

The findings have practical significance as they highlight the limitations of current evaluation methods for large language models. This research can inform the development of more robust benchmarks and models that truly predict molecular properties rather than simply retrieving memorized values. Accurate prediction of molecular properties has significant implications for fields like drug discovery and materials science.

Key Findings

• widespread verbatim retrieval of published values
• retrieval is benchmark-specific
• interrupting retrieval changes predictive capability