LGMT: A Tool for Testing the Logical Reasoning Reliability of LLMs

TL;DR

LGMT is an open-source tool that tests logical reasoning consistency in LLMs by generating logically equivalent reformulations of input queries and checking for prediction agreement. It uses first-order logic-based metamorphic relations to detect contradictions without requiring ground-truth labels, revealing inconsistency rates of 20.84% to 29.44% across six LLMs.

Why It Matters

Logical consistency is critical for LLMs in safety-critical applications like automated reasoning and decision support. LGMT enables developers to identify and mitigate reasoning flaws that static benchmarks miss, improving trustworthiness in real-world deployments.

Key Findings

• LGMT detects reasoning inconsistencies in LLMs by leveraging metamorphic relations grounded in first-order logic.
• Inconsistency rates across six LLMs range from 20.84% to 29.44% on a benchmark of 3,091 metamorphic test groups.
• The tool supports custom metamorphic relations verified via the Z3 theorem prover and operates offline with a web studio interface.