Evaluating Large Language Models on Misconceptions in Multi-Turn Medical Conversations

TL;DR

The study introduces ThReadMed-QA, a dataset of 2,437 multi-turn medical conversations derived from real patient interactions, to assess how well large language models detect and correct medical misconceptions over time. Evaluations show that even top models like GPT-5 and Claude-Haiku see a sharp decline in misconception correction accuracy from ~85% initially to ~50% after two follow-ups. This degradation is largely due to error propagation, indicating that current models struggle with maintaining consistent, safe medical guidance across conversation threads.

Why It Matters

Medical conversations often involve evolving misconceptions that require persistent correction. The findings reveal a critical safety gap in current LLMs, as their performance deteriorates over time in multi-turn settings, risking inaccurate or harmful advice in real-world patient interactions. This underscores the urgent need for evaluation frameworks that test models in dynamic, longitudinal dialogue contexts.

Key Findings

• Frontier LLMs correct misconceptions at ~85% accuracy on initial turns but drop to ~50% after two follow-up turns.
• Error propagation from prior incorrect model responses significantly contributes to performance degradation.
• Even when provided with correct prior context, models still fail to consistently correct misconceptions, indicating inherent limitations in multi-turn reasoning.