What it is
The authors explored how frontier language models perform at sequential diagnosis, a paradigm that captures the iterative nature of clinical reasoning: given a brief clinical presentation, the diagnostician selectively acquires information to reach a diagnosis, which lets both diagnostic accuracy and the cost of the workup be assessed. They developed an interactive benchmark and an AI system that simulates a panel of physicians, continually revising the differential diagnosis and selecting cost-effective tests. On this benchmark the system reached 83.2% accuracy on New England Journal of Medicine clinicopathological conference cases and 80.3% on MIMIC-IV-ED emergency department admissions, outperforming generalist physicians, who reached 36.1% and 46.8% respectively. The authors note that these findings were obtained in a controlled benchmark setting, and say they highlight the promise of language models for supporting more accurate, lower-cost diagnoses.
Why it matters
On NEJM case-conference cases, a language-model system simulating a physician panel diagnosed 83.2% correctly in a benchmark, against 36.1% for generalist physicians. In sequential diagnosis the diagnostician starts from a brief presentation and acquires information step by step, so a benchmark can score both diagnostic accuracy and the cost of the workup. On MIMIC-IV-ED emergency department admissions the system reached 80.3% against 46.8% for generalist physicians, a smaller gap than on the case conferences, and it is built to keep revising its differential diagnosis while choosing cost-effective tests.
Underlined numbers link to their source. Every metric and quoted figure is listed under Sources and data below.
Filed underclinical AI, diagnostic reasoning, language models, benchmarks, healthcare
