🔍 Read the full analysis: What Microsoft’s AI Doctor Study Actually Measured on ThorstenMeyerAI.com
Get movie nights delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A peer-reviewed Microsoft study tested MAI-DxO on historical medical case records, where it scored 83.2% diagnostic accuracy on 304 complex cases. The comparison was against individual generalist physicians, not specialists or clinical teams, and did not involve real patients. A prospective study alongside patient care is planned at Beth Israel Deaconess Medical Center.
Microsoft’s MAI-DxO diagnostic AI scored 83.2% on 304 complex historical medical cases, compared with 36.1% for generalist physicians using textbooks and the internet, according to a peer-reviewed study published in Science on October 22. The results measure performance in a text-based simulator, not care delivered to real patients, and the comparison did not include specialists or physician teams.
The study, titled Sequential diagnosis with language models, evaluated how an AI system gathered information as well as whether it reached the case’s final diagnosis. Each case began with a short summary. The system or physician had to request further information—such as a question’s answer or a test result—from a simulated case file. The benchmark also assigned costs to requested tests, allowing the researchers to assess resource use alongside diagnostic performance.
On a separate set of 325 emergency department cases, Microsoft reported 80.3% accuracy for MAI-DxO and 46.8% for physicians. The supplied account does not specify the same reference-use breakdown for this comparison. Across the complex cases, the physician scores reported in the Science version were 22.3% without reference materials and 36.1% with textbooks and internet access.
The results are Microsoft’s findings for its own system. The paper was peer-reviewed, but that process is not an independent replication of the results. The study authors are listed with Microsoft affiliations, including Microsoft AI chief executive Mustafa Suleyman. The full Science article is behind a paywall, so some methodological details in the source account come from the earlier 2025 preprint and may differ from the final paper.
Infographic 01 · The scoreboard
83% vs. 36%: the number, and who it was measured against.
Microsoft’s diagnostic system beat individual generalist physicians by a wide margin on a simulated, step-by-step test. The comparison group decides how much that says.
Hard cases (NEJM)
Diagnostic accuracy, 304 case records
Physician averages rest on 662 (open book) and 764 (closed book) case attempts, per Microsoft’s chart data.
Who was compared
The fine print behind “beats doctors”
- Generalists only: primary care doctors and hospital generalists. No specialists, no teams.
- Working alone: no colleagues; in the preprint also no search engines or language models.
- Text-only, historical cases: no real patients, no physical exam, no imaging.
- Hard by design: rare teaching cases; no healthy patients, so false alarms were not measured.
- Self-reported: all 15 authors are listed with Microsoft affiliations; peer-reviewed, not independently replicated.
Sources: Science 394, 423–428 (22 Oct 2026); Microsoft AI, “Building Medical Superintelligence” (22 Oct 2026) and its charts; physician cohort details from the arXiv 2506.22405 preprint (2025).
Powered by Thorsten Meyer AI
What the Scores Mean for Care
The study matters because it tests a part of diagnosis that conventional question-and-answer benchmarks can miss: deciding what information to seek next and when to stop. A tool that can ask targeted questions and weigh tests against their cost could, if it works safely in clinical settings, affect how clinicians investigate difficult cases and use medical resources.
But the reported accuracy gap should not be read as evidence that MAI-DxO is ready to replace doctors or improves patient outcomes. The benchmark used historical text records, with no physical examination or real-time imaging. Its physician comparison involved solo generalists, while difficult cases in actual practice may be handled by specialists or teams. The results therefore describe performance under the study’s conditions, not a head-to-head test of the system against typical hospital care.
Cost figures also need care. The study assigned estimated prices to tests using billing codes and a 2023 U.S. hospital price list, plus a fixed $300 physician-visit cost, according to the source account. The authors described these as first-order approximations. They are not evidence that hospitals would achieve the same savings in practice, where prices, workflows and patient needs vary.
As an affiliate, we earn on qualifying purchases.
How the Diagnostic Benchmark Worked
Microsoft’s Sequential Diagnosis Benchmark, or SDBench, is designed to resemble a staged diagnostic process. Rather than giving participants every fact at once, it starts with a short presentation. A “gatekeeper” model holds the rest of the case information, which participants request through follow-up questions and tests. The benchmark then scores the resulting diagnosis and the cost of the information-gathering process.
MAI-DxO organizes a language model as a panel of virtual physicians. In successive rounds, the agents propose diagnoses, choose a next question or test, and challenge one another. One detail reported from the earlier preprint is that a separate model graded answers against the published final diagnosis on a five-point scale. Physicians re-graded a sample, with good but imperfect agreement, according to the source account. When a requested test result was absent from the record, the gatekeeper generated a synthetic result.
The comparison group was limited. The preprint described 21 U.S. and UK generalists—17 in primary care and four hospital generalists—with a median of 12 years’ experience. They worked alone, without search engines or language models in one arm. The Science version also reported an open-book physician arm. According to the source account, doctors were scored on the 56 most recent held-out cases, while the AI was evaluated on all 304 complex cases, a difference relevant when interpreting the reported scores.
“1.7 times as accurate as physicians on emergency department cases”
— Microsoft, in its blog post, “Building Medical Superintelligence”
As an affiliate, we earn on qualifying purchases.
Limits the Benchmark Leaves Open
The study does not show how MAI-DxO performs when interacting with patients, examining them, reviewing live imaging or working within a hospital’s normal clinical workflow. It also does not establish whether using the system changes treatment decisions, patient outcomes, diagnostic delays or total costs. Those questions require testing beyond historical case records.
The cases were selected for diagnostic difficulty and, according to the source account, did not include healthy patients or harmless explanations. The preprint’s authors noted that this meant false alarms could not be measured. A benchmark built around hard cases cannot by itself show how often a system would over-test or wrongly flag disease across a broader patient population.
It is also not clear from the available account whether all details of the final Science methods match the 2025 preprint. The source says some benchmark specifics are drawn from that earlier, non-peer-reviewed version because the final article is paywalled. Independent reproduction of Microsoft’s reported scores has not been established in the supplied material.
As an affiliate, we earn on qualifying purchases.
Testing MAI-DxO With Patients
Microsoft says a prospective study is planned at Beth Israel Deaconess Medical Center in Boston. In that study, consenting patients are expected to interact with the system alongside their usual care. That would move testing beyond retrospective case records, although the supplied information does not give a start date, enrollment target, study protocol or outcome measures.
Results from such a study would help show whether the system can operate in a clinical setting and how its recommendations compare with normal care. Until those results are available—and independently evaluated—the Science paper is evidence of performance on specified benchmarks, not proof of clinical benefit or readiness for routine use.
As an affiliate, we earn on qualifying purchases.
Key Questions
What did Microsoft’s MAI-DxO study measure?
It measured diagnostic accuracy on historical case records and tracked the questions and tests requested during a simulated diagnostic process. It also estimated test costs using assigned prices; it did not measure outcomes in patients receiving care.
How did MAI-DxO score on the complex cases?
Microsoft reported 83.2% accuracy across 304 complex cases. Generalist physicians scored 36.1% with textbooks and internet access and 22.3% without them, according to the Science version figures in the source account.
Was the AI compared with medical specialists?
No specialists or physician teams were included in the comparison described by the source. The physician group consisted of generalists working individually, while the AI system used multiple virtual agents.
Does the study show that MAI-DxO improves patient care?
No. The study used historical text cases in a simulator and did not test patient outcomes, clinical safety or routine hospital use. A prospective study alongside normal care is planned at Beth Israel Deaconess Medical Center.
Have outside researchers reproduced the results?
The supplied source material does not report an independent reproduction. Peer review of the Science paper does not itself establish that outside teams have replicated Microsoft’s findings.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
