Microsoft’s AI diagnostic system, MAI-DxO, correctly diagnosed 83% of complex New England Journal of Medicine cases in a peer-reviewed study in Science, against 36% for generalist physicians who could use reference tools.
The paper, “Sequential diagnosis with language models”, was published on October 22. Instead of handing an AI a complete case write-up, it makes the system start from a short presentation and decide what to ask and which tests to order, as a doctor would.
How it works
The team built a benchmark called SDBench from 304 NEJM clinicopathological cases and 325 emergency department admissions from the MIMIC-IV-ED dataset. A “Gatekeeper” agent releases only what a question or test would realistically reveal, and a “Judge” agent scores the final diagnosis.
MAI-DxO, the Microsoft AI Diagnostic Orchestrator, works like a panel of virtual physicians: each proposes diagnoses, picks questions or tests and challenges the emerging view, while the system weighs each test’s cost.
The numbers
On the NEJM cases, the system running on GPT-5 scored 83.2% at an average estimated test cost of $5,431. Physicians averaged 36.1% with search and reference tools and 22.3% without. Even the best individual physician, at 51.8%, was beaten on both accuracy and cost.
On more routine emergency cases, MAI-DxO reached 80.3% versus 46.8% for physicians. Microsoft says the gains held, and grew, when it re-ran the evaluation on GPT-5.5 without changing the orchestration.
The caveats
The authors are upfront about the limits. The tests were text-only, compared the AI with individual generalists rather than specialists or teams, and measured “diagnostic reasoning under idealized information conditions rather than bedside performance or patient outcomes.” All of the authors are Microsoft employees.
That’s why the next step matters. Microsoft says it will run a prospective study with Beth Israel Deaconess Medical Center in Boston, where patients interact with the system directly. It’s also open-sourcing SDBench code so others can test other models and challenge the findings.
Why it matters
GPs are already turning to ChatGPT and other tools for diagnoses, and tools like the NHS’s Mia cancer tool have shown AI can catch what doctors miss. A structured, cost-aware panel like MAI-DxO points to a more rigorous version of that, but until it’s proven with real patients, it’s a benchmark result, not a clinical one.
