Microsoft's MAI-DxO beats doctors on hard cases in Science

The diagnosis numbers are striking. The fine print matters more.

Microsoft AI says its diagnostic system, MAI-DxO, solved 83% of 304 complex medical cases, against 22% for physicians working without references and 36% with them. The work was published in Science on Oct. 22, according to Microsoft.

A panel of virtual doctors, priced per test

MAI-DxO is not one model answering at once. Microsoft describes it as a panel of virtual physicians that propose diagnoses, pick questions and tests, and argue over several rounds before settling. It weighs what each test is likely to reveal against what it costs.

The 304 complex cases come from the New England Journal of Medicine's case records. On typical emergency department cases from the MIMIC-IV-ED dataset, Microsoft says MAI-DxO scored 80% against 47% for physicians.

Microsoft also says it gets there "at half or even one third of the cost," with fewer tests, and that one cost-tuned setup reached 81% on the NEJM cases "at less than half the cost of comparative systems."

Those are Microsoft's numbers on benchmark cases, not patients.

The model is doing most of the work

Read TechCrunch's account of the paper before you forward the headline.

The doctors in the comparison were primary care physicians and general internists. Specialists and physician teams, who usually handle cases this hard, were not in it, TechCrunch reports.

Microsoft ran the orchestrator on models from OpenAI, Google, Anthropic, Meta and DeepSeek. The study's own data suggests most of the gain comes from the underlying model, with the orchestrator adding some on top, according to TechCrunch. It helps most on complex cases and weaker models, and added little on the emergency cases.

So the panel helps, and the model under it matters more. For builders, that is the useful finding.

What builders can take from it

If you build anything that has to make a high-stakes call, the design is the part to copy. Several agents that argue, a rule that every extra step has a price, and a final answer only after the debate. That is a multi-agent system with a budget built in.

You can test the claim yourself, partly. Microsoft says it is open-sourcing SDBench, its benchmark, for select NEJM cases, including the judge and cost-estimation code. The SDBench page ships five synthetic sample cases and says the real NEJM case set must be licensed from NEJM. Microsoft describes MAI-DxO itself as research and has not announced it as a product.

Treat the 83% the way you would treat any AI benchmark a vendor built and scored: as a claim worth checking against your own evals, not a fact about your patients or your users.

What to watch

Microsoft says a prospective study with Beth Israel Deaconess Medical Center will show "where the technology succeeds, where it falls short." Patients there get standard care, and those who consent also use the system, with its recommendations compared to their doctors', TechCrunch reports.

That study is the one that counts. Benchmarks rank systems. Wards test them.