Microsoft AI diagnosed 83% of hard cases; generalist doctors got 36%

The ER test: On a second set of 325 more typical emergency-department admissions from the MIMIC-IV-ED database, the AI scored 80.3% against 46.8% for physicians, Medical Economics reported. Microsoft described that as 1.7 times more accurate. The physicians recruited were primary care, hospital and acute care doctors, including internists and emergency physicians.

How it works: Instead of handing the AI a finished case write-up, the test made the AI and the doctors work step by step. A separate AI "gatekeeper" released information only when asked, and every test ordered carried an estimated price. The system simulates a panel of virtual doctors that propose diagnoses, challenge each other and weigh each test's cost before ordering it. It ran on OpenAI's GPT-5 in the main results. On average it asked 25.5 questions before its first test; physicians asked 5.6.

The cost fine print: Microsoft says the system reaches the right answer "at half or even one third of the cost." That depends on the setup. With no budget limit, the AI's average testing bill per case was higher than the physicians': $5,431 vs. $4,725 on the complex cases and $4,979 vs. $3,400 on the ER cases, according to the study. Budget-capped versions cut costs to as little as $1,303 per case while still scoring 79.6% on the complex cases. TechCrunch noted that much of the gain comes from stronger underlying AI models, with Microsoft's orchestration adding less on top, and almost none on ER cases with newer models.

In the paper's own words: "These comparisons do not establish superiority over specialists, multidisciplinary teams, or real-world clinical practice." The authors said the benchmark was text-only, with no imaging or pathology slides, excluded healthy and benign visits, and measured only diagnosis, not treatment, safety or patient outcomes. Every author is a Microsoft employee with equity in the company, and eight are named on a related Microsoft patent application, Medical Economics reported.

How outlets framed it: The Verge led with Suleyman's claim that the tool beats doctors at diagnosis and quoted him saying human-level performance could come "in the next couple of years." TechCrunch put the caveats high, noting that cases this complex usually go to specialists, who weren't part of the comparison. Medical Economics, a trade publication for physicians, devoted a full section to the study's limits and the authors' Microsoft ties.

What's still unknown: How the system performs on real patients who forget or leave out details, whether it beats specialists matched to each case, what it would cost hospitals, and whether it will need FDA review before clinical use. Suleyman told The Verge it "needs clinical validation." Microsoft has not said when the Beth Israel study will report results.

Accuracy and cost figures come from the peer-reviewed Science paper as reported by Microsoft, TechCrunch, The Verge and Medical Economics. Every author of the study is a Microsoft employee holding company equity, according to the paper's disclosures, and the system has not been tested on live patients. Photos are file photos and do not show the study.