In June 2021, a research team led by Andrew Wong at the University of Michigan published a study in JAMA Internal Medicine that quietly became one of the most consequential papers in clinical AI. The subject was the Epic Sepsis Model (ESM), a proprietary early-warning algorithm embedded in the Epic electronic health record and running, by the vendor's own account, at several hundred U.S. hospitals. The finding: when tested on 27,697 real hospitalizations at Michigan Medicine between December 2018 and October 2019, the model's area under the receiver-operating-characteristic curve (AUC) was 0.63. Epic had told hospitals to expect 0.76 to 0.83.
The practical translation of that gap was blunt. At a threshold Epic recommended for clinical use, the model's sensitivity was 33 percent — meaning it missed 67 percent of sepsis cases outright — while its positive predictive value was just 12 percent, so the vast majority of its alerts fired on patients who did not have sepsis. Clinicians at sites using the tool described exactly what you'd predict from those numbers: real cases slipping through, and alert fatigue from a firehose of false positives eroding trust in the tool over time.
Why did the numbers diverge so sharply? Part of the answer is architectural. The ESM was a proprietary, closed model: hospitals could see the alerts it generated but not the full feature weighting or training methodology behind them, which made it difficult for any single site to know whether the model's assumptions matched their patient population, their documentation habits, or their definition of sepsis onset. A related, and arguably bigger, issue was validation practice itself. Epic's headline performance numbers were derived largely from the same kinds of retrospective, in-sample data the model was built on — a setup prone to overstating real-world accuracy — rather than from prospective, independent testing at hospitals with no hand in building the tool. Subsequent multi-site work, including a 2023 analysis of the ESM across nine networked hospitals, found substantial variability in performance from one site to the next, reinforcing that a single reported AUC can mask very different real-world behavior depending on local data and workflows.
To Epic's credit, the response was not silence. Following the Michigan study and the scrutiny that followed, Epic overhauled the model, as reported by STAT News in October 2022. The revised model (ESM v2) changed the definition used for sepsis onset to align with more widely accepted clinical criteria, reduced reliance on clinician antibiotic orders as a signal (a feature that had let the model partly grade its own homework), and began recommending that hospitals fine-tune the model on their own local data before clinical use. Epic reported that live sites saw more timely alerts and fewer false positives after the update, and later prospective multicenter validation work has examined the revised model's performance directly.
It's also worth noting the ESM is not proof that all sepsis-prediction AI is unreliable. Systems built and validated more transparently — such as Johns Hopkins' TREWS, developed with published methodology and evaluated in peer-reviewed, multi-site prospective studies in Nature Medicine and elsewhere — have reported strong performance and clinical benefit under scrutiny that the original ESM never underwent before widespread deployment. The differentiator was not the underlying technology so much as the discipline: publish the method, test prospectively, validate outside the walls that built the model.
The story did not end with the 2022 overhaul. In a 2026 JAMA Network Open study, the same University of Michigan team that first exposed the model's shortfall — again led by Andrew Wong — prospectively validated Epic Sepsis Model v2 across 227,091 inpatient encounters at four major U.S. health systems. The headline AUC climbed to a 0.82-0.92 range, a genuine improvement, but the researchers still found substantial variability from one health system to the next, low positive predictive value, and a heavy alert burden — and concluded that hospitals still need to run their own local validation rather than trust a vendor-reported number. A separate 2025 emergency-department study comparing v1 and v2 head-to-head found the update meaningfully cut false alarms, with sepsis-encounter alert rates falling from 26.1% to 6.8%. The lesson from 2021 evidently stuck: even after a redesign, external, site-specific validation — not vendor marketing — remains the only way to know what a clinical AI model actually does at your hospital.
The lesson for clinical AI generally is not "don't use proprietary models" so much as "don't let vendor-reported metrics substitute for independent, site-specific, prospective validation." An AUC computed on a vendor's own retrospective data, using a vendor's own definitions, tells a hospital very little about how a tool will behave on their patients, their charting patterns, and their alert fatigue thresholds. As more health systems adopt AI-driven clinical decision support, external validation and methodological transparency are shifting from academic nicety to basic due diligence — the difference between a tool that quietly stops being trusted by exhausted clinicians and one that actually catches the patient who needs help.