What Is the Epic Sepsis Model? Accuracy and Validation Results

What is the Epic Sepsis Model, and how accurate was it?

Epic's sepsis model failed because hundreds of hospitals ran it at scale on a vendor-reported accuracy number nobody outside Epic could check. When researchers finally got the data, the model performed far worse than advertised: an AUC of 0.63 against Epic's claimed 0.76 to 0.83, missing 67% of actual sepsis cases while firing alerts on 18% of all hospitalized patients. The model is built into the Epic electronic health record used by a majority of US hospitals. It's now the standard case study for what happens when a health system trusts a proprietary algorithm instead of validating it.

How was the Epic Sepsis Model built and sold?

Epic released the Epic Sepsis Model (ESM) in 2017 as a proprietary penalized logistic regression tool, trained on roughly 405,000 patient encounters pulled from three health systems between 2013 and 2015. Epic reported internal accuracy figures in the AUC range of 0.76 to 0.83. That number circulated in conference materials and sales conversations for years before any outside researcher could test it independently. It's the same dynamic you see whenever AI vendors publish their own benchmark results and call it proof.

Epic's EHR already ran in roughly 56% of US hospitals and health systems, so the ESM reached an enormous footprint almost by default. Hospitals on Epic's platform could switch the sepsis tool on without building or validating anything themselves. For years, that convenience passed for a feature. It was also the mechanism that let a flawed model spread largely unchecked.

What did the validation study that exposed the gap find?

The reckoning came from a team at Michigan Medicine, led by Andrew Wong, published in JAMA Internal Medicine in 2021. The researchers tested the ESM against 38,455 hospitalizations and 2,552 confirmed sepsis cases, then compared its real-time alerts against actual clinical outcomes.

The results:

• Hospitalization-level AUC of 0.63, barely better than chance and well below Epic's advertised range.
• 67% of sepsis cases missed despite the model flagging 18% of all hospitalized patients with a score of 6 or higher.
• Positive predictive value of just 12%, meaning the vast majority of alerts were false alarms.
• 109 patients evaluated per true positive when clinicians acted on repeated alerts within a 4-hour window.

An accompanying JAMA Internal Medicine editorial didn't soften the finding. It noted the model's sensitivity of 33% and specificity of 83%, and warned that the "black box" nature of proprietary machine learning tools demands special caution from health systems that rely on them.

Why did the antibiotics-as-predictor flaw make Epic's sepsis model fail?

The root technical cause came out later, through reporting by STAT News. The ESM used whether a clinician had already ordered antibiotics as one of its predictive inputs, a detail Epic hadn't disclosed in its public marketing materials. That created circular logic: if a clinician already suspected sepsis and started antibiotics, the model was more likely to fire. The alert didn't warn clinicians early. It mostly confirmed what they already knew, after the fact.

That single design choice explains most of the gap between Epic's internal numbers and the Michigan findings. A model partly trained on the record of a clinician's own suspicion will look accurate in a dataset built the same way, and look much worse once it's asked to catch sepsis before anyone has acted on it.

Why did nobody catch the black-box problem sooner?

Epic treated the ESM's internal workings as a trade secret, so no independent peer review happened before the model reached patients at hundreds of hospitals. Michael Pencina, a bioinformatics professor at Duke, put it to STAT: "I would say Epic and other developers have used the environment to their advantage. Our naivete has enabled this kind of behavior."

Financial incentives compounded the opacity. Epic's "honor roll" program paid hospitals, in some cases close to $1 million, for adopting technical upgrades that included its predictive algorithms. That doesn't prove the model was bad. It does mean wide adoption was never evidence of accuracy. It was evidence of a sales program working as designed.

Performance also wasn't consistent across sites, which made any single vendor claim even less generalizable. A separate 2019 study across five UCHealth hospitals in Colorado, co-authored with an Epic employee, found an AUC of 0.73: better than Michigan's 0.63, still short of Epic's headline range, and reached under a different sepsis definition and threshold. A model that swings from 0.73 to 0.63 depending on which hospital runs it isn't a model you validate once and trust everywhere.

How did Epic respond, and what changed in the 2022 overhaul?

Epic's first instinct was to push back. Executives drafted a statement contesting the Michigan findings and later wrote that "tens of thousands of clinicians have access to the sepsis model and transparency into how it works," rejecting the suggestion that Epic hadn't been forthcoming.

The pressure didn't let up. STAT's follow-up reporting kept the antibiotics-order flaw and the honor-roll payments in public view, and by 2022 Epic changed course. Internal documents obtained by STAT showed Epic now recommending that hospitals retrain the model on their own local data before using it clinically, update its definition of sepsis onset to a more standard one, and reduce its reliance on antibiotic orders as a predictive signal. After years of marketing a plug-and-play model, Epic had effectively walked that claim back.

Why is validation the hospital's job, not the vendor's?

The generalizable lesson for anyone deploying clinical AI isn't subtle: a vendor's reported accuracy number is a claim, not a finding, until someone outside the vendor checks it against local data. The University of Melbourne's Centre for Digital Transformation of Health filed the ESM under a blunt label: "Deployed worldwide. Validated nowhere." Their analysis points to the fix that got skipped. A prospective silent trial, run before the model touches a single patient's care, would likely have caught the antibiotics-order problem and the inflated accuracy claims before they reached production. It's the same discipline teams now build into testing AI-generated code before it reaches production: catch the failure mode mechanically, before a human has to notice it in the wild.

That's the same instinct behind owning more of your mission-critical software stack rather than renting a vendor's claims on faith, whether that's a sepsis score or an API endpoint you can't inspect. Hospitals that want to keep a sepsis model honest over time also need their own monitoring layer sitting on top of it.

What has changed since: regulation, FDA clearances, and competing models?

The sepsis-AI landscape has moved since 2017, partly because of the ESM saga. In 2024, Prenosis's Sepsis ImmunoScore became the first FDA-authorized AI diagnostic tool for sepsis, cleared through the agency's De Novo pathway for a device type the FDA had never evaluated before. That's a meaningfully different bar than the one Epic cleared in 2017, when no regulatory review of the ESM's clinical claims existed at all.

Regulatory clearance alone doesn't solve the problem. But it does give hospitals evaluating sepsis AI today an accountability structure to point to, something that didn't exist when the ESM shipped. The same gap shows up with any autonomous system a hospital hands real decisions to: without a registry and an owner for every model making calls, nobody can trace back why an alert fired or who signed off on the threshold.

What should hospitals check before buying clinical AI?

The Epic case gives healthcare organizations a concrete checklist for any clinical AI purchase, not just sepsis tools:

1. Demand the validation data, not the summary slide. Ask for the AUC, sensitivity, specificity, and positive predictive value on a population that resembles your own, not just the vendor's development cohort.
2. Run a silent trial before go-live. Let the model score real patients without surfacing alerts to clinicians, then compare its calls against actual outcomes before it touches care decisions.
3. Retrain on local data. Epic itself now recommends this for the ESM. A model trained on three health systems in 2013 to 2015 will not behave the same way on your patients in 2026.
4. Treat vendor AUC claims as a starting hypothesis, not a fact. Performance varied from 0.83 claimed, to 0.73 at one regional system, to 0.63 at another. That spread is the data point, not the outlier.
5. Own the ongoing monitoring. Accuracy can drift as clinical workflows and documentation habits change. Someone inside the hospital, not the vendor, needs to be watching for that drift continuously.

None of this requires rejecting vendor AI outright. It requires treating a model's claimed accuracy the way a hospital treats any other clinical claim: tested against evidence, documented, and revisited, not purchased on trust and left running in the background of a life-or-death decision.