In 2019, a team led by Ziad Obermeyer published a finding in Science that became one of the most consequential case studies in algorithmic fairness: a commercial care-management algorithm used across U.S. health systems, and estimated to affect risk predictions for roughly 200 million Americans annually, was systematically assigning Black patients lower risk scores than White patients with the same underlying burden of illness.
The algorithm didn't ask for anyone's race. It didn't need to. That is the part of the story healthcare AI teams still have to sit with.
What the algorithm actually predicted
The tool was designed to identify patients who would benefit from extra care-management resources — nurse outreach, care coordination, closer monitoring — by predicting a risk score. But the researchers found the model was not trained to predict future illness. It was trained to predict future healthcare costs, used as a proxy for health need, because cost is easy to measure at scale and illness is not.
That substitution seemed reasonable on its face — sicker patients generally cost more to treat. But the relationship breaks down when access to care is unequal. Because of a combination of factors including lower income, transportation and time barriers, and, as other research on clinical trust and treatment patterns has documented, less consistent engagement with the health system, Black patients in the dataset had historically had less money spent on their care than White patients who were equally sick. Cost, in other words, was measuring access as much as it was measuring need — and access was unevenly distributed.
The algorithm learned that pattern faithfully. It wasn't wrong about the data; the data was an accurate record of a biased healthcare system. The model just inherited the bias baked into its label.
The size of the effect
At any given risk score, Black patients in the study population were sicker — measured by objective indicators like chronic conditions and lab values — than White patients receiving the same score. When the researchers modeled what would happen if the algorithm predicted health need directly instead of cost, the share of Black patients flagged for extra care at the same eligibility threshold rose from 17.7% to 46.5%. Put differently: at the original threshold, more than half of Black patients who needed and would have qualified for extra care under an unbiased measure were being screened out.
Why this wasn't a data problem or a bad-actor problem
It is tempting to file this under "biased training data" and move on, but that framing undersells the lesson. The individual data points — costs, diagnoses, lab results — were accurate. Race was never an input. The developers, by the researchers' account, had not intended discrimination. The failure was a modeling choice made under reasonable-seeming constraints: cost is a cheap, abundant, well-documented label; illness is not. That single label-choice decision was enough to launder a structural inequity into an operational one, at the scale of a national algorithm.
This is the core mechanism healthcare AI teams need internalized: a proxy variable can be statistically excellent and still be the wrong thing to predict, if the process generating that proxy is itself unevenly applied across the population.
What happened next
Working with the researchers, the algorithm's developer reformulated the model to incorporate direct predictors of health need rather than relying on cost alone, and reported that doing so reduced the bias substantially. The case became a reference point well beyond that one product. It featured prominently in subsequent federal attention to healthcare algorithms: HHS's Office of the National Coordinator finalized the HTI-1 rule in December 2023, requiring certified health IT developers to disclose information about the data sources, development, and validation of predictive algorithms — explicitly citing the goal of letting clinicians assess fairness, appropriateness, validity, effectiveness, and safety. Separately, HHS's Section 1557 nondiscrimination regulations were updated to cover the use of AI and clinical algorithms in patient care decisions, and FDA has continued to expand its guidance on predetermined change control plans and lifecycle management for AI-enabled medical devices.
The forward-looking practice: audit the proxy, not just the outcome
That regulatory backdrop has since shifted. Since early 2025, HHS has moved to narrow rather than expand the AI-nondiscrimination provisions in Section 1557 — rescinding prior interpretive guidance and, per legal analysts, replacing the 2024 rule's explicit coverage of clinical decision-support algorithms with an approach closer to the pre-2024, 2016-era regulation. On the transparency side, ONC (now ASTP) has granted health IT developers enforcement discretion on several HTI-1 certification deadlines into 2026 and proposed a follow-on rule, HTI-5, that would roll back parts of the algorithm-transparency and certification framework the original HTI-1 rule established. For teams building on the Obermeyer lesson, the practical takeaway is unchanged — proxy-variable auditing is what actually prevents this failure mode — but it now has to be treated as an internal engineering discipline rather than something a shifting regulatory floor will enforce for you.
Fairness testing in healthcare AI too often stops at checking whether protected attributes are excluded from a model, or whether aggregate error rates look similar across groups. The Obermeyer case shows that isn't sufficient — the damage was done by a proxy variable that was silently correlated with unequal access, several steps removed from anything resembling a protected attribute. The durable lesson is that every predictive label built on healthcare data — cost, utilization, adherence, even readmission — is a measurement of a system that has never applied care equally, and needs to be interrogated as such before it becomes a target function. Proxy-variable auditing, done at model-design time and revisited across the deployment lifecycle, is becoming as core to healthcare AI safety as clinical validation itself: not a compliance checkbox, but the question every team building a risk score should ask first — what, exactly, are we teaching this model to want to predict, and whose history is embedded in that answer?