"Just connect the datasets" is the most expensive sentence in clinical data.
Because the moment you try, you hit the wall everyone underestimates.
Is "MI," "myocardial infarction," "heart attack," and ICD-10 I21 the same condition? (Yes — but your system has to be told.)
Is the drug in the EHR (brand name), the claims data (NDC code), and the trial protocol (generic plus the in-house compound ID) the same drug? (Yes — but mapping it is a project, not a query.)
Is patient 4471 in dataset A the same human as patient 8823 in dataset B? (Maybe — and getting it wrong is a safety problem.)
Conditions: five names, one diagnosis, lossy in between
Each clinical data source speaks a different vocabulary. EHRs lean on SNOMED CT — over 350,000 active concepts. Claims speak ICD-10-CM. Labs speak LOINC — 109,325 terms in version 2.82. [1] Reconciling them is the entire reason the OHDSI/OMOP standardized vocabularies exist: 8.76 million valid concepts drawn from 136 vocabularies, just to give the same clinical idea one home. [2]
And the mapping is genuinely lossy — not a clerical detail. One ICD-10-to-SNOMED study reached only 83% intercoder reliability, with a cited ophthalmology study as low as 35% agreement among three coders. [3] Even the canonical crosswalk is imperfect: a randomized study found 12% of SNOMED-to-ICD-10-CM coding failures were caused by errors in the official NLM map itself. [4] And the targets keep moving — CMS added 252 new ICD-10-CM codes for FY2025 alone. [5]
Drugs: many NDCs, one ingredient
Same story, different axis. The EHR records a brand name. The claim records an 11-digit NDC. The trial protocol records a generic name plus an in-house compound code. RxNorm exists to bridge them — and the fan-out it has to collapse is the point: the April 2025 release maps 248,144 distinct NDCs onto roughly 14,525 ingredients. [6] Worse, the NDC on a product label is printed in 10-digit form in one of three segment patterns (4-4-2, 5-3-2, or 5-4-1) that must be correctly zero-padded into the 11-digit billing format before any two datasets can even be joined on drug. [7] Get the padding wrong and "the same drug" silently splits into two.
Patients: the linkage problem with a body count
This is the one that isn't merely expensive — it's dangerous. The US has no national patient identifier: Section 510 of the annual Labor-HHS appropriations bill has barred HHS from adopting one every year since 1999, even though HIPAA called for it in 1996. [8] So patient identity has to be inferred. And the inference is shaky: match rates run as low as 80% within a single facility — one in five patients unmatched — and as low as 50% between organizations even when they share the same EHR vendor. [9] AHIMA attributes 35% of denied claims to inaccurate patient identification, costing the system more than $6.7 billion a year — and notes misidentification "has caused medical errors and, in some instances, even death." [10]
The unglamorous 80%
None of this is "AI-hard" in the flashy sense. It's normalization-hard and linkage-hard. It's the unglamorous majority of every real-world-evidence project that never makes the conference slide — and it's measurable: in the peer-reviewed EHDEN network, the median time just to ETL-map one dataset to the OMOP model was 358 days, with vocabulary mapping the most time-intensive step. [11] And the stakes just rose: in December 2025 the FDA moved to accept real-world evidence without always requiring identifiable individual patient data, raising the value of cleanly de-identified, well-linked datasets. [12]
Why this is an entity-resolution problem, not a search problem
Entity resolution is the task of deciding which records — across systems, vocabularies, and owners — refer to the same real-world thing: the same molecule, the same device, the same patient, the same product. It sounds like deduplication. It isn't. Deduplication assumes the duplicates are obvious. Entity resolution is the hard version, where the "same thing" wears a different name, code, and owner in every system it touches, and nothing in your data records that they're connected.
Two properties make it genuinely hard. First, it's quadratic: to know whether any record is a duplicate, you in principle have to compare it against every other record. As Papadakis and colleagues put it in the VLDB literature, entity resolution "typically scales to large volumes of data through blocking… each entity has to be compared with all others in order to find its matches." [ER-1] A thousand records is half a million comparisons; a million records is half a trillion. Double the data and you roughly quadruple the work.
Second, the right answer for one record depends on all the others — entity resolution needs global consistency. Whether record A and record B are the same can hinge on what C, D, and E say. This is exactly where throwing a bigger language model at it disappoints: large models reason locally, one context window at a time. They will confidently link two records that, seen against the whole dataset, obviously shouldn't be — and miss links that only become clear once you hold the full graph in view. The semantic layer that makes messy data usable isn't built by a model guessing one row at a time. It's built by deciding, consistently and across the whole set, that these N records are one thing.
The semantic layer isn't built by a model
Here's the part people miss in 2026: throwing a bigger LLM at this doesn't fix it. Entity resolution needs global consistency — the right answer for one record depends on all the others. A model that reasons locally, one window at a time, will confidently link two things that, seen across the whole dataset, obviously shouldn't be linked. The empirical fingerprints of local reasoning are already in the data above: 35% coder agreement, 12% errors in the official map, 50% cross-organization patient match rates. Those degrade precisely because decisions get made per-record instead of against the whole graph.
The semantic layer that makes clinical data usable isn't built by a model. It's built by people — and systems — that treat "same condition, five names" as the actual job.
What's the normalization problem that's quietly eaten the most hours on your team?