The pattern is everywhere right now. A team embeds its documents into a vector database, wires up retrieval, points a language model at the results, and demos something that looks like magic. Two weeks later, a real user asks a real question and the system confidently cites the wrong drug.
The reflex is to blame the model — pick a bigger one, tune the prompt, add a reranker. But RAG doesn't fail on life-science data because the model is weak. It fails for structural reasons that no amount of model-swapping touches. Here's what's actually breaking.
Failure 1: chunking destroys the context that meaning depends on
RAG works by splitting documents into chunks, embedding each chunk, and retrieving the closest ones. That's fine for a blog post. It's a disaster for a drug label or a 510(k), where a fact's meaning depends on context several paragraphs away — the dose that belongs to this population, the contraindication that applies to that formulation. Chunk it, and you retrieve a true-sounding sentence stripped of the qualifier that made it correct. The model then states it as fact.
Failure 2: embedding similarity is not entity identity
This is the big one. Vector search retrieves text that is similar. It has no concept of whether two records are the same thing. "Anti-CD20" and "anti-MS4A1" are the same target and embed differently; "acetaminophen" and "paracetamol" are the same molecule with different vectors; a compound's internal code name and its generic name look unrelated to an embedding model. So the retriever happily returns one identity of an entity and misses the other four, or merges two genuinely different things because their text looks alike.
This is entity resolution, and embeddings can't do it — because resolving identity requires global consistency: whether record A and record B are the same can depend on what C, D, and E say. Embedding similarity is a local, pairwise signal. It will never give you the guarantee that "these and only these records are the same drug." We've written about why that breaks naive approaches across clinical data, BD, and devices — RAG inherits the exact same failure.
Failure 3: the model reasons one window at a time
Even with perfect retrieval, the language model sees only the chunks in its context window. It reasons locally. Ask it whether two trials study the same drug and it will answer from whatever happens to be in front of it, not from the whole dataset — so it links things that, seen globally, obviously shouldn't be linked, and misses links that only emerge from the full graph. A bigger context window pushes the wall back; it doesn't remove it. The fundamental mismatch is that entity resolution is global and the model is local.
Why "just use a bigger model" keeps disappointing
This is why the upgrade treadmill never quite fixes it, and the macro data agrees. Gartner predicts organizations will abandon 60% of AI projects through 2026 for lack of AI-ready data, with 63% of data leaders lacking or unsure of adequate data-management practices. [1] MIT found roughly 95% of enterprise generative-AI pilots produced no measurable P&L impact, almost always due to the integration-and-data gap rather than model quality. [2] The model was rarely the bottleneck.
What actually fixes it
The fix doesn't live in the model. It lives below it — in the data layer that RAG quietly assumes you've already built. The unglamorous truth is that the work has to happen before retrieval, not after: getting your data into a shape where the system knows what each thing actually is, and which records are the same thing, before any chunk is ever embedded.
That means treating identity and structure as first-class parts of the system rather than afterthoughts. When that groundwork is done well, retrieval stops operating on raw look-alike strings and starts operating on resolved entities and their relationships — so the right drug comes back whether the question said "paracetamol," "acetaminophen," or an NDC. When it's skipped, no amount of model tuning recovers it. (This isn't fast or free; mapping even a single dataset into usable shape took a median of 358 days in one peer-reviewed network. [3])
None of this is anti-RAG. Retrieval and generation are still the right top layer. The point is that they sit on top of a data stack — and RAG bolted directly onto raw documents skips the layers underneath that are what make the answers true.
Garbage in, confident garbage out. The model isn't lying to you; the retrieval layer just handed it the wrong entity and asked it to sound sure.
When your RAG system gets a life-science answer wrong, can you tell whether it retrieved the wrong thing or reasoned wrong — and is anything resolving entities before retrieval?