The appealing pitch is "use AI to predict which targets and chemotypes demand will move toward." The unglamorous truth is that the model is the easy part. The signal is buried in evidence that is entity-fragmented and temporally treacherous, and those two problems — not the predictor — decide whether you get a real lead time or a backtested mirage.
What the public evidence layer actually computes
Take Open Targets, the canonical target–disease evidence resource. Its association score is not a vote count. Each data source's score is a normalized harmonic sum: sort the evidence in descending order, divide each piece by the square of its rank, sum, and normalize to 0–1; the overall score is a weighted harmonic sum across sources with tunable weights. [1] [2]
That 1/rank² weighting is a deliberate prior, not a neutral readout: it is top-heavy, so a couple of strong pieces of evidence dominate a long tail of weak ones. Change the aggregation or the source weights and the ranking moves. The score is one opinionated summary of biased evidence — useful, but a starting point you have to understand, not ground truth you can resell.
Problem one: the target wears many names — and the field can't even keep them straight
The same biological target appears as a gene symbol in one source, a protein name in another, an Ensembl or UniProt accession in a third, and a withdrawn alias in a fourth. Resolve those to one entity or "rising interest in target X" smears across six synonyms and vanishes. And lest this sound like a solved nuisance: the genomics field is so bad at entity hygiene that Microsoft Excel silently renames gene symbols into dates — SEPT2 becomes "September 2," MARCH1 becomes a date — and a 2016 audit found such errors in the supplementary files of about one in five published genomics papers, a rate that had grown when re-examined years later. [3] [4] If a default spreadsheet setting corrupts a fifth of the literature's gene lists, the idea that you can read a clean demand signal off raw target mentions — without a real resolution layer across HGNC, Ensembl, and UniProt — is fantasy. The same applies on the chemical side: one molecule, many names, which is why structural identity has to be settled before any signal is counted, not after.
Problem two: you cannot leak the future
This is the subtle one, and it's where most "we'd have caught it 18 months early" claims quietly collapse. To prove lead time you must reconstruct what each source said as of a past date — far harder than filtering a date column. Controlled vocabularies back-label old records (a paper is assigned subject terms years later). "Current-state" data pulls return today's values dressed as historical. Citation counts are always live. A backtest that doesn't freeze every source as-of the cutoff — stripping back-labeled fields, using dated snapshots — leaks the answer and reports a lead time it never had.
This failure mode is not hypothetical or niche; it is the central finding of the reproducibility reckoning in ML-based science. Kapoor and Narayanan catalogued a taxonomy of eight distinct kinds of data leakage and documented leakage-driven reproducibility failures across 17 scientific fields, affecting 329 papers, repeatedly producing wildly overoptimistic results that evaporated once the leak was closed. [5] A demand backtest is exactly the kind of temporal prediction task where this happens by default. The dated supply-side deposits that do help — the Antibody Registry timestamps reagent batches, PubChem carries dated vendor deposits — give a genuine temporal axis, but the dating is per-batch, not per-item, so even the ground truth has a resolution limit you must respect rather than launder. [6]
Why the model is the part you build last
Restated plainly: a leading demand signal is a causal-temporal inference over biased, entity-fragmented evidence. The biases are real — publication bias, hype cascades, survivorship — and separating genuine emergence from a citation echo is part of the work. The ranking model sits downstream of two things that determine whether it's worth anything: clean entity resolution and a leak-proof temporal frame. Get either wrong and a more sophisticated predictor merely overfits the flaw more confidently — which is precisely how 329 papers got it wrong. [7]
The unglamorous work — resolving the entities so a target is one thing, and freezing every source so the backtest can't see the future — is the work. The forecasting model is the part you'd build last, and the part that matters least.
If your demand model claims an N-month lead time, can it replay the frozen, as-of-date inputs that produced it — or is the lead time, like the gene lists and the 329 papers, an artifact of data that quietly knew the answer?