The Long Question Problem: Answering One Hard Question From One Messy Dataset

Here is a question a power developer asked us, close to verbatim:

“In this interconnection queue, which active projects over 100 MW could plausibly energize before 2029 in a county that already carries a large load, how much should I discount their stated in-service dates given what has happened to comparable projects, and which of those counties has more than one such project competing for the same substation?”

One dataset. One question. And nothing about it is a query.

It is four questions wearing a trench coat: a filter, a fan-out over entities, an aggregation over history, and a judgment. It has an implicit join the dataset does not declare. It contains a term — “large load” — that appears nowhere in the schema. And one of its four parts, as we will show, is not answerable from that dataset at all, which means the right system has to say so rather than produce a number.

This is the shape of nearly every real analytical request, and it is the shape almost no retrieval architecture is designed for. The industry has spent three years optimising two things — translating a short question into SQL, and finding a passage that resembles a short question — and a long question is neither. This piece is about what is actually known: what the benchmarks say, where each mainstream approach breaks and why, what the credible vendors are really selling, and what a system that survives contact with a messy 40-million-row table has to do that none of the defaults do.

A note on figures. Benchmark numbers in this field move fast, and several of the most-quoted ones are already stale. Every leaderboard figure here was checked against the public board in August 2026 and is dated where it matters; where a widely cited number has been superseded, both the original and the current figure are given.

---

What a “long question” actually is

It helps to be precise, because the word “complex” is doing far too much work in this field. A long question has a specific decomposable structure, and each part fails differently.

Table 1. The six components of a long question. Most published systems address exactly one of them.

Notice that the difficulty is not linguistic. Every one of those parts is easy to state. The difficulty is that they have different computational shapes, and any single retrieval mechanism is good at one shape and structurally bad at the others.

---

Why both default architectures miss

The structured path, and the enterprise cliff

If the data has a schema, the obvious move is to generate SQL. The benchmark history here is the most instructive dataset in the whole field, because it is a story about what happens when a benchmark stops flattering you.

Spider (Yu et al., 2018) established the task: 10,181 questions over 200 clean, small, single-purpose databases. It is now effectively saturated, with reported execution accuracy above 91%. BIRD (Li et al., NeurIPS 2023) deliberately made it harder — 12,751 question–SQL pairs over 95 large databases with dirty values and questions that require external knowledge to answer. Human data engineers score 92.96% on it. Systems now reach roughly 80%, but almost entirely through elaborate multi-step pipelines rather than raw model capability.

Then Spider 2.0 (Lei et al., ICLR 2025) took 632 tasks from real enterprise warehouses — BigQuery and Snowflake projects with schemas that routinely exceed 1,000 columns and gold queries that routinely exceed 100 lines. The same code-agent framework that scores 91.2% on Spider 1.0 and 73.0% on BIRD solves 21.3% of Spider 2.0.

And then it moved — fast. Anyone quoting that 21.3% today is quoting history. Here is the public Spider 2.0 leaderboard as of August 2026:

Table 2. Twenty-one months after the paper reported 21.3%. The cliff did not stay put — which is the more interesting story.

So the honest 2026 reading is not “enterprise text-to-SQL is unsolved.” It splits three ways, and each part matters for how you build.

First, the gap narrowed through agent scaffolding — retrieval over the schema, execution feedback, iterative correction, test-time search — and not through a better one-shot translator. What closed the gap is precisely the machinery this article is about. That is evidence for the argument here, not against it.

Second, the residual gap concentrates where the work is most realistic. Spider 2.0-DBT, the variant closest to how analytics is actually built, still sits at 65.6 — twenty-six points below saturated Spider 1.0. Difficulty tracks realism with striking consistency.

Third, and most uncomfortable: the highest number on that board sits on the variant with the worst-documented annotation quality. The audit described in the next section found a 66.1% annotation error rate in Spider 2.0-Snow — and Spider 2.0-Snow is the variant now reporting 96.70. Those two facts belong in the same sentence. A 96.70 measured against gold queries that are two-thirds unreliable is not a number to build a roadmap on.

The underlying difficulty has not gone anywhere. What changed is how much of it the scaffolding absorbs — and how much confidence the benchmarks themselves deserve. It remains the gap between a schema you can paste into a prompt and a schema you cannot, and every real dataset is the second kind.

The dominant error class is schema linking: choosing the right tables, columns and literal values. One error analysis on BIRD, following the DIN-SQL taxonomy, attributes roughly 37% of failures to schema linking and a further 36% to JOIN errors — between them, nearly three-quarters of what goes wrong is about relating the question to the schema, not about writing SQL. And here the field has a genuine, unresolved argument worth knowing about. Maamari et al. (2024), in a paper titled The Death of Schema Linking?, argued that with strong reasoning models you should stop pruning the schema — filtering reliably removes required columns, and modern models sift irrelevant ones tolerably well. They took first place on BIRD at 71.83% with no schema linking at all. The counter-position, argued by the LinkAlign line of work at EMNLP 2025, is that this only holds while the schema fits in the context window, which on Spider 2.0-scale warehouses it emphatically does not.

Both are right, and the boundary between them is the design decision: schema linking is optional exactly when your schema is small enough not to need it. For any dataset worth calling big, retrieval over the schema is not a legacy technique, it is the load-bearing one.

An uncomfortable detour: the benchmarks are dirtier than the data

Before treating any of those numbers as ground truth, it is worth knowing what Jin, Choi, Zhu and Kang presented at CIDR 2026. They hand-audited the annotations in two widely used text-to-SQL benchmarks and found an error rate of 52.8% in BIRD Mini-Dev and 66.1% in Spider 2.0-Snow — gold queries that omit a required predicate, annotators who misunderstood the domain terminology, questions that are genuinely ambiguous. The most common pattern in both was a mismatch between the question’s semantics and the database’s.

Re-evaluating five leading agents on their corrected set changed absolute scores by −2% to 19% and moved systems by up to three leaderboard positions: CHESS went from 62% to 81% and from fourth place to first. The leaderboard was not measuring what everyone assumed.

The practical lesson is not cynicism about benchmarks. It is this: if the people who built the reference benchmark got half their own annotations wrong, your internal golden set is wrong too. We will come back to what to do about that.

The unstructured path, and the retrieval ceiling

The other default is to embed everything and retrieve by similarity. Four well-documented problems arrive in order.

Embeddings alone lose to keyword search more often than the marketing suggests. BEIR (Thakur et al., 2021) evaluated ten retrieval systems across 18 datasets and found BM25 a robust baseline that dense retrievers often fail to beat out of domain; re-ranking and late-interaction models win, at materially higher compute cost. Five years on, this is still the first thing a new team rediscovers the hard way. The production answer is hybrid: run both, fuse the rankings with Reciprocal Rank Fusion (Cormack et al., SIGIR 2009 — the one-line Σ 1/(k + rank) algorithm, k = 60, now the default in Elasticsearch, OpenSearch, Weaviate, Qdrant and Azure AI Search), then re-rank the fused head.

The chunk is the wrong unit. A chunk severed from its document loses the context that made it findable. Anthropic’s contextual retrieval work quantified the fix cleanly: prepending 50–100 generated tokens of situating context before embedding cut the top-20 retrieval failure rate from 5.7% to 3.7%; adding a contextual BM25 index took it to 2.9% (a 49% reduction); adding re-ranking took it to 1.9% (67%). Those are large, cheap gains — and note what they imply: a third of your baseline failures were caused by chunking, not by the embedding model.

Filtering breaks approximate search. Every long question carries hard predicates — over 100 MW, active, before 2029. Combining a filter with an ANN index is genuinely hard: post-filtering over-fetches and misses, pre-filtering guarantees recall but degenerates to brute force. Predicate-aware graph methods (ACORN, Filtered-DiskANN, HQANN) exist, but the failure mode is sharp and under-appreciated. Kim and Choe, analysing ACORN-1 in 2026, put a number on it: below roughly 5% selectivity traversal paths progressively fragment and recall begins to degrade, and below 1% recall collapses — because the count of predicate-satisfying candidates in each node’s neighbourhood shrinks in proportion with selectivity, severing the paths the search depends on. Your most valuable queries are the most selective ones. This is precisely where vector search is weakest and where a boring inverted index or a WHERE clause is strongest.

And similarity does not encode the question’s structure. “How often do these dates slip?” has no similar passage. It has an answer that must be computed.

The third temptation: put the whole thing in the context window

Million-token context windows make it tempting to skip retrieval. Two results should temper that.

Liu et al. (TACL 2024), Lost in the Middle, showed a U-shaped curve: models use information at the start and end of their context and degrade sharply on material in the middle — in the worst configuration, GPT-3.5-Turbo answered multi-document questions worse with the documents than without them. NoLiMa (ICML 2025) removed the literal lexical overlap that needle-in-a-haystack tests accidentally provide, and at 32K tokens 11 of the tested models fell below half of their own short-context baseline; GPT-4o dropped from 99.3% to 69.7%. Effective context is much shorter than advertised context whenever the answer has to be inferred rather than matched.

Also: a 40-million-row table does not fit in a million tokens, and never will.

---

What the numbers say about long questions specifically

Three benchmarks isolate the multi-step property directly, and they agree with each other.

Table 3. Decomposition is worth roughly 25 points (FRAMES). It moved DABstep by more than 30. It is still not sufficient.

Read those three together and the design brief writes itself. Decomposition into steps is the single highest-leverage move available — it is worth more than any model upgrade in the same period, and it is what has driven nearly every headline gain since these benchmarks were published. And the residual failure — the majority of DABstep’s hard split that still goes unsolved — lives in the place this article started: reconciling a messy dataset with the meaning of its own fields.

---

Anatomy of a system that answers the long question

Stage 1 — Profile the dataset before you ever see a question

This is the stage that separates systems that work from demos that work once, and it is almost never discussed, because it is unglamorous and it happens offline.

Before any question arrives, you compute and store, per field: the grain (what exactly is one row?), cardinality, null rate, the actual value dictionary for every low-cardinality column, unit and scale, and the observed dirt patterns. For our queue dataset that profile immediately produces the facts that decide every later answer:

• status is not a controlled vocabulary — it holds Active, active, IA Executed, In Progress, Suspended, Withdrawn, WD, and blanks. Any question containing the word “active” requires a stored mapping, not a guess.
• Capacity appears in three columns — nameplate, summer, winter — and they disagree. “Over 100 MW” is ambiguous in the data, not in the question.
• The same physical project appears under several queue IDs across revisions. Counting rows over-counts projects by a double-digit percentage.
• County is free text with no FIPS code.
• The proposed in-service date is overwritten in place. The snapshot keeps only the current value.

That last bullet is the one that matters most, and no amount of retrieval sophistication will recover from missing it. The user asked how much to discount stated in-service dates given what happened to comparable projects. The dataset has no history. It cannot answer that. A system that profiles first knows this before it starts; a system that starts at retrieval will happily produce a slippage figure by comparing today’s dates to today’s dates and calling it a trend.

This is also where the leakage trap lives. If you ever backfill history by joining later snapshots, you must record as-of time per row or you will silently train and evaluate on information that did not exist at the decision point.

Stage 2 — Decompose into a plan with a coverage contract

Decomposition is worth about 25 points (FRAMES, above), so do it explicitly and make the plan a first-class object rather than a hidden chain of thought. The plan should name, for each sub-question: the access path, the fields it will touch, the expected result grain, and — the part everyone omits — what would count as complete coverage.

The coverage contract is what turns a fan-out from a demo into an answer. “Active projects over 100 MW” is not three examples; it is a set with a cardinality, and the assembled answer must be able to state that it covered 1,412 of 1,412. Without that contract there is no observable difference between a complete answer and a lucky sample, which is exactly why systems that look good in demos disappoint in production.

Stage 3 — Route each sub-question to the right access path

The central architectural claim of this article is that “RAG versus text-to-SQL” is a false choice made at the wrong altitude. The choice is per sub-question, and one long question typically uses four paths at once.

Table 4. Six access paths. A long question is a routing problem before it is a retrieval problem.

Applied to the opening question, the routing is unambiguous once you see it: the 100 MW and date constraints are filters; “active” is a value-dictionary lookup from the profile; “large load” is a semantic match against free-text project descriptions plus a threshold; the slippage question is an aggregation that turns out to be unanswerable; and “the same substation” is a join on a dirty key. Send all of that to one vector index and you will get a fluent paragraph and no correct numbers.

Stage 4 — Resolve entities before you count anything

Every aggregate in a long question is wrong by exactly the amount your entity resolution is wrong, and this is the least fashionable and most decisive part of the stack. The methods are old and good: the Fellegi–Sunter probabilistic model, with blocking to escape the quadratic comparison space, as implemented in tools like Splink (UK Ministry of Justice) with EM-estimated match weights. Nothing about language models has improved on this; they are useful for generating candidate blocking keys and for adjudicating hard pairs, not for replacing the model.

Two practical rules. Resolve into stable internal keys and keep the mapping, so an answer can be audited back to the raw rows. And measure resolution separately — recall and precision on a hand-labelled pair sample — because a resolution defect surfaces downstream as a subtly wrong number, which is the hardest kind of bug to notice.

Stage 5 — Execute: code over data, not tokens over data

Numbers must be produced by an execution engine. The model’s job is to write the query or the code, not to perform the arithmetic; the engine’s job is to produce a result that can be re-run tomorrow and match. This is the single practice most correlated with systems that survive audit, and it is why the credible enterprise products all converged on generating queries rather than generating answers.

For the text portions, the settled recipe is: hybrid lexical + dense retrieval, fused with RRF, contextualised chunks, and a cross-encoder re-rank on the head of the fused list. Anthropic’s measured stack — 5.7% → 2.9% → 1.9% failure — is a reasonable target shape. Where memory is the constraint, binary quantization with full-precision rescoring is close to free: 32× memory reduction with recall in the high 0.90s once you oversample and rescore the shortlist.

Stage 6 — Assemble with provenance, and declare the gaps

The output of a long question is not a paragraph. It is a structured answer in which every claim carries its provenance — which rows, which path, which computation — and in which the coverage contract from stage 2 is reported rather than assumed.

And it must distinguish three states that naive systems collapse into one: the dataset says no, the dataset does not know, and the retrieval was silent. A source that returns nothing has contributed nothing; it has not contributed evidence of absence. Treating silence as a negative finding is, in our experience, the most damaging single defect a research system can have, because it produces confident conclusions with no visible symptom.

---

The practices, compared

Table 5. The practices ranked by what they actually buy. Note that the two cheapest — contextual chunking and explicit decomposition — buy the most.

---

The providers, compared

Fair warning on how to read this: every vendor benchmark in this space is run on the vendor’s own evaluation set, and after the CIDR 2026 annotation audit, no one should treat any such number as more than directional. What follows is about architectural fit, not scores.

Table 6. No row in this table answers the opening question by itself. That is the finding, not a complaint about any product.

One production data point worth having, because it is refreshingly concrete: Uber’s QueryGPT, described on their engineering blog in 2024, is a multi-agent text-to-SQL system whose central design idea is not the model but the workspace — domain-scoped bundles of tables and example queries that constrain generation. Uber reports query authoring time falling from roughly ten minutes to three. The lesson is the same one the Snowflake and dbt results point at from another direction: constraining the search space with curated domain structure beats improving the generator.

---

Four challenges no benchmark scores you on

1. Grain and unit mismatch. “Ten records” means something different when a record is a project, a queue revision, or a substation. Nearly every wrong aggregate we have traced started as a grain confusion, not a retrieval miss. Grain must be declared per source and carried through every stage.

2. Silence versus absence. Discussed above, and worth repeating because it is the failure with no symptom. A dependency that is unreachable, rate-limited or returning empty must be logged as unavailable and must contribute nothing — and the assembled answer must say which of its parts had no evidence available. Absence has to be asserted, never inferred.

3. Temporal leakage. Snapshots that overwrite their own history make retrospective questions unanswerable and, worse, quietly answerable-looking. Any dataset used to estimate how something changes needs as-of timestamps at row level, or a documented refusal.

4. Composed confidence. A long answer’s reliability is the product of its weakest stage, not the confidence of its final sentence. If entity resolution ran at 0.9 precision and the filter mapping covered 8 of 9 status values, the aggregate downstream is not “high confidence,” whatever the prose says.

---

How to evaluate it — and why your golden set is probably wrong

The CIDR 2026 result should change how you build evaluation sets, not just how you read leaderboards. If domain experts building a flagship benchmark produced a 52.8% annotation error rate, an internally written golden set assembled in an afternoon is worse.

What has worked for us:

• Per-source golden sets, not one global set. Each dataset gets its own questions with hand-checked expected records, written by someone who read the schema and executed the query.
• Evaluate the stages separately. Retrieval recall@k, entity-resolution precision and recall, plan coverage, and claim-level attribution accuracy are four different numbers. A single end-to-end score tells you nothing about where to work.
• Measure the abstention rate. A system that never says “this dataset cannot answer that” is not more capable; it is less honest. Include unanswerable questions in the set on purpose — ours is roughly one in ten.
• Verify the golden set against the data, twice. Execute every expected answer. Ambiguity in the question is an annotation bug, not a hard case.

---

What we have been building

We arrived at all of this from the other end — not from a retrieval thesis, but from accumulating datasets until the defaults visibly stopped working.

Our catalogue now spans dozens of authoritative sources across energy and grid, regulatory and trade, life sciences, innovation and funding, materials and minerals, and healthcare economics. They are wildly heterogeneous, and that heterogeneity is the whole problem: over two million measured protein–ligand binding affinities alongside 52 offshore wind lease records; interconnection queues with free-text status fields alongside rule texts where the retrievable unit is a passage; award records keyed on entity identifiers that three agencies spell three ways.

Two things became clear quickly. A single Elasticsearch cluster over all of it does not resolve these questions — not because the engine is inadequate, but because a question that needs a cohort statistic, a passage of a rule, and an entity join is not an index problem. And embedding everything into one vector space is worse, because it discards precisely the structure — grain, keys, units, dates — that the numeric parts of the question depend on.

So we build per-source retrieval models rather than one universal index. Each source is profiled, gets a hand-verified golden set, then a query planner, a compiler and an executor built and measured against that golden set, and is released as a versioned bundle that a specialist can compose with others. Composition happens at the question layer, not the storage layer: a question is decomposed, each part is routed to the bundle whose grain matches it, and results are assembled with per-claim provenance.

Some measured findings from that work, including the ones that went against our expectations:

• Retrieval recall@20 lands around 0.88 on held-out golden sets across released bundles — good enough to be useful, and nowhere near good enough to be left unchecked at the assembly stage.
• Multi-query fan-out did not beat a single well-formed conjunctive query on our biomedical index. We built the fan-out machinery expecting a clear win; the ceiling turned out to be the backend’s ranking behaviour, not the planner’s cleverness. Generating more queries against a retrieval layer that cannot rank them is motion, not progress.
• The most valuable engineering rule we adopted is the least sophisticated one: a source that returns nothing is reported as silent. Every honesty defect we have found downstream traces back to that boundary.
• Grain, declared per source, prevents more wrong answers than any reranker we have deployed.

Anyone can assemble sources into a model from the catalogue and call it from an API or a widget — that part is self-serve and free to build.

---

The thesis

The field spent years optimising the generator when the binding constraint was the data’s self-description — and the last two years have proved the point from the other direction. Spider 2.0 went from 21.3% to between 65 and 96 depending on the variant, DABstep from 14.55% to 45.2%, and essentially none of that came from a better one-shot translator. It came from scaffolding: retrieval over the schema, decomposition, execution feedback, correction loops. The pipeline was the answer.

What has not moved is the part underneath. The residual gap still sits on the variants closest to real work, the largest reported number sits on the benchmark with a 66.1% annotation error rate, and reconciling a messy table with the documentation that explains it remains the unsolved problem. Decomposition buys tens of points; contextual chunking halves retrieval failures; both are cheap, and both are still dwarfed by what you lose to an unprofiled dataset.

So the useful summary is short. Profile before you retrieve. Route per sub-question, because one long question has four computational shapes in it. Compute numbers in an engine, never in a model. Resolve entities before you count. Carry grain everywhere. Report coverage, and report silence.

And accept the part that is genuinely hard: for a real dataset, some part of a real question is unanswerable, and a system whose answer never says so is not more capable than one that does. It is just less useful, more slowly.