ResearchQA: A Citation-Grounded Benchmark for Scientific Paper Question-Answering

A 6,211-question benchmark for single-paper scientific QA, paired with a deterministic citation matcher that discriminates leading models 4× more sharply than LLM-evaluator metrics.

May 8, 2026

📄 Read the full paper on arXiv → · Dataset on HuggingFace · Evaluation harness on GitHub.

Abstract

Large language models are increasingly used to assist scientific reading, but existing evaluation methods often fail to detect whether answers are supported by verifiable citations. We introduce ResearchQA, a benchmark of 6,211 single-paper question-answer pairs from 494 open-access papers spanning eight domains and four question types: lookup, comprehension, multi-hop, and adversarial. ResearchQA is designed for citation-grounded evaluation: it permits multiple valid supporting passages for a claim and rewards grounded refusal when the source paper does not support an answer. We evaluate eight leading closed- and open-weight models in a citation-grounded chat-with-paper setting using a deterministic citation matcher and an LLM-based rubric evaluator. Citation-based metrics separate systems more clearly than LLM-evaluator scores: section coverage and citation accuracy vary substantially across models, while evaluator scores remain tightly compressed. We further find that open-weight models approach the best closed-model citation accuracy while achieving 3 to 6 times lower per-example latency.

Introduction

Researchers increasingly rely on large language models to read and reason over scientific literature, where two properties matter more than fluency: every claim should be traceable to verifiable evidence in the source, and the answer should surface the complete, relevant context rather than a confident subset of it. Standard evaluation methods miss both — LLM-evaluator scores reward plausible prose, and embedding similarity passes fabricated-but-on-topic text. ResearchQA measures citation-grounded answering directly, pairing LLM-generated questions with a deterministic check that a cited passage actually appears in the paper it claims to.

Our design extends the paper-level, evidence-anchored, multi-question-type template of QASPER to a domain-diverse corpus, replacing its paid human annotation with a frontier LLM paired with deterministic verification against the source text. Two methodology pitfalls shape the rest of the design: LLM-evaluator scoring suffers from rubric collapse (scores pile at the endpoints of a 1–5 scale unless every level is anchored), and citation checking faces a verbatim-vs-paraphrase tradeoff between brittle substring matching (which fails on PDF extraction noise) and embedding similarity (which passes fabricated-but-on-topic quotes). A known confound we characterize but do not eliminate in v1 is same-family bias, where an evaluator over-rates responses from its own model family.

Methods

The benchmark has three concrete artifacts — the dataset, the harness that runs models against it, and the grader that scores their outputs — plus two pieces of grader machinery (the citation matcher and the LLM evaluator) that account for most of the signal in the results.

Dataset construction

Figure 1: ResearchQA dataset construction pipeline. The corpus is drawn from OpenAlex across 8 domains. Questions and expected citations are generated by Gemini 3.1 Pro before benchmarking via the evaluation harness.

The paper corpus is sourced from OpenAlex's open-access subset across eight domains (machine learning, public health, education, environmental science, history and humanities, mathematics, psychology, and social science), selected to balance technical density (numbers, methods, tables) with prose-heavy disciplines that exercise abstractive comprehension. Question/answer/evidence triples are generated per paper chunk by Gemini 3.1 Pro under a structured-output schema that constrains the model's output shape and validates each row before it is written.

Figure 2: Row generation fan-out. The agent reads the paper and simultaneously extracts grounded chunks (pairing them with lookup and comprehension questions), constructs multi-hop questions requiring cross-section synthesis, and generates adversarial questions containing false premises.

The dataset has 6,211 rows with a four-way taxonomy:

lookup (extractive): a factual question whose answer is a single passage in the paper — e.g. "what was the sample size?" — testing whether the model can locate and quote the right span.
comprehension (abstractive): an open-ended question about themes, methodology, or implications that requires the model to synthesize a passage rather than copy it.
multi_hop: a question whose answer cannot be obtained from any single passage and requires combining evidence from two or more distant sections — e.g. comparing a result against a stated baseline, or computing a derived quantity from numbers spread across methods, results, and tables.
adversarial: a question containing a false premise or asking about something the paper does not address — e.g. asking about a placebo arm in a single-arm study — where the correct behavior is to identify the false premise and refuse rather than fabricate an answer.

Scoring uses AND across sections, OR within each section's alternatives: a model satisfies a row when it cites at least one alternative from every required section. For lookup and comprehension rows this typically resolves to a single section with multiple alternatives, so coverage is binary. For multi-hop rows it is fractional — a model that addresses two of three required sections gets 0.667 credit.

Benchmark harness

The harness routes each question through Open Paper's full chat-with-paper pipeline — retrieval over the indexed paper, the citation contract that requires the model to anchor each claim to a quoted passage, and the structured-output parser that extracts citations from the model's response.

Metrics

Each row is scored on up to six metrics. The four deterministic metrics are:

citation_precision: the fraction of the model's citations that match some alternative in some required section.
section_coverage: the fractional satisfaction of required sections — whether the model touched every required section.
citation_accuracy: the fraction of the model's citations that are verifiable substrings of the paper's raw text — whether the model fabricated quotes.
refusal_correctness (adversarial rows only): 1.0 if the model refused entirely or every citation it produced is grounded in the paper, 0.0 if any cited passage is fabricated.

The two LLM-evaluated metrics, each on a 1–5 scale with per-level anchored definitions, are factual_accuracy (whether every factual claim is consistent with the expected answer) and completeness (whether every key point in the expected answer is addressed). We deliberately measure the deterministic and evaluator metrics independently so that the citation-grounding signal cannot be drowned out by evaluator noise.

The citation matcher decides whether a model's quoted citation appears in the paper's extracted text. Because PDF text extraction introduces noise — line-break hyphenation, mid-word spaces, ligatures, smart quotes — that breaks a naive substring check, the matcher normalizes both sides through a pipeline with a whitespace-stripped fallback. We chose this over embedding similarity because the failure modes diverge: a normalization-based matcher rejects fabricated content that does not appear in the paper, while an embedding matcher would pass a fabricated-but-on-topic quote.

LLM Judge Architecture Figure 3: LLM evaluator flow. The evaluator operates on a strict 1–5 anchored rubric designed to combat score collapse, and it must output a detailed justification before outputting the final numerical score to enforce a chain-of-thought analysis.

Results

We ran the benchmark end-to-end through the Open Paper chat-with-paper harness against eight LLMs in two configurations each — a more intelligent model and a faster model: gemini-3.1-pro-preview / gemini-3-flash-preview, gpt-5.4 / gpt-4.1, claude-opus-4-7 / claude-haiku-4-5, and the Cerebras-hosted gpt-oss-120b / zai-glm-4.7. Each system answered the same 100-row evenly-spaced sample of the dataset, graded by the deterministic citation metrics and the LLM evaluator described in Methods.

Model	Cite. Prec	Sect. Cov	Cite. Acc	Refusal	Factual	Compl.	Lat. (s)
gemini-3.1-pro-preview	0.723	0.955	0.817	0.870	5.000	4.940	18.3
gpt-5.4	0.647	0.902	0.834	0.783	4.970	4.870	12.3
zai-glm-4.7	0.744	0.776	0.840	0.783	4.906	4.875	5.6
gpt-4.1	0.776	0.755	0.786	0.870	4.910	4.730	10.9
claude-opus-4-7	0.569	0.876	0.809	0.739	4.980	4.980	30.4
claude-haiku-4-5	0.716	0.843	0.845	0.739	4.802	4.659	9.8
gemini-3-flash-preview	0.491	0.913	0.835	0.652	4.990	4.940	14.8
gpt-oss-120b	0.660	0.636	0.700	0.478	4.745	4.571	2.9

Table 1: Aggregate performance of all evaluated models across the OpenPaper benchmark. Models are sorted by a composite of citation-grounding and answer-quality scores. Best scores in each column are bolded.

The citation-grounding metrics discriminate; the evaluator metrics largely saturate. The largest cross-provider spread sits in refusal_correctness (0.478–0.870, a 39-point gap on a 0–1 scale) and section_coverage (0.636–0.955, a 32-point gap). citation_precision (0.491–0.776, a 28-point gap) and citation_accuracy (0.700–0.845, a 14-point gap) follow. The LLM evaluator, by contrast, separates the field by far less: factual_accuracy (4.745–5.000, a 0.255-point gap on a 1–5 scale) and completeness (4.571–4.980, a 0.409-point gap). We read this as "current-generation models are uniformly correct on the kinds of factual claims the dataset asks about, but they vary substantially in which evidence they choose to cite and whether that evidence is verifiable " — exactly the distinction the citation metrics were designed to expose.

Figure 4: Model Performance Profiles. Frontier models (solid) and their faster counterparts (dashed) plotted across normalized metrics. Evaluator metrics (Factual, Completeness) are saturated at the perimeter, while strict citation metrics (Precision, Coverage, Refusal) reveal significant differences in model behavior. No single system dominates all dimensions.

No single system dominates. Gemini 3.1 Pro leads on composite quality, but four different models top at least one metric in Table 1 — every system is best at something and weakest at something else. Claude Opus 4.7 illustrates the pattern: it ties for the most complete answers but has the lowest citation precision, the signature of a verbose model that finds the right answers and then over-cites. The two Cerebras-hosted models stake out a separate speed-quality frontier: Zai GLM 4.7 reaches near-frontier quality at roughly a third of Gemini 3.1 Pro's latency, and GPT-OSS-120B is six times faster than Gemini 3.1 Pro at the cost of meaningful ground on refusal correctness and coverage.

Latency varies by an order of magnitude. End-to-end per-row latency ranges from 2.9 s (GPT-OSS-120B) to 30.4 s (Claude Opus 4.7). Six of the eight models run under 15 seconds per row; only the two slowest reasoning models — Gemini 3.1 Pro at 18.3 s and Claude Opus 4.7 at 30.4 s — cross that threshold.

Figure 5: Speed-Quality Frontier. Average latency plotted against a composite quality score (average of citation and normalized evaluator metrics). Fast models like gpt-oss-120b and zai-glm-4.7 establish the low-latency edge, while gemini-3.1-pro-preview leads on quality.

Multi-hop is the consistently hardest slice. Across the eight systems, multi-hop questions show the widest spread on every metric: section_coverage (0.542–0.958, a 42-point gap on a 0–1 scale), citation_accuracy (0.444–0.821, a 38-point gap), factual_accuracy (4.417–5.000, a 0.583-point gap on a 1–5 scale), and completeness (4.000–5.000, a full 1.000-point gap). Lookup and comprehension are partially saturated — every system hits at least 0.703 on lookup citation accuracy — confirming that single-passage retrieval has become an easy slice for current models. The multi-hop and adversarial categories now drive most of the cross-system signal.

Figure 6: Multi-hop Performance Drop. A comparison of section coverage on lookup versus multi-hop questions. While most models achieve near-perfect coverage on simple lookups (green), performance degrades and spreads significantly on multi-hop questions (red).

Adversarial behavior splits the field cleanly. On adversarial questions, the correct response is to identify the false premise and either refuse or refute with grounded evidence. The refusal_correctness metric — which marks a row correct when every cited passage is verifiable in the paper text — produces the largest cross-provider gap of any metric: Gemini 3.1 Pro and GPT-4.1 score 0.870, the next tier (GPT-5.4, Zai GLM 4.7) scores 0.783, the Claude pair scores 0.739, Gemini 3 Flash scores 0.652, and GPT-OSS-120B scores 0.478.

Performance varies significantly by subject matter. Aggregating the strict citation metrics reveals a clear gradient of domain difficulty. Models perform relatively well and consistently on Economics and History/Humanities. Conversely, Education and Machine Learning proved to be the most challenging domains, exhibiting the lowest median scores and the widest variance across models.

Figure 7: Domain Difficulty Variation. The domains ordered from highest to lowest median performance on the strict citation metrics. Black dots represent individual model scores, illustrating the lower medians and widening spreads on domains like Machine Learning and Education.

Implications. The deterministic citation metrics do the work the LLM evaluator cannot: they flag when a model fabricates a citation, when it ignores a required section in a multi-hop question, and when it confidently rebuts a false premise with invented evidence. Splitting measurements along citation-grounding lets us measure the differentiator directly.

Limitations

The full paper details these in depth; the headline caveats are:

No human-annotated ground truth in v1. Questions, expected answers, and evidence passages are LLM-generated and verified only by the citation matcher and the LLM evaluator. A future release will validate a stratified, expert-labeled subset and report inter-annotator agreement.
Single-evaluator, single-generator. One Gemini model grades every system including itself (same-family bias), and every question was authored by Gemini 3.1 Pro under one schema. Diversifying the evaluator and generator across model families is planned.
Contamination over time. The dataset is publicly released on HuggingFace, so future model versions may have seen these papers during training — an accepted cost of an open benchmark.
Text-centric benchmark. The harness feeds native PDFs to models that can read them, but questions are authored from text and never require interpreting a figure or diagram; a multimodal extension built specifically for visual scientific understanding is a natural follow-up.
Single-paper scope. The most natural extension applies the same generation paradigm to multi-paper QA, where a system must gather evidence across documents to answer a single question.

Resources

Paper: arXiv:2607.11074 →
Dataset: khoj-ai/ResearchQA on HuggingFace
Evaluation harness & evaluator prompt: Open Paper repository, server/evals/