Executive summary

• Situation. A researcher asking about an NLP paper needs an answer that cites the exact passage behind it. I built a retrieval pipeline over the QASPER benchmark (Dasigi et al., 2021) and wrote evaluation code that ended up larger than the pipeline it tests.
• Action. I fused keyword and semantic search and ranked passages with ColBERT v2. Llama-3.1-8B then wrote short answers with a citation on every sentence. Because 40% of my test questions needed a table or figure, I extracted table bodies from each paper's arXiv LaTeX source.
• Result. On 150 questions, checked against human-marked evidence with no AI grader, evidence recall rose from 0.485 to 0.829. The share of table and figure evidence reaching the model rose from 0.000 to 0.700.

Problem statement

A language model can answer a question about a paper fluently and still cite the wrong passage. In my first evaluation runs, 32 of 150 answers drew on passages from a different paper. A researcher can only trust an answer they can check against its source.

Technical formulation. I framed this as question answering over one known paper. Given a question and its QASPER paper ID, the system retrieves passages from that paper and writes an answer of 1 or 2 sentences, citing a passage as [Doc N] in every sentence.

In scope. The 888 papers of QASPER's training split, questions with free-form reference answers, retrieval, generation, and the evaluation code itself.

Out of scope. Finding the right paper among all 888, comparison with the QASPER leaderboard (I don't yet compute its official Answer-F1), and serving. The chat interface lives in a separate repository.

Success criteria.

• Retrieval: gold evidence reaches the generator, measured as evidence recall (the share of annotated evidence that reaches the model) against QASPER's annotations.
• Citations: each answer sentence is supported by what it cites, measured with ALCE citation precision and recall.
• Grader validity: the automatic judges agree with an independent check, measured with Cohen's kappa.
• Coverage: every mean is reported next to the number of questions actually answered.

Data engineering

Source. QASPER (allenai/qasper on Hugging Face) holds 5,049 questions over 1,585 NLP papers. I indexed the 888 papers of its training split.

What I found first. Of my 150 test questions, 60 had a table or figure as gold evidence, and my index held neither. On questions answerable only from a table or figure, context recall was 0.094, against 0.494 for text questions.

What I did.

• A custom QasperChunker emits the abstract, paragraphs in 500-token chunks with 10% overlap, and one chunk per table or figure caption.
• I pulled table bodies from each paper's arXiv LaTeX source, rate-limited to 1 request every 3 seconds. Source was available for 809 of 888 papers, and I matched 3,259 of 3,631 table captions to a body, stored as Markdown.
• Every chunk starts with "Title: <paper title>. Section: <section heading>." so keyword and dense scoring both see which paper and section it came from.
• Dense vectors (SPECTER2, 768 dimensions) sit in a FAISS flat inner-product index. A BM25 index uses NLTK tokens with stop words removed and Porter stemming.

An earlier text-only build held about 47,800 chunks. I didn't log the size of the current index with tables.

Architecture

Baseline. My first pipeline passed the 10 best ColBERT passages to the model in score order. A CRAG step deleted passages that scored below a threshold, and the index held text only. Evidence recall: 0.485.

The four-stage design.

1. Hybrid retrieval: SPECTER2 dense search and BM25, fused with Reciprocal Rank Fusion (k=60), up to 100 candidates within the paper.
2. Ranking: ColBERT v2 late interaction (MaxSim over token embeddings) scores every candidate.
3. CRAG check (Yan et al., 2024): each passage gets a Correct, Ambiguous, or Incorrect label.
4. Generation: the top 20 passages go to Llama-3.1-8B-Instruct on vLLM, in their order within the paper, under a 6,000-token budget. The prompt asks for brief reasoning, then a cited final answer.

What decides the answer in practice. Most papers have 100 chunks or fewer, so stage 1 returns the whole paper for 94% of them. The answer then depends on the ColBERT ranking and the paper's own order. HyDE query expansion runs only when it can change the candidates. I keep the full design for ablations and for open-domain use, where stage 1 has to search across papers.

Trade-off I made on CRAG. On the 39 of 150 questions where deletion triggered, gold text evidence reached the model 42% of the time, against 85% elsewhere. Harder questions explain part of that gap. CRAG also assumes an outside source to replace what it removes, and this pipeline has none. So CRAG now labels passages and deletes nothing.

Choices I tested and rejected. RRF as the final ranking lowered text recall on the new index, so it stays optional. CRAG strip refinement scored 0.4 to 1.1 points lower on evidence recall than label-only mode.

Every model here is pretrained. I didn't train one in this project.

Evaluation and results

The first step came from context selection. The tables added the second. The baseline table/figure hit is zero by construction, since that index held no tables. A table counts as found when its caption reaches the model.

A full run on the current pipeline is still pending. The last attempt crashed out of memory on 2026-10-04, and I've fixed the cause.

Measurement fixes, kept apart from model gains. Four bugs in my grading code moved metrics while the system stayed the same:

• A 500-character cut on each passage hid the evidence in 64% of chunks, and context recall read 0.0616.
• A judge prompt put the claim where the graded answer belongs, and ALCE fell to about 0.012.
• Precision and recall shared one denominator, so they matched on all 128 scored rows.
• Context recall read 6 of the 10 passages the model received.

I report each fix as a correction to the instrument.

Where it still fails. BM25 drops numeric tokens on a benchmark full of numbers. At 20 passages the model cited 4 to 5 per sentence, and a prompt change cut the median to 2 per answer in a 12-question check.

Impact. The model can only cite what reaches it, so evidence recall caps answer quality. Status: research prototype, evaluated offline.

Links

• Research repository with the pipeline and evaluation code
• QASPER paper (Dasigi et al., 2021)
• RAG chatboard project: the serving code and chat interface.