Problem statement

A researcher querying a corpus of NLP papers needs an answer they can trace back to a specific passage in a specific document. Standard prompting misses that bar by a wide margin: models invent citations, blend findings from separate papers, and describe methodology that was never in the source, all with the same confident fluency.

The technical formulation is an open-domain extractive-generative QA task. Given a natural-language question and a large corpus of full-text research papers, retrieve the most relevant passages and generate a short answer with inline citations pointing at the source documents. Evaluation runs on the QASPER benchmark (Dasigi et al., 2021), 5,049 QA pairs over full-text NLP papers.

Data engineering

A custom QasperChunker splits raw paper text into overlapping 500-token chunks (10% overlap, about 50 tokens). Each chunk carries a header: Title: <paper title>. Section: <section heading>. The header anchors the chunk to its origin for both BM25 keyword matching and ColBERT scoring. Strip it out and short chunks lose the signal that tells them apart.

The corpus that comes out is 47,810 chunks, stored as a FAISS flat index (IndexFlatIP, 768-dim FP16 vectors) alongside a pickled BM25 model built from an NLTK-tokenized, stop-word-filtered, Porter-stemmed corpus.

The 4 stages

Stage 0. HyDE query expansion. A query shorter than 10 words triggers Hypothetical Document Embeddings: the model writes a passage that would answer the question, and that passage becomes the dense query in place of the raw one. It closes the vocabulary gap between a short question and the dense scientific text in the index.

Stage 1. Hybrid retrieval (top-60, RRF k=60). Dense retrieval uses allenai/specter2_base (768-dim, FP16) with its retrieval adapter. SPECTER2 trains on the global citation graph, so if Paper A cites Paper B the two land near each other whether or not they share vocabulary, and the embedding captures scientific intent rather than surface jargon. Sparse retrieval through rank-bm25 (NLTK, Porter stemming) covers what dense search misses: exact matches on distinctive method names and acronyms. Reciprocal Rank Fusion (k=60) merges the 2 ranked lists without normalizing scores that were never on the same scale.

Stage 2. ColBERT v2 reranking (top-7). ColBERT stores per-token embeddings offline and scores a query-passage pair with MaxSim: for each query token, take the highest cosine similarity against any passage token, then sum across query tokens. In scientific text, relevance often hinges on one technical term matching exactly, and token-level MaxSim catches that where a sentence-level bi-encoder averages it away.

Stage 3. The CRAG relevance gate. The best reranker logit is compared against a calibrated threshold. Below it, the system returns “insufficient context” instead of generating from weak evidence. calibrate_crag.py sets the threshold in either of 2 modes: F1-max, which grid-searches 300 candidates, or a percentile fallback that sets the threshold at the P-th percentile of the best_rerank_score distribution and needs no RAGAS labels.

Stage 4. Generation. The top-7 passages go to Llama-3.1-8B-Instruct through vLLM on a dynamically allocated GPU pool (tensor-parallel across multiple GPUs, a 50% VRAM share on a single GPU, and an Ollama fallback on CPU). The prompt requires Chain-of-Thought reasoning followed by a <Final Answer> block with [Doc N] inline citations. extract_final_answer() strips the reasoning block before RAGAS and ALCE score it, and that one fix took ALCE Recall from about 0.057 to about 0.84.

Evaluation framework

Everything runs locally, with no external API calls. Two frameworks share a single SLURM job. RAGAS (Es et al., 2023) scores retrieval and generation through an LLM judge on Context Precision, Context Recall, Faithfulness, and Answer Relevancy; the judge is Llama-3.1-8B-Instruct, served by the same vLLM instance that does the generating. ALCE (Gao et al., 2023, EMNLP) scores citation-level grounding through local NLI entailment: for each sentence in the Final Answer block, it decides whether the cited passage actually supports the claim.

Context Recall (0.5882) is the open bottleneck. The retrieval stack fails to surface the gold evidence for roughly 41% of questions, and the ColBERT v2 upgrade targets exactly that through token-level MaxSim, which is less sensitive to exact-term mismatches. Results from that run are pending.