Problem statement

Scientific literature grows faster than any researcher can read it, and the questions that matter span a whole corpus rather than one paper. Keyword search hands back a ranked list and leaves the reader to open, skim, compare, and assemble the answer. This system returns the cited answer directly, streamed, over a corpus of NLP research papers, served by two Docker services.

The task is open-domain question answering over a pre-indexed corpus. Given a natural-language query, retrieve the top 7 passages from an index of about 47,800 chunks built on FAISS and BM25 (from the 888 training-split papers of the QASPER benchmark, which holds 5,049 questions over 1,585 NLP papers), then write a grounded, cited answer with a local model.

Application architecture

The stack is 2 services under Docker Compose: a FastAPI backend on port 8080 and a Chainlit chat UI on port 8001. Both containers run with network_mode: host, which gives the backend direct access to Ollama on the host. The frontend waits on a health-gated backend (condition: service_healthy), polling /health every 30 seconds with a 300-second start period so the first-boot artifact download has time to finish.

The FAISS indices run about 230 MB (dense.index at 147 MB, metadata at 31 MB, sparse.pkl at 51 MB). gdown fetches them from Google Drive on first boot. compose.yml mounts no volume for them, so they download again whenever the container is recreated. The backend detects CUDA and picks HuggingFace Transformers on GPU or Ollama on CPU, with no configuration change from the user.

The RAG pipeline, in 4 stages

Stage 1. Hybrid retrieval. FAISS dense search (SPECTER2 embeddings, 768 dimensions, a flat inner-product index) fuses with BM25 sparse search through Reciprocal Rank Fusion (k=60). The app keeps the top 50 candidates.

Stage 2. ColBERT v2 reranking. Document token embeddings are computed offline, so query time costs only a MaxSim over token embeddings. That buys cross-encoder precision at retrieval scale, cutting the top-50 down to the top-7.

Stage 3. The CRAG gate. The CRAG check (Yan et al., 2024) labels each passage Correct, Ambiguous, or Incorrect. If no passage survives it, the backend falls back to the top 3 passages and still answers, rather than refusing.

Stage 4. Streamed generation. On a GPU, HuggingFace Transformers runs Llama-3.1-8B-Instruct. Without one, Ollama runs Llama 3. The model writes brief reasoning, then a final answer with inline [Doc N] citations and 6 turns of conversation memory. An async queue pushes tokens to an NDJSON streaming endpoint.

CI/CD pipeline

Every push to master triggers a GitHub Actions workflow that authenticates to GHCR with the automatic GITHUB_TOKEN, so there are no secrets to manage, then builds and pushes both Docker images under 2 tags: :latest and :sha-<commit>. The backend image clones the research pipeline at a pinned commit. compose.yml pulls the frontend image from GHCR and builds the backend locally.

Evaluation

The evaluation lives in the research repository, on 150 QASPER questions checked against human-marked evidence. In its baseline run of 2026-09-02, ALCE citation precision was 0.654 and recall 0.674, with 128 of 150 questions answered. Fixing the grading code mattered as much as the pipeline: precision and recall had shared one denominator, so they matched on every scored row. The full study, including the evidence-recall gains from indexing tables, is on the hybrid RAG project page.

Impact

• Pre-built images. CI publishes the backend and frontend images to GHCR on every push. compose.yml pulls the frontend image and builds the backend, which downloads its indices from Google Drive on first boot, so there's no Python environment to create and no index to download by hand. CPU mode needs Ollama installed on the host.
• Runs on CPU. The Ollama backend works on commodity hardware, at roughly 2 to 5 tokens per second.
• Auditable reasoning. Every answer shows its Reasoning block before the Final Answer, so a reader can check the logic that produced it.
• Follow-up questions work. 6 turns of memory mean the user doesn't have to restate context.