Research Paper Intelligence System
Upload a library of papers, ask questions across all of them, get answers that cite the exact page they came from.
- Period
- 2026
- Role
- Sole engineer — architecture, retrieval, infrastructure
- Team
- Solo
- Status
- Live on GitHub
3-way
Hybrid retrieval — semantic, lexical, rank fusion
Per-page
Citations mapped back to source sections
CI/CD
Containerised and tested on every push
The problem
Reading research is not a search problem, it is a cross-referencing problem. You have thirty PDFs open, you half-remember that one of them made a claim about latency under load, and no tool will tell you which. General-purpose chatbots will happily answer and invent the citation, which is worse than not answering at all.
What I built
- 01 Designed a Retrieval-Augmented Generation service in FastAPI where users upload papers and hold a conversation scoped to just those documents.
- 02 Implemented hybrid retrieval that runs semantic search through Pinecone alongside lexical BM25 search, then merges the two ranked lists with Reciprocal Rank Fusion so neither strategy's blind spot decides the answer.
- 03 Built structured chunking that preserves section and page boundaries, so every generated claim can be traced to the exact place it came from.
- 04 Wired conversation management APIs that keep retrieval scoped per conversation rather than across the whole corpus.
- 05 Containerised the whole thing with Docker, covered it with pytest, and set up GitHub Actions for continuous delivery, with Cloudflare R2 behind file storage.
Hardest part
Making citations trustworthy. Semantic search alone returns passages that feel related and cite the wrong page; keyword search alone misses anything paraphrased. Reciprocal Rank Fusion over both was the fix, but it only works if chunking respects document structure — so the retrieval quality problem turned out to be a parsing problem two layers down. That is the part I would not have predicted from the outside.
Outcome
A working system that answers across an entire uploaded library and points at the page it used, running in containers with tests and a deploy pipeline rather than living in a notebook.
The version of this that most people build stops at “embed the PDF, ask the LLM, print the answer.” That works in a demo and falls apart the moment somebody checks a citation.
I wanted the opposite failure mode: if the system cannot find support for a claim in the documents you gave it, it should say so rather than fill the gap. That single requirement drives almost every design decision here — the hybrid retrieval, the rank fusion, the structure-aware chunking, and the per-conversation scoping all exist to make the answer defensible rather than merely fluent.