paper-corpus-rag
Build grounded question answering and retrieval-augmented generation (RAG) over your own collection of research papers, with answers that cite the exact paper and passage. Use when a user wants to "chat with" or search a folder of PDFs, synthesize evidence across a literature corpus, find which paper says X, or build a vector/hybrid index with SQLite FTS5, pgvector (Postgres), Chroma, Qdrant or FAISS. Includes a zero-dependency local full-text index with citable hits.
- License 许可
- MIT
- 最近更新
- 2026-11-14
- Stars
- ★ 1.6k
- Forks
- 491
01 前言 Front Matter
- name
- paper-corpus-rag
- license
- MIT
- metadata
- { "tags": "RAG, retrieval, vector database, pgvector, Chroma, Qdrant, FAISS, SQLite, full-text search, literature synthesis", "author": "Kalaris Labs (Sayan Chowdhury)", "version": "1.0", "category": "knowledge-and-rag", "maintainer": "Kalaris Labs" }
- description
- Build grounded question answering and retrieval-augmented generation (RAG) over your own collection of research papers, with answers that cite the exact paper and passage. Use when a user wants to "chat with" or search a folder of PDFs, synthesize evidence across a literature corpus, find which paper says X, or build a vector/hybrid index with SQLite FTS5, pgvector (Postgres), Chroma, Qdrant or FAISS. Includes a zero-dependency local full-text index with citable hits.
- compatibility
- Python 3.9+ with SQLite FTS5 (standard in CPython builds). Optional pypdf, sentence-transformers, psycopg/pgvector, chromadb or qdrant-client for the vector tiers.
02 资源文件 1
- 脚本scripts/paper_index.py6.7 KB
03 SKILL.md 正文
RAG over a Research Paper Corpus
The goal is answers a researcher can check: every claim is traceable to a specific passage of a specific paper. Retrieval quality and citation discipline matter more than the choice of vector database.
To discover new candidate papers before building your local collection, use
firecrawl-research-index for hosted paper search and passage reads. Save only
papers you have a lawful copy of, with their DOI or source ID. This skill indexes
the user's own files; Firecrawl's hosted index is not a local corpus download.
Tier 0: citable full-text search in one minute (no dependencies)
- Convert PDFs to Markdown first (layout-aware converters beat raw PDF text): use the
markitdownorliteparseskill, orpip install pypdffor the built-in fallback. - Index and query:
python scripts/paper_index.py index papers_md/ --db corpus.sqlite # incremental: re-run after adding papers
python scripts/paper_index.py query "effect of batch size on calibration" --db corpus.sqlite -k 8
python scripts/paper_index.py query '"label smoothing" calibration' --db corpus.sqlite --json --full
Each hit is file#chunk + section heading + BM25 score. Quote phrases for exact matches.
BM25 is strong for scientific text full of exact terms (gene names, method names, datasets),
so start here before adding embeddings.
Tier 1: semantic and hybrid retrieval
Add dense embeddings when queries are conceptual ("methods that avoid labeled data").
Use hybrid scoring: BM25 and vector ranks fused with Reciprocal Rank Fusion (RRF),
score = Σ 1/(60 + rank). It is robust and needs no tuning.
Embedding model choice: a scientific-domain or strong general model from
sentence-transformers (see that skill). Embed chunks, not whole papers. Store the
model name and dimension with the index, because changing models means re-embedding.
pgvector (Postgres). Best when you already run Postgres or need SQL filters (year, venue, author):
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE chunks (id bigserial PRIMARY KEY, paper_id text, chunk int, section text,
body text, tsv tsvector GENERATED ALWAYS AS (to_tsvector('english', body)) STORED,
embedding vector(768));
CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);
CREATE INDEX ON chunks USING gin (tsv);
-- hybrid: fetch top-k from each, fuse with RRF in the application
SELECT id FROM chunks ORDER BY embedding <=> $1 LIMIT 50;
SELECT id FROM chunks WHERE tsv @@ websearch_to_tsquery('english', $2)
ORDER BY ts_rank(tsv, websearch_to_tsquery('english', $2)) DESC LIMIT 50;
Chroma / Qdrant / FAISS. Local-first prototyping (Chroma), production filtering and
payloads (Qdrant), or in-memory at scale (FAISS). See the chroma, qdrant-vector-search
and faiss skills for APIs. Keep paper_id, chunk, section, year as metadata on every vector.
Chunking rules for papers
- 800-1500 characters with 10-20% overlap. Split on paragraph boundaries, never mid-sentence.
- Keep the section heading with each chunk ("Methods > Training details"), since it disambiguates.
- Index figure/table captions as their own chunks. Drop reference lists from the main index (they pollute BM25), but keep them in a separate table for citation lookups.
- Store bibliographic metadata (DOI, title, authors, year) per paper, looked up by DOI, not guessed.
Answering protocol (grounding)
- Retrieve 8-20 chunks. Re-rank if a cross-encoder is available.
- Answer only from retrieved text. Cite every factual sentence as
[file#chunk](or author-year once mapped to the bibliography). - If the corpus does not contain the answer, say so. Do not fill gaps from memory.
- When papers disagree, present both with citations rather than picking one silently.
- For quantitative claims, quote the number verbatim with its context (dataset, metric, split).
Integrity rules
- Never fabricate a citation handle or quote. Every cited
file#chunkmust exist in the index output. - Verify numbers against the source passage before reporting them.
Evaluation
Before trusting a pipeline, write 20-50 question→expected-passage pairs from the corpus and measure recall@k of retrieval. Most "hallucination" in paper QA is actually a retrieval miss. Re-check after changing chunking, the embedding model or the corpus.
Related skills
research-knowledge-graph (citation/concept graphs of the same corpus),
markitdown, liteparse, sentence-transformers, chroma, qdrant-vector-search,
faiss, open-notebook, literature-review, firecrawl-research-index.
04 同仓库更多 Skills
worktable-review-with-annotations
Create, inspect, reply to, update, or resolve situated Worktable annotations. Use when feedback belongs to a Doc section, block, text range, or HTML Doc, or when the user asks to address existing review feedback and coordinate a requested artifact edit.
worktable-find-and-synthesize-context
Find, assess, and synthesize context from Worktable Docs and Records. Use when the user asks what a workspace contains, wants workspace-backed research, needs current facts reconciled across artifacts, or requests a cited synthesis without changing workspace state.
worktable-create-or-update-docs
Create, edit, rename, or maintain narrative Worktable Docs. Use for plans, decisions, briefs, notes, research syntheses, and other durable prose when the user requests a new artifact or a safe update to an existing one.
worktable-create-or-manage-records
Organize repeated structured information as durable Worktable Records. Use for research datasets, inventories, directories, pipelines, trackers, and other collections whose items need identity, fields, filtering, links, validation, or individual updates.
worktable-create-or-manage-interactive-html
Create or improve durable visual and interactive artifacts in Worktable. Use for dashboards, trackers, calculators, mini apps, slide presentations, and other requests that benefit from a polished HTML Doc.
worktable-collaborate-threads
Read, answer, and hand off messages through durable Worktable threads with users or other authenticated participants. Use when the user asks to check addressed messages, reply to a participant, continue a thread, request another agent's help, wait briefly for a response, or watch for comments and messages sent to the agent.