AGENT SKILL

paper-corpus-rag

Build grounded question answering and retrieval-augmented generation (RAG) over your own collection of research papers, with answers that cite the exact paper and passage. Use when a user wants to "chat with" or search a folder of PDFs, synthesize evidence across a literature corpus, find which paper says X, or build a vector/hybrid index with SQLite FTS5, pgvector (Postgres), Chroma, Qdrant or FAISS. Includes a zero-dependency local full-text index with citable hits.

License 许可
MIT
最近更新
2026-11-14
Stars
★ 1.6k
Forks
491

01 前言 Front Matter

name
paper-corpus-rag
license
MIT
metadata
{ "tags": "RAG, retrieval, vector database, pgvector, Chroma, Qdrant, FAISS, SQLite, full-text search, literature synthesis", "author": "Kalaris Labs (Sayan Chowdhury)", "version": "1.0", "category": "knowledge-and-rag", "maintainer": "Kalaris Labs" }
description
Build grounded question answering and retrieval-augmented generation (RAG) over your own collection of research papers, with answers that cite the exact paper and passage. Use when a user wants to "chat with" or search a folder of PDFs, synthesize evidence across a literature corpus, find which paper says X, or build a vector/hybrid index with SQLite FTS5, pgvector (Postgres), Chroma, Qdrant or FAISS. Includes a zero-dependency local full-text index with citable hits.
compatibility
Python 3.9+ with SQLite FTS5 (standard in CPython builds). Optional pypdf, sentence-transformers, psycopg/pgvector, chromadb or qdrant-client for the vector tiers.

02 资源文件 1

03 SKILL.md 正文

RAG over a Research Paper Corpus

The goal is answers a researcher can check: every claim is traceable to a specific passage of a specific paper. Retrieval quality and citation discipline matter more than the choice of vector database.

To discover new candidate papers before building your local collection, use firecrawl-research-index for hosted paper search and passage reads. Save only papers you have a lawful copy of, with their DOI or source ID. This skill indexes the user's own files; Firecrawl's hosted index is not a local corpus download.

Tier 0: citable full-text search in one minute (no dependencies)

  1. Convert PDFs to Markdown first (layout-aware converters beat raw PDF text): use the markitdown or liteparse skill, or pip install pypdf for the built-in fallback.
  2. Index and query:
python scripts/paper_index.py index papers_md/ --db corpus.sqlite      # incremental: re-run after adding papers
python scripts/paper_index.py query "effect of batch size on calibration" --db corpus.sqlite -k 8
python scripts/paper_index.py query '"label smoothing" calibration' --db corpus.sqlite --json --full

Each hit is file#chunk + section heading + BM25 score. Quote phrases for exact matches. BM25 is strong for scientific text full of exact terms (gene names, method names, datasets), so start here before adding embeddings.

Tier 1: semantic and hybrid retrieval

Add dense embeddings when queries are conceptual ("methods that avoid labeled data"). Use hybrid scoring: BM25 and vector ranks fused with Reciprocal Rank Fusion (RRF), score = Σ 1/(60 + rank). It is robust and needs no tuning.

Embedding model choice: a scientific-domain or strong general model from sentence-transformers (see that skill). Embed chunks, not whole papers. Store the model name and dimension with the index, because changing models means re-embedding.

pgvector (Postgres). Best when you already run Postgres or need SQL filters (year, venue, author):

CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE chunks (id bigserial PRIMARY KEY, paper_id text, chunk int, section text,
                     body text, tsv tsvector GENERATED ALWAYS AS (to_tsvector('english', body)) STORED,
                     embedding vector(768));
CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);
CREATE INDEX ON chunks USING gin (tsv);
-- hybrid: fetch top-k from each, fuse with RRF in the application
SELECT id FROM chunks ORDER BY embedding <=> $1 LIMIT 50;
SELECT id FROM chunks WHERE tsv @@ websearch_to_tsquery('english', $2)
  ORDER BY ts_rank(tsv, websearch_to_tsquery('english', $2)) DESC LIMIT 50;

Chroma / Qdrant / FAISS. Local-first prototyping (Chroma), production filtering and payloads (Qdrant), or in-memory at scale (FAISS). See the chroma, qdrant-vector-search and faiss skills for APIs. Keep paper_id, chunk, section, year as metadata on every vector.

Chunking rules for papers

  • 800-1500 characters with 10-20% overlap. Split on paragraph boundaries, never mid-sentence.
  • Keep the section heading with each chunk ("Methods > Training details"), since it disambiguates.
  • Index figure/table captions as their own chunks. Drop reference lists from the main index (they pollute BM25), but keep them in a separate table for citation lookups.
  • Store bibliographic metadata (DOI, title, authors, year) per paper, looked up by DOI, not guessed.

Answering protocol (grounding)

  1. Retrieve 8-20 chunks. Re-rank if a cross-encoder is available.
  2. Answer only from retrieved text. Cite every factual sentence as [file#chunk] (or author-year once mapped to the bibliography).
  3. If the corpus does not contain the answer, say so. Do not fill gaps from memory.
  4. When papers disagree, present both with citations rather than picking one silently.
  5. For quantitative claims, quote the number verbatim with its context (dataset, metric, split).

Integrity rules

  • Never fabricate a citation handle or quote. Every cited file#chunk must exist in the index output.
  • Verify numbers against the source passage before reporting them.

Evaluation

Before trusting a pipeline, write 20-50 question→expected-passage pairs from the corpus and measure recall@k of retrieval. Most "hallucination" in paper QA is actually a retrieval miss. Re-check after changing chunking, the embedding model or the corpus.

Related skills

research-knowledge-graph (citation/concept graphs of the same corpus), markitdown, liteparse, sentence-transformers, chroma, qdrant-vector-search, faiss, open-notebook, literature-review, firecrawl-research-index.

04 同仓库更多 Skills

hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-review-with-annotations

Create, inspect, reply to, update, or resolve situated Worktable annotations. Use when feedback belongs to a Doc section, block, text range, or HTML Doc, or when the user asks to address existing review feedback and coordinate a requested artifact edit.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-find-and-synthesize-context

Find, assess, and synthesize context from Worktable Docs and Records. Use when the user asks what a workspace contains, wants workspace-backed research, needs current facts reconciled across artifacts, or requests a cited synthesis without changing workspace state.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-create-or-update-docs

Create, edit, rename, or maintain narrative Worktable Docs. Use for plans, decisions, briefs, notes, research syntheses, and other durable prose when the user requests a new artifact or a safe update to an existing one.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-create-or-manage-records

Organize repeated structured information as durable Worktable Records. Use for research datasets, inventories, directories, pipelines, trackers, and other collections whose items need identity, fields, filtering, links, validation, or individual updates.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-create-or-manage-interactive-html

Create or improve durable visual and interactive artifacts in Worktable. Use for dashboards, trackers, calculators, mini apps, slide presentations, and other requests that benefit from a polished HTML Doc.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-collaborate-threads

Read, answer, and hand off messages through durable Worktable threads with users or other authenticated participants. Use when the user asks to check addressed messages, reply to a participant, continue a thread, request another agent's help, wait briefly for a response, or watch for comments and messages sent to the agent.

★ 1.6k