[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6MANRWKFRC7CYHFCHFW67QN":3,"skill-same:01M6MANRWKFRC7CYHFCHFW67QN":87},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":23,"publishedAt":45,"linkOnly":46,"markdownContent":47,"markdownBody":48,"frontMatter":49,"refName":57,"commitSha":58,"skillDirectory":59,"forksCount":60,"repositoryHtmlUrl":61,"repositoryHomepageUrl":62,"ownerHtmlUrl":63,"ownerBlogUrl":64,"ownerDescription":65,"updatedAt":66,"install":67,"resources":80},"01M6MANRWKFRC7CYHFCHFW67QN","paper-corpus-rag",null,"Build grounded question answering and retrieval-augmented generation (RAG) over your own collection of research papers, with answers that cite the exact paper and passage. Use when a user wants to \"chat with\" or search a folder of PDFs, synthesize evidence across a literature corpus, find which paper says X, or build a vector\u002Fhybrid index with SQLite FTS5, pgvector (Postgres), Chroma, Qdrant or FAISS. Includes a zero-dependency local full-text index with citable hits.","hashgraph-online\u002Fawesome-codex-plugins","hashgraph-online","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F160371962?v=4","01M6CNB6NRJVNNGN14TR01Q6D3","HOL (Hashgraph Online)","01M6JT1HSKKG9V68KQAQGBA7XP",1575,"MIT","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fpaper-corpus-rag\u002FSKILL.md",[18],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000021","资料检索与研究","SCENARIO","使用场景",[24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44],"paper corpus rag","paper","rag","postgres","when","which","fts5","papers","literature","local","across","synthesize","question","vector database","text","knowledge","sqlite","collection","vector","search","database","2026-11-10 10:12:30",false,"---\nname: paper-corpus-rag\ndescription: Build grounded question answering and retrieval-augmented generation (RAG) over your own collection of research papers, with answers that cite the exact paper and passage. Use when a user wants to \"chat with\" or search a folder of PDFs, synthesize evidence across a literature corpus, find which paper says X, or build a vector\u002Fhybrid index with SQLite FTS5, pgvector (Postgres), Chroma, Qdrant or FAISS. Includes a zero-dependency local full-text index with citable hits.\nlicense: MIT\ncompatibility: Python 3.9+ with SQLite FTS5 (standard in CPython builds). Optional pypdf, sentence-transformers, psycopg\u002Fpgvector, chromadb or qdrant-client for the vector tiers.\nmetadata:\n  version: \"1.0\"\n  category: knowledge-and-rag\n  maintainer: Kalaris Labs\n  author: Kalaris Labs (Sayan Chowdhury)\n  tags: RAG, retrieval, vector database, pgvector, Chroma, Qdrant, FAISS, SQLite, full-text search, literature synthesis\n---\n\n# RAG over a Research Paper Corpus\n\nThe goal is answers a researcher can check: every claim is traceable to a\nspecific passage of a specific paper. Retrieval quality and citation discipline\nmatter more than the choice of vector database.\n\nTo discover *new* candidate papers before building your local collection, use\n`firecrawl-research-index` for hosted paper search and passage reads. Save only\npapers you have a lawful copy of, with their DOI or source ID. This skill indexes\nthe user's own files; Firecrawl's hosted index is not a local corpus download.\n\n## Tier 0: citable full-text search in one minute (no dependencies)\n\n1. Convert PDFs to Markdown first (layout-aware converters beat raw PDF text): use the\n   `markitdown` or `liteparse` skill, or `pip install pypdf` for the built-in fallback.\n2. Index and query:\n\n```bash\npython scripts\u002Fpaper_index.py index papers_md\u002F --db corpus.sqlite      # incremental: re-run after adding papers\npython scripts\u002Fpaper_index.py query \"effect of batch size on calibration\" --db corpus.sqlite -k 8\npython scripts\u002Fpaper_index.py query '\"label smoothing\" calibration' --db corpus.sqlite --json --full\n```\n\nEach hit is `file#chunk` + section heading + BM25 score. Quote phrases for exact matches.\nBM25 is strong for scientific text full of exact terms (gene names, method names, datasets),\nso start here before adding embeddings.\n\n## Tier 1: semantic and hybrid retrieval\n\nAdd dense embeddings when queries are conceptual (\"methods that avoid labeled data\").\nUse hybrid scoring: BM25 and vector ranks fused with Reciprocal Rank Fusion (RRF),\n`score = Σ 1\u002F(60 + rank)`. It is robust and needs no tuning.\n\nEmbedding model choice: a scientific-domain or strong general model from\n`sentence-transformers` (see that skill). Embed **chunks**, not whole papers. Store the\nmodel name and dimension with the index, because changing models means re-embedding.\n\n**pgvector (Postgres).** Best when you already run Postgres or need SQL filters (year, venue, author):\n\n```sql\nCREATE EXTENSION IF NOT EXISTS vector;\nCREATE TABLE chunks (id bigserial PRIMARY KEY, paper_id text, chunk int, section text,\n                     body text, tsv tsvector GENERATED ALWAYS AS (to_tsvector('english', body)) STORED,\n                     embedding vector(768));\nCREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);\nCREATE INDEX ON chunks USING gin (tsv);\n-- hybrid: fetch top-k from each, fuse with RRF in the application\nSELECT id FROM chunks ORDER BY embedding \u003C=> $1 LIMIT 50;\nSELECT id FROM chunks WHERE tsv @@ websearch_to_tsquery('english', $2)\n  ORDER BY ts_rank(tsv, websearch_to_tsquery('english', $2)) DESC LIMIT 50;\n```\n\n**Chroma \u002F Qdrant \u002F FAISS.** Local-first prototyping (Chroma), production filtering and\npayloads (Qdrant), or in-memory at scale (FAISS). See the `chroma`, `qdrant-vector-search`\nand `faiss` skills for APIs. Keep `paper_id`, `chunk`, `section`, `year` as metadata on every vector.\n\n## Chunking rules for papers\n\n- 800-1500 characters with 10-20% overlap. Split on paragraph boundaries, never mid-sentence.\n- Keep the section heading with each chunk (\"Methods > Training details\"), since it disambiguates.\n- Index figure\u002Ftable captions as their own chunks. Drop reference lists from the main index\n  (they pollute BM25), but keep them in a separate table for citation lookups.\n- Store bibliographic metadata (DOI, title, authors, year) per paper, looked up by DOI, not guessed.\n\n## Answering protocol (grounding)\n\n1. Retrieve 8-20 chunks. Re-rank if a cross-encoder is available.\n2. Answer **only** from retrieved text. Cite every factual sentence as `[file#chunk]`\n   (or author-year once mapped to the bibliography).\n3. If the corpus does not contain the answer, say so. Do not fill gaps from memory.\n4. When papers disagree, present both with citations rather than picking one silently.\n5. For quantitative claims, quote the number verbatim with its context (dataset, metric, split).\n\n## Integrity rules\n\n- Never fabricate a citation handle or quote. Every cited `file#chunk` must exist in the index output.\n- Verify numbers against the source passage before reporting them.\n\n## Evaluation\n\nBefore trusting a pipeline, write 20-50 question→expected-passage pairs from the corpus\nand measure recall@k of retrieval. Most \"hallucination\" in paper QA is actually a\nretrieval miss. Re-check after changing chunking, the embedding model or the corpus.\n\n## Related skills\n\n`research-knowledge-graph` (citation\u002Fconcept graphs of the same corpus),\n`markitdown`, `liteparse`, `sentence-transformers`, `chroma`, `qdrant-vector-search`,\n`faiss`, `open-notebook`, `literature-review`, `firecrawl-research-index`.\n","# RAG over a Research Paper Corpus\n\nThe goal is answers a researcher can check: every claim is traceable to a\nspecific passage of a specific paper. Retrieval quality and citation discipline\nmatter more than the choice of vector database.\n\nTo discover *new* candidate papers before building your local collection, use\n`firecrawl-research-index` for hosted paper search and passage reads. Save only\npapers you have a lawful copy of, with their DOI or source ID. This skill indexes\nthe user's own files; Firecrawl's hosted index is not a local corpus download.\n\n## Tier 0: citable full-text search in one minute (no dependencies)\n\n1. Convert PDFs to Markdown first (layout-aware converters beat raw PDF text): use the\n   `markitdown` or `liteparse` skill, or `pip install pypdf` for the built-in fallback.\n2. Index and query:\n\n```bash\npython scripts\u002Fpaper_index.py index papers_md\u002F --db corpus.sqlite      # incremental: re-run after adding papers\npython scripts\u002Fpaper_index.py query \"effect of batch size on calibration\" --db corpus.sqlite -k 8\npython scripts\u002Fpaper_index.py query '\"label smoothing\" calibration' --db corpus.sqlite --json --full\n```\n\nEach hit is `file#chunk` + section heading + BM25 score. Quote phrases for exact matches.\nBM25 is strong for scientific text full of exact terms (gene names, method names, datasets),\nso start here before adding embeddings.\n\n## Tier 1: semantic and hybrid retrieval\n\nAdd dense embeddings when queries are conceptual (\"methods that avoid labeled data\").\nUse hybrid scoring: BM25 and vector ranks fused with Reciprocal Rank Fusion (RRF),\n`score = Σ 1\u002F(60 + rank)`. It is robust and needs no tuning.\n\nEmbedding model choice: a scientific-domain or strong general model from\n`sentence-transformers` (see that skill). Embed **chunks**, not whole papers. Store the\nmodel name and dimension with the index, because changing models means re-embedding.\n\n**pgvector (Postgres).** Best when you already run Postgres or need SQL filters (year, venue, author):\n\n```sql\nCREATE EXTENSION IF NOT EXISTS vector;\nCREATE TABLE chunks (id bigserial PRIMARY KEY, paper_id text, chunk int, section text,\n                     body text, tsv tsvector GENERATED ALWAYS AS (to_tsvector('english', body)) STORED,\n                     embedding vector(768));\nCREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);\nCREATE INDEX ON chunks USING gin (tsv);\n-- hybrid: fetch top-k from each, fuse with RRF in the application\nSELECT id FROM chunks ORDER BY embedding \u003C=> $1 LIMIT 50;\nSELECT id FROM chunks WHERE tsv @@ websearch_to_tsquery('english', $2)\n  ORDER BY ts_rank(tsv, websearch_to_tsquery('english', $2)) DESC LIMIT 50;\n```\n\n**Chroma \u002F Qdrant \u002F FAISS.** Local-first prototyping (Chroma), production filtering and\npayloads (Qdrant), or in-memory at scale (FAISS). See the `chroma`, `qdrant-vector-search`\nand `faiss` skills for APIs. Keep `paper_id`, `chunk`, `section`, `year` as metadata on every vector.\n\n## Chunking rules for papers\n\n- 800-1500 characters with 10-20% overlap. Split on paragraph boundaries, never mid-sentence.\n- Keep the section heading with each chunk (\"Methods > Training details\"), since it disambiguates.\n- Index figure\u002Ftable captions as their own chunks. Drop reference lists from the main index\n  (they pollute BM25), but keep them in a separate table for citation lookups.\n- Store bibliographic metadata (DOI, title, authors, year) per paper, looked up by DOI, not guessed.\n\n## Answering protocol (grounding)\n\n1. Retrieve 8-20 chunks. Re-rank if a cross-encoder is available.\n2. Answer **only** from retrieved text. Cite every factual sentence as `[file#chunk]`\n   (or author-year once mapped to the bibliography).\n3. If the corpus does not contain the answer, say so. Do not fill gaps from memory.\n4. When papers disagree, present both with citations rather than picking one silently.\n5. For quantitative claims, quote the number verbatim with its context (dataset, metric, split).\n\n## Integrity rules\n\n- Never fabricate a citation handle or quote. Every cited `file#chunk` must exist in the index output.\n- Verify numbers against the source passage before reporting them.\n\n## Evaluation\n\nBefore trusting a pipeline, write 20-50 question→expected-passage pairs from the corpus\nand measure recall@k of retrieval. Most \"hallucination\" in paper QA is actually a\nretrieval miss. Re-check after changing chunking, the embedding model or the corpus.\n\n## Related skills\n\n`research-knowledge-graph` (citation\u002Fconcept graphs of the same corpus),\n`markitdown`, `liteparse`, `sentence-transformers`, `chroma`, `qdrant-vector-search`,\n`faiss`, `open-notebook`, `literature-review`, `firecrawl-research-index`.\n",{"name":5,"license":15,"metadata":50,"description":7,"compatibility":56},{"tags":51,"author":52,"version":53,"category":54,"maintainer":55},"RAG, retrieval, vector database, pgvector, Chroma, Qdrant, FAISS, SQLite, full-text search, literature synthesis","Kalaris Labs (Sayan Chowdhury)","1.0","knowledge-and-rag","Kalaris Labs","Python 3.9+ with SQLite FTS5 (standard in CPython builds). Optional pypdf, sentence-transformers, psycopg\u002Fpgvector, chromadb or qdrant-client for the vector tiers.","main","2b63f856ce19c2a797929cacabd9e0fc936812fa","plugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fpaper-corpus-rag",491,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins","https:\u002F\u002Fhol.org\u002Fplugins\u002Fbest-codex-plugins","https:\u002F\u002Fgithub.com\u002Fhashgraph-online","https:\u002F\u002Fhol.org","Open-source standards and developer tools for interoperable AI agents: discovery, identity, communication, payments, privacy and security.","2026-11-14 14:06:13",{"source":8,"skillName":5,"commands":68,"prompt":78,"zipAvailable":79},[69,72,75],{"tool":70,"command":71},"npx","npx skills add hashgraph-online\u002Fawesome-codex-plugins --skill paper-corpus-rag",{"tool":73,"command":74},"bunx","bunx skills add hashgraph-online\u002Fawesome-codex-plugins --skill paper-corpus-rag",{"tool":76,"command":77},"pnpm","pnpm dlx skills add hashgraph-online\u002Fawesome-codex-plugins --skill paper-corpus-rag","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fpaper-corpus-rag\u002FSKILL.md",true,[81],{"path":82,"resourceType":83,"resourceTypeDesc":84,"sizeBytes":85,"sourceHtmlUrl":86},"scripts\u002Fpaper_index.py","SCRIPT","脚本",6851,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fpaper-corpus-rag\u002Fscripts\u002Fpaper_index.py",[88,97,105,113,121,129],{"skillNo":89,"name":90,"nameZh":6,"description":91,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":92,"sourceHtmlUrl":93,"categories":94,"keywords":95,"publishedAt":96},"01M7DV332YQMGD8WTRJ9DZQXTB","worktable-review-with-annotations","Create, inspect, reply to, update, or resolve situated Worktable annotations. Use when feedback belongs to a Doc section, block, text range, or HTML Doc, or when the user asks to address existing review feedback and coordinate a requested artifact edit.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-review-with-annotations\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-review-with-annotations\u002FSKILL.md",[],[],"2026-11-15 14:19:17",{"skillNo":98,"name":99,"nameZh":6,"description":100,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":101,"sourceHtmlUrl":102,"categories":103,"keywords":104,"publishedAt":96},"01M7DV2YGHCZVBEB1AHYB6CGHE","worktable-find-and-synthesize-context","Find, assess, and synthesize context from Worktable Docs and Records. Use when the user asks what a workspace contains, wants workspace-backed research, needs current facts reconciled across artifacts, or requests a cited synthesis without changing workspace state.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-find-and-synthesize-context\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-find-and-synthesize-context\u002FSKILL.md",[],[],{"skillNo":106,"name":107,"nameZh":6,"description":108,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":109,"sourceHtmlUrl":110,"categories":111,"keywords":112,"publishedAt":96},"01M7DV2TQN1H11ME2SAF9MXN7V","worktable-create-or-update-docs","Create, edit, rename, or maintain narrative Worktable Docs. Use for plans, decisions, briefs, notes, research syntheses, and other durable prose when the user requests a new artifact or a safe update to an existing one.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-update-docs\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-update-docs\u002FSKILL.md",[],[],{"skillNo":114,"name":115,"nameZh":6,"description":116,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":117,"sourceHtmlUrl":118,"categories":119,"keywords":120,"publishedAt":96},"01M7DV2NE452NEA8TXATKSNAJH","worktable-create-or-manage-records","Organize repeated structured information as durable Worktable Records. Use for research datasets, inventories, directories, pipelines, trackers, and other collections whose items need identity, fields, filtering, links, validation, or individual updates.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-records\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-records\u002FSKILL.md",[],[],{"skillNo":122,"name":123,"nameZh":6,"description":124,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":125,"sourceHtmlUrl":126,"categories":127,"keywords":128,"publishedAt":96},"01M7DV2HENRBR2KSCVM064WH0Y","worktable-create-or-manage-interactive-html","Create or improve durable visual and interactive artifacts in Worktable. Use for dashboards, trackers, calculators, mini apps, slide presentations, and other requests that benefit from a polished HTML Doc.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-interactive-html\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-interactive-html\u002FSKILL.md",[],[],{"skillNo":130,"name":131,"nameZh":6,"description":132,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":133,"sourceHtmlUrl":134,"categories":135,"keywords":136,"publishedAt":96},"01M7DV2CADR5R7RQFFYPNAGR9Y","worktable-collaborate-threads","Read, answer, and hand off messages through durable Worktable threads with users or other authenticated participants. Use when the user asks to check addressed messages, reply to a participant, continue a thread, request another agent's help, wait briefly for a response, or watch for comments and messages sent to the agent.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-collaborate-threads\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-collaborate-threads\u002FSKILL.md",[],[]]