[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M71K6HBKGHD284FTDX6J5MMW":3,"skill-same:01M71K6HBKGHD284FTDX6J5MMW":51},{"skillNo":4,"name":5,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":22,"publishedAt":26,"linkOnly":27,"markdownContent":28,"markdownBody":28,"frontMatter":29,"refName":30,"commitSha":31,"skillDirectory":32,"forksCount":33,"repositoryHtmlUrl":34,"repositoryHomepageUrl":6,"ownerHtmlUrl":35,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":36,"install":37,"resources":50},"01M71K6HBKGHD284FTDX6J5MMW","rag-pipeline",null,"KeiSeiLab\u002FKeiSeiKit","KeiSeiLab","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F283315582?v=4","01M71K2EW81JRFHSGF0T844B32","KeiSei Lab","01M71K2EWXTB52KWC65E55S6M2",7,"Apache-2.0","https:\u002F\u002Fgithub.com\u002FKeiSeiLab\u002FKeiSeiKit\u002Fblob\u002Fmain\u002Fskills\u002Frag-pipeline\u002Fskill.md",[17],{"categoryNo":18,"categoryName":19,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000002","文档处理","SCENARIO","使用场景",[23,24,25],"rag","pipeline","rag pipeline","2026-11-10 10:12:30",false,"\u003C!-- migrated from skills\u002Frag-pipeline\u002Fskill.md (lowercase legacy filename) on 2026-05-02 -->\n---\nname: rag-pipeline\ndescription: Use when building RAG (Retrieval-Augmented Generation) systems — embedding pipeline, vector database, document ingestion, semantic search, hybrid search. Triggers on \"RAG\", \"embeddings\", \"vector search\", \"semantic search\", \"document ingestion\", \"knowledge base\".\narguments:\n  - name: command\n    description: \"Command: init, ingest, search, upgrade\"\n    required: false\n  - name: provider\n    description: \"Embedding provider: openai, gemini, voyage, cohere, local (default: openai)\"\n    required: false\n---\n\n# RAG Pipeline Skill\n\n## When to use\n\n- Building a RAG system: embedding pipeline, vector store ingestion, semantic or hybrid search over documents.\n- Choosing between embedding providers (OpenAI, Gemini, Voyage, local) or vector databases (LanceDB, Qdrant, Pinecone).\n- Adding a knowledge base or document search capability to an existing application.\n\nBuild retrieval-augmented generation systems with swappable components.\n\n## Architecture\n\n```\nDocuments → Ingestion → Chunking → Embedding → Vector DB\n                                                    ↓\nQuery → Embed Query → Hybrid Search (dense + BM25) → Rerank → LLM Context\n```\n\n## Tier Selection\n\n| Tier | Embedding | Vector DB | Cost | Use Case |\n|------|-----------|-----------|------|----------|\n| **Minimal** | OpenAI small ($0.02\u002FMTok) | LanceDB (embedded) | ~$0 | Prototyping, offline |\n| **Production** | Voyage-4 or OpenAI large | LanceDB hybrid \u002F Qdrant | Low | Most projects |\n| **Multimodal** | Gemini Embedding 2 ($0.20\u002FMTok) | LanceDB \u002F Pinecone | Medium | Text + images + video |\n\n## Step 1: Init — Choose Stack\n\n### Default: LanceDB + OpenAI (zero infrastructure)\n\n```bash\nnpm install lancedb @lancedb\u002Fvectordb openai\n```\n\nLanceDB: embedded (no server), Apache Arrow, hybrid search via RRF, scales to billions, Node.js + Python native. Free forever [E1].\n\n### Embedding Providers [E1]\n\n| Provider | Model | $\u002FMTok | Dims | Context | Multimodal |\n|----------|-------|--------|------|---------|------------|\n| OpenAI | text-embedding-3-small | $0.02 | 1536 | 8K | No |\n| OpenAI | text-embedding-3-large | $0.13 | 3072 | 8K | No |\n| Gemini | Embedding 2 | $0.20 | 3072 | 8K | Text+Image+Video+Audio |\n| Voyage | voyage-3.5 | $0.06 | flex | 32K | No |\n| Cohere | Embed 4 | $0.12 | 1536 | 128K | Text+Image |\n| Local | nomic-embed-text-v2-moe | FREE | 768 | 8K | No |\n\n**Decision:** OpenAI small for text-only (cheapest quality). Gemini 2 for multimodal (only unified embedding space). Voyage for domain-specific (code\u002Flaw\u002Ffinance). Local nomic for privacy\u002Foffline.\n\n### Vector DB Comparison [E1]\n\n| DB | Type | Free Tier | Hybrid Search | Setup |\n|----|------|-----------|---------------|-------|\n| **LanceDB** | Embedded | Unlimited (OSS) | Yes (RRF) | `npm install` |\n| ChromaDB | Embedded | Unlimited (OSS) | Yes (BM25) | `pip install` |\n| Pinecone | Cloud | 2GB, 2M writes\u002Fmo | Yes | API key |\n| Qdrant | Cloud+self | 1GB RAM free | Yes | Docker or API |\n\n**Default: LanceDB** — zero ops, no server, embedded, free.\n\n## Step 2: Ingest — Document Processing\n\n### PDF Parsing [E2]\n\n**Python (best quality):**\n```bash\npip install pymupdf4llm  # PyMuPDF with LLM-optimized markdown output\n```\n```python\nimport pymupdf4llm\nmd_text = pymupdf4llm.to_markdown(\"document.pdf\")\n```\n\n**Node.js:**\n```bash\nnpm install pdf-parse  # basic text extraction\n```\n\nFor complex PDFs with tables\u002Fimages: use LlamaParse API or call PyMuPDF via subprocess.\n\n### Chunking Strategy [E2]\n\n**Default: Recursive character splitting (512 tokens, 50 overlap)**\n\n```typescript\nfunction chunkText(text: string, maxTokens = 512, overlap = 50): string[] {\n  const separators = ['\\n\\n', '\\n', '. ', ' '];\n  const chunks: string[] = [];\n  let remaining = text;\n\n  for (const sep of separators) {\n    if (remaining.length \u003C= maxTokens * 4) break; \u002F\u002F ~4 chars\u002Ftoken\n    const parts = remaining.split(sep);\n    let current = '';\n    for (const part of parts) {\n      if ((current + sep + part).length > maxTokens * 4) {\n        if (current) chunks.push(current.trim());\n        current = part;\n      } else {\n        current = current ? current + sep + part : part;\n      }\n    }\n    remaining = current;\n  }\n  if (remaining.trim()) chunks.push(remaining.trim());\n  return chunks;\n}\n```\n\n**Advanced (production):**\n- Semantic chunking: split on topic boundaries (+70% accuracy vs fixed) [E2]\n- Contextual retrieval: prepend document context to each chunk (-69% error rate with hybrid) [E2]\n- Hierarchical: paragraph + section level chunks for multi-granularity retrieval\n\n## Step 3: Embed & Store\n\n### Embedding\n\n```typescript\nimport OpenAI from 'openai';\nconst openai = new OpenAI();\n\nasync function embed(texts: string[]): Promise\u003Cnumber[][]> {\n  const res = await openai.embeddings.create({\n    model: 'text-embedding-3-small',\n    input: texts,\n  });\n  return res.data.map(d => d.embedding);\n}\n```\n\n### Gemini Multimodal (images + video + audio in same space)\n\n```typescript\nimport { GoogleGenAI } from '@google\u002Fgenai';\nconst ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY });\n\nconst result = await ai.models.embedContent({\n  model: 'gemini-embedding-exp-03-07',\n  contents: [{ parts: [{ text: 'query' }] }],\n  config: { taskType: 'RETRIEVAL_DOCUMENT', outputDimensionality: 768 },\n});\n```\n\n### Store in LanceDB\n\n```typescript\nimport lancedb from 'lancedb';\n\nconst db = await lancedb.connect('.\u002Fvectors');\nconst table = await db.createTable('docs', [\n  { id: '1', text: 'chunk text', vector: embedding, source: 'file.pdf', page: 1 },\n]);\n```\n\n## Step 4: Search\n\n### Dense Search (cosine similarity)\n\n```typescript\nconst results = await table.search(queryEmbedding).limit(5).toArray();\n```\n\n### Hybrid Search (dense + BM25 via RRF) [E2]\n\n```typescript\nconst results = await table\n  .search(queryEmbedding, 'vector')   \u002F\u002F dense\n  .search('keyword query', 'text')     \u002F\u002F full-text BM25\n  .rerank('rrf')                       \u002F\u002F Reciprocal Rank Fusion\n  .limit(5)\n  .toArray();\n```\n\nHybrid search reduces error rate ~69% vs dense-only when combined with contextual retrieval [E2].\n\n### Vercel AI SDK Pattern\n\n```typescript\nimport { embed, cosineSimilarity } from 'ai';\n\nconst { embedding } = await embed({\n  model: openai.embedding('text-embedding-3-small'),\n  value: query,\n});\n\nconst results = chunks\n  .map(c => ({ ...c, score: cosineSimilarity(embedding, c.embedding) }))\n  .sort((a, b) => b.score - a.score)\n  .slice(0, 5);\n```\n\n### Claude Tool-Based Retrieval\n\n```typescript\nconst tools = [{\n  name: 'search_documents',\n  description: 'Search the knowledge base for relevant information',\n  input_schema: {\n    type: 'object',\n    properties: {\n      query: { type: 'string', description: 'Search query' },\n      limit: { type: 'number', description: 'Max results (default 5)' },\n    },\n    required: ['query'],\n  },\n}];\n\u002F\u002F Claude decides when to search. Backend queries vector DB, returns as tool result.\n```\n\n## Cost Calculator\n\nFor 1000 documents (~500 pages, ~0.4M tokens):\n\n| Component | OpenAI small | Gemini 2 | Local |\n|-----------|-------------|----------|-------|\n| Embedding | $0.008 | $0.080 | $0 |\n| Storage (LanceDB) | $0 | $0 | $0 |\n| Per query embed | $0.000002 | $0.00002 | $0 |\n| **LLM call dominates query cost** | ~$0.003-0.015 per query |\n\n## Upgrade Paths\n\n- **Minimal → Production:** Add hybrid search (BM25 + vector), add reranking\n- **Production → Multimodal:** Switch to Gemini Embedding 2, add image\u002Fvideo ingestion\n- **Embedded → Cloud:** Swap LanceDB for Qdrant Cloud or Pinecone (API-compatible)\n",{},"main","32e85218a997294f68da27ff3dfd638196186ae5","skills\u002Frag-pipeline",8,"https:\u002F\u002Fgithub.com\u002FKeiSeiLab\u002FKeiSeiKit","https:\u002F\u002Fgithub.com\u002FKeiSeiLab","2026-10-11 11:17:46",{"source":7,"skillName":5,"commands":38,"prompt":48,"zipAvailable":49},[39,42,45],{"tool":40,"command":41},"npx","npx skills add KeiSeiLab\u002FKeiSeiKit --skill rag-pipeline",{"tool":43,"command":44},"bunx","bunx skills add KeiSeiLab\u002FKeiSeiKit --skill rag-pipeline",{"tool":46,"command":47},"pnpm","pnpm dlx skills add KeiSeiLab\u002FKeiSeiKit --skill rag-pipeline","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FKeiSeiLab\u002FKeiSeiKit\u002Fblob\u002Fmain\u002Fskills\u002Frag-pipeline\u002Fskill.md",true,[],[52,82,106,141,169,195],{"skillNo":53,"name":54,"nameZh":6,"description":55,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":56,"categories":57,"keywords":61,"publishedAt":26},"01M71K887N78V089XR5BWT0JAV","wave-audit","3-wave parallel audit — Wave 1 discovery (4 agents), Wave 2 verification (4 agents), Wave 3 cross-optimization (1 synthesizer). Each wave runs agents in parallel. Combines security review, runtime analysis, quality check, and constructor pattern into one unified workflow with evidence grading and Go\u002FNo-Go verdict.","https:\u002F\u002Fgithub.com\u002FKeiSeiLab\u002FKeiSeiKit\u002Fblob\u002Fmain\u002Fskills\u002Fwave-audit\u002FSKILL.md",[58],{"categoryNo":59,"categoryName":60,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000023","Skill与智能体开发",[62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81],"wave audit","audit","unified workflow","workflow","analysis","review","verification","security review","pattern","grading","go","security","parallel","evidence","runtime","quality check","discovery","Optimization","one","check",{"skillNo":83,"name":84,"nameZh":6,"description":85,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":86,"categories":87,"keywords":88,"publishedAt":26},"01M71K85100JQS4Y10EGCTWQXW","vm-provision","End-to-end VPS provisioning — select provider → plan → provision → harden → verify (ssh-check + firewall-diff hard-gate) → handoff. 6 phases, ≥6 AskUserQuestion calls, defensive-only. Stops if either verification primitive fails.","https:\u002F\u002Fgithub.com\u002FKeiSeiLab\u002FKeiSeiKit\u002Fblob\u002Fmain\u002Fskills\u002Fvm-provision\u002FSKILL.md",[],[89,90,91,92,93,94,95,96,97,98,80,99,100,81,68,101,102,103,104,105],"vm","vm provision","firewall","diff","provider","select","end","calls","only","either","api","hetzner","plan","defensive","verify","intent","line",{"skillNo":107,"name":108,"nameZh":6,"description":109,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":110,"licenseFileUrl":6,"sourceHtmlUrl":111,"categories":112,"keywords":121,"publishedAt":26},"01M71K80ZRH2H5ES83QS97BB91","visual-explainer","Generate beautiful, self-contained HTML pages that visually explain systems, code changes, plans, and data. Use when the user asks for a diagram, architecture overview, diff review, plan review, project recap, comparison table, or any visual explanation of technical concepts. Also use proactively when you are about to render a complex ASCII table (4+ rows or 3+ columns) — present it as a styled HTML page instead.","MIT","https:\u002F\u002Fgithub.com\u002FKeiSeiLab\u002FKeiSeiKit\u002Fblob\u002Fmain\u002Fskills\u002Fvisual-explainer\u002FSKILL.md",[113,116],{"categoryNo":114,"categoryName":115,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000009","前端开发",{"categoryNo":117,"categoryName":118,"taxonomyType":119,"taxonomyTypeDesc":120},"01JBSEEDCAT000000000000033","数据分析师","PROFESSION","适用职业",[122,123,124,125,126,127,128,129,101,130,131,132,133,134,135,136,137,138,139,140],"visual","visual explainer","explainer","project","files","image","you","architecture","generate","present","concepts","table","as","view","for","ascii table","ascii","surf","use",{"skillNo":142,"name":143,"nameZh":6,"description":144,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":145,"categories":146,"keywords":150,"publishedAt":26},"01M71K7VSD5ZWN26QJEC4T0770","video-gen","Use when generating video from frame sequences — FFmpeg extraction, WebP\u002FAVIF conversion, sprite sheets, scroll-synced playback, video encoding\u002Ftranscoding. Covers the full pipeline from video source to web-ready frame sequence or optimized video.","https:\u002F\u002Fgithub.com\u002FKeiSeiLab\u002FKeiSeiKit\u002Fblob\u002Fmain\u002Fskills\u002Fvideo-gen\u002FSKILL.md",[147],{"categoryNo":148,"categoryName":149,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000016","图像与多媒体",[151,152,153,154,24,155,156,157,158,159,127,160,161,162,163,164,165,166,167,168],"Video","gen","video gen","ready","default","generated","synced","false","when","conversion","directory","sequences","sequence","source","full","path","webp","target",{"skillNo":170,"name":171,"nameZh":6,"description":172,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":173,"categories":174,"keywords":178,"publishedAt":26},"01M71K7R88RM8ENVCRZHZSMM5T","test-matrix","Use when a project needs testing BEYOND unit tests — fuzzing, property-based, load, E2E, or mutation. Five-phase hub-and-spoke pipeline composes the right mix per language × critical path × CI target, scaffolds configs + corpus + fixtures, wires CI jobs, and defines the crash\u002Fregression triage workflow. Pure-click: every decision except intake is an AskUserQuestion.","https:\u002F\u002Fgithub.com\u002FKeiSeiLab\u002FKeiSeiKit\u002Fblob\u002Fmain\u002Fskills\u002Ftest-matrix\u002FSKILL.md",[175],{"categoryNo":176,"categoryName":177,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000008","测试与质量",[179,180,181,182,168,183,125,184,24,185,186,187,188,189,190,191,192,140,193,194],"test","matrix","test matrix","unit","click","regression","mix","intake","testing","decision","ci","tests","e2e","mutation","load","fuzzing",{"skillNo":196,"name":197,"nameZh":6,"description":198,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":199,"categories":200,"keywords":202,"publishedAt":26},"01M71K7N3ZY9FMMH3CSE1H8K5J","test-gen","Use when generating tests for untested code — happy path, edge cases, error handling","https:\u002F\u002Fgithub.com\u002FKeiSeiLab\u002FKeiSeiKit\u002Fblob\u002Fmain\u002Fskills\u002Ftest-gen\u002FSKILL.md",[201],{"categoryNo":176,"categoryName":177,"taxonomyType":20,"taxonomyTypeDesc":21},[203,152,179,204,168,136,205,190,206,140,159,207,208,209,210,166],"test gen","error","true","edge","file","code","error handling","name"]