[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6XGGEPYZPATQ5CQCKHHNB8X":3,"skill-same:01M6XGGEPYZPATQ5CQCKHHNB8X":98},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":23,"publishedAt":45,"linkOnly":46,"markdownContent":47,"markdownBody":48,"frontMatter":49,"refName":64,"commitSha":65,"skillDirectory":66,"forksCount":14,"repositoryHtmlUrl":67,"repositoryHomepageUrl":6,"ownerHtmlUrl":68,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":69,"install":70,"resources":83},"01M6XGGEPYZPATQ5CQCKHHNB8X","training-data-pipeline",null,"Build training datasets for LLM specialization from production data, frontier model distillation, and synthetic bootstrapping. Use when formatting production logs into SFT data, distilling from frontier APIs, or preparing data for fine-tuning. Covers JSONL formatting, data quality validation, deduplication, and train\u002Feval splitting.","fadrienne\u002FMyGitHub","fadrienne","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F197874571?v=4","01M6X64V8VWCRSMFZ4EPS2BWXG","Francesca Adrienne","01M6X64VD5MW4F9PHDWQKS7ZAF",0,"MIT","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fopenscience\u002Fbackend\u002Fcli\u002Fskills\u002Fml-training\u002Ftraining-data-pipeline\u002FSKILL.md",[18],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000033","数据分析师","PROFESSION","适用职业",[24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44],"training","training data pipeline","Training Data","pipeline","data","data pipeline","logs","Transformers","deduplication","tuning","model","eval","use","formatting","synthetic","distillation","jsonl","for","when","openai","synthetic data","2026-11-10 10:12:30",false,"---\nname: training-data-pipeline\ndescription: Build training datasets for LLM specialization from production data, frontier model distillation, and synthetic bootstrapping. Use when formatting production logs into SFT data, distilling from frontier APIs, or preparing data for fine-tuning. Covers JSONL formatting, data quality validation, deduplication, and train\u002Feval splitting.\ncategory: ml-training\nversion: 1.0.0\nauthor: Synthetic Sciences\nlicense: MIT\ntags: [Training Data, Data Pipeline, Fine-Tuning, Distillation, Synthetic Data, Production Data, JSONL, Data Quality]\ndependencies: [datasets, transformers, openai]\n---\n\n# Training Data Pipeline\n\n## When to Use This Skill\n\nUse this skill when you need to:\n- **Format production logs** into SFT training data (API logs, user corrections, accept\u002Freject signals)\n- **Distill from frontier models** using batch APIs (OpenAI, Anthropic) to label production inputs\n- **Bootstrap synthetic data** when fewer than 1000 real examples exist\n- **Validate data quality** before training (dedup, schema check, diversity metrics)\n- **Split data** into train\u002Feval sets with production data reserved for evaluation\n\n### Three Data Paths\n\n| Path | When to Use | Data Source | Cost |\n|------|-------------|-------------|------|\n| A) Production data | Have API logs or user feedback | Your own production systems | Free (already collected) |\n| B) Frontier distillation | Have production inputs but no labels | OpenAI\u002FAnthropic batch APIs | ~50% of real-time API cost |\n| C) Synthetic bootstrap | \u003C 1000 real examples | Frontier model generation | Varies by volume |\n\n**Always prefer Path A** — production data is the moat competitors can't replicate.\n\n## JSONL Chat Format\n\nAll training platforms (Tinker, Unsloth, TRL, Axolotl) accept this standard chat format:\n\n```jsonl\n{\"messages\": [{\"role\": \"system\", \"content\": \"You are a helpful assistant.\"}, {\"role\": \"user\", \"content\": \"What is 2+2?\"}, {\"role\": \"assistant\", \"content\": \"4\"}]}\n{\"messages\": [{\"role\": \"user\", \"content\": \"Translate to French: Hello\"}, {\"role\": \"assistant\", \"content\": \"Bonjour\"}]}\n```\n\n### Format Rules\n- One JSON object per line, no trailing commas\n- `messages` array with `role` and `content` fields\n- Roles: `system` (optional, first only), `user`, `assistant` (alternating)\n- Multi-turn: alternate user\u002Fassistant pairs within a single messages array\n- UTF-8 encoding, no BOM\n- `assistant` messages are the training targets — everything else is context\n\n### Platform-Specific Notes\n\n**Tinker**: Standard chat format above. Max 32K tokens per example. System message optional.\n\n**Unsloth\u002FTRL**: Same format. Also accepts `{\"prompt\": \"...\", \"completion\": \"...\"}` for simple pairs. Chat format preferred for multi-turn.\n\n**Axolotl**: Supports multiple formats via config. Recommend `chat_template` type with standard JSONL.\n\n## Path A: Production Data Collection\n\n### From API Logs\n\nIf you log API requests\u002Fresponses, convert them directly:\n\n```python\nimport json\n\ndef api_log_to_training(log_entry):\n    \"\"\"Convert an API request\u002Fresponse log to training format.\"\"\"\n    messages = []\n\n    # Add system prompt if present\n    if log_entry.get(\"system_prompt\"):\n        messages.append({\n            \"role\": \"system\",\n            \"content\": log_entry[\"system_prompt\"]\n        })\n\n    # Add the user's input\n    messages.append({\n        \"role\": \"user\",\n        \"content\": log_entry[\"user_input\"]\n    })\n\n    # Add the response (use corrected version if available)\n    response = log_entry.get(\"corrected_response\") or log_entry[\"api_response\"]\n    messages.append({\n        \"role\": \"assistant\",\n        \"content\": response\n    })\n\n    return {\"messages\": messages}\n\n# Process logs\nwith open(\"api_logs.jsonl\") as f, open(\"training_data.jsonl\", \"w\") as out:\n    for line in f:\n        log = json.loads(line)\n        example = api_log_to_training(log)\n        out.write(json.dumps(example) + \"\\n\")\n```\n\n### From User Corrections\n\nUser corrections (edits to model output) are the highest-quality training signal:\n\n```python\ndef correction_to_training(original_input, corrected_output, system_prompt=None):\n    \"\"\"Convert a user correction into a training example.\n    The corrected output becomes the training target.\"\"\"\n    messages = []\n    if system_prompt:\n        messages.append({\"role\": \"system\", \"content\": system_prompt})\n    messages.append({\"role\": \"user\", \"content\": original_input})\n    messages.append({\"role\": \"assistant\", \"content\": corrected_output})\n    return {\"messages\": messages}\n```\n\n### From Accept\u002FReject Signals\n\nIf users accept or reject model outputs, use accepted outputs as positive examples:\n\n```python\ndef filter_accepted(logs):\n    \"\"\"Keep only examples where user accepted the output.\"\"\"\n    accepted = []\n    for log in logs:\n        if log.get(\"user_action\") == \"accepted\":\n            accepted.append({\n                \"messages\": [\n                    {\"role\": \"user\", \"content\": log[\"input\"]},\n                    {\"role\": \"assistant\", \"content\": log[\"output\"]}\n                ]\n            })\n    return accepted\n```\n\n## Path B: Frontier Distillation\n\nUse frontier models to label your production inputs. Best when you have real inputs but no gold labels.\n\n### OpenAI Batch API (50% discount)\n\n```python\nimport json\n\ndef create_batch_file(inputs, system_prompt, model=\"gpt-4o\"):\n    \"\"\"Create a batch file for OpenAI Batch API.\"\"\"\n    requests = []\n    for i, user_input in enumerate(inputs):\n        requests.append({\n            \"custom_id\": f\"request-{i}\",\n            \"method\": \"POST\",\n            \"url\": \"\u002Fv1\u002Fchat\u002Fcompletions\",\n            \"body\": {\n                \"model\": model,\n                \"messages\": [\n                    {\"role\": \"system\", \"content\": system_prompt},\n                    {\"role\": \"user\", \"content\": user_input}\n                ],\n                \"max_tokens\": 4096\n            }\n        })\n\n    with open(\"batch_input.jsonl\", \"w\") as f:\n        for req in requests:\n            f.write(json.dumps(req) + \"\\n\")\n    return \"batch_input.jsonl\"\n\n# Submit batch\n# openai api batches create -i batch_input.jsonl -e \u002Fv1\u002Fchat\u002Fcompletions -c 24h\n```\n\n### Anthropic Batch API\n\n```python\nimport anthropic\n\nclient = anthropic.Anthropic()\n\ndef create_anthropic_batch(inputs, system_prompt, model=\"claude-sonnet-4-5-20250929\"):\n    \"\"\"Create batch request for Anthropic Message Batches API.\"\"\"\n    requests = []\n    for i, user_input in enumerate(inputs):\n        requests.append({\n            \"custom_id\": f\"request-{i}\",\n            \"params\": {\n                \"model\": model,\n                \"max_tokens\": 4096,\n                \"system\": system_prompt,\n                \"messages\": [\n                    {\"role\": \"user\", \"content\": user_input}\n                ]\n            }\n        })\n\n    batch = client.messages.batches.create(requests=requests)\n    return batch.id\n```\n\n### Processing Batch Results\n\n```python\ndef batch_results_to_training(results_file, inputs, system_prompt=None):\n    \"\"\"Convert batch API results into training JSONL.\"\"\"\n    training = []\n    with open(results_file) as f:\n        for line in f:\n            result = json.loads(line)\n            idx = int(result[\"custom_id\"].split(\"-\")[1])\n            messages = []\n            if system_prompt:\n                messages.append({\"role\": \"system\", \"content\": system_prompt})\n            messages.append({\"role\": \"user\", \"content\": inputs[idx]})\n            # Extract assistant response from batch result\n            content = result[\"response\"][\"body\"][\"choices\"][0][\"message\"][\"content\"]\n            messages.append({\"role\": \"assistant\", \"content\": content})\n            training.append({\"messages\": messages})\n\n    with open(\"distilled_training.jsonl\", \"w\") as f:\n        for example in training:\n            f.write(json.dumps(example) + \"\\n\")\n    return len(training)\n```\n\n## Path C: Synthetic Bootstrapping\n\nGenerate training data from scratch when you have \u003C 1000 real examples. Use as a starting point, then replace with production data as it accumulates.\n\n### Seed Prompt Strategy\n\n```python\nimport openai\n\nclient = openai.OpenAI()\n\ndef generate_synthetic_examples(task_description, seed_examples, n=500, model=\"gpt-4o\"):\n    \"\"\"Generate diverse synthetic training examples from seed examples.\"\"\"\n\n    meta_prompt = f\"\"\"You are generating training data for an LLM that will be fine-tuned for:\n{task_description}\n\nHere are {len(seed_examples)} real examples of the desired behavior:\n{json.dumps(seed_examples[:5], indent=2)}\n\nGenerate a NEW, diverse example. The input should cover a different scenario than\nthe seeds. The output should match the quality and style of the examples above.\n\nReturn JSON: {{\"input\": \"...\", \"output\": \"...\"}}\"\"\"\n\n    examples = []\n    for i in range(n):\n        response = client.chat.completions.create(\n            model=model,\n            messages=[{\"role\": \"user\", \"content\": meta_prompt}],\n            response_format={\"type\": \"json_object\"},\n            temperature=0.9,  # High temp for diversity\n        )\n        example = json.loads(response.choices[0].message.content)\n        examples.append({\n            \"messages\": [\n                {\"role\": \"user\", \"content\": example[\"input\"]},\n                {\"role\": \"assistant\", \"content\": example[\"output\"]}\n            ]\n        })\n    return examples\n```\n\n### Diversity Strategies\n- Vary temperature (0.7-1.0) across generation batches\n- Use different frontier models (GPT-4o, Claude, Gemini) to reduce model-specific bias\n- Seed with representative prompts from different categories\u002Fdifficulty levels\n- Include edge cases and adversarial examples explicitly in seed prompts\n\n## Data Quality Validation\n\n### Schema Validation\n\n```python\ndef validate_jsonl(filepath):\n    \"\"\"Validate JSONL training file format.\"\"\"\n    errors = []\n    valid = 0\n    with open(filepath) as f:\n        for i, line in enumerate(f, 1):\n            try:\n                obj = json.loads(line)\n            except json.JSONDecodeError as e:\n                errors.append(f\"Line {i}: Invalid JSON — {e}\")\n                continue\n\n            if \"messages\" not in obj:\n                errors.append(f\"Line {i}: Missing 'messages' key\")\n                continue\n\n            msgs = obj[\"messages\"]\n            if not isinstance(msgs, list) or len(msgs) \u003C 2:\n                errors.append(f\"Line {i}: 'messages' must be a list with >= 2 entries\")\n                continue\n\n            # Check roles\n            has_user = any(m.get(\"role\") == \"user\" for m in msgs)\n            has_assistant = any(m.get(\"role\") == \"assistant\" for m in msgs)\n            if not has_user or not has_assistant:\n                errors.append(f\"Line {i}: Must have at least one user and one assistant message\")\n                continue\n\n            for j, msg in enumerate(msgs):\n                if \"role\" not in msg or \"content\" not in msg:\n                    errors.append(f\"Line {i}, message {j}: Missing 'role' or 'content'\")\n                elif msg[\"role\"] not in (\"system\", \"user\", \"assistant\"):\n                    errors.append(f\"Line {i}, message {j}: Invalid role '{msg['role']}'\")\n\n            valid += 1\n\n    return {\"valid\": valid, \"errors\": errors, \"total\": valid + len(errors)}\n```\n\n### Deduplication (MinHash)\n\n```python\nfrom datasketch import MinHash, MinHashLSH\n\ndef deduplicate_dataset(examples, threshold=0.8):\n    \"\"\"Remove near-duplicate examples using MinHash LSH.\"\"\"\n    lsh = MinHashLSH(threshold=threshold, num_perm=128)\n    unique = []\n\n    for i, ex in enumerate(examples):\n        # Hash the assistant's response (training target)\n        text = ex[\"messages\"][-1][\"content\"]\n        m = MinHash(num_perm=128)\n        for word in text.lower().split():\n            m.update(word.encode(\"utf-8\"))\n\n        key = f\"doc-{i}\"\n        if not lsh.query(m):\n            lsh.insert(key, m)\n            unique.append(ex)\n\n    removed = len(examples) - len(unique)\n    print(f\"Removed {removed} duplicates ({removed\u002Flen(examples)*100:.1f}%)\")\n    return unique\n```\n\n### Diversity Metrics\n\n```python\nfrom collections import Counter\n\ndef distinct_n(texts, n=2):\n    \"\"\"Calculate distinct-n metric (ratio of unique n-grams to total n-grams).\"\"\"\n    total_ngrams = Counter()\n    for text in texts:\n        words = text.lower().split()\n        ngrams = [tuple(words[i:i+n]) for i in range(len(words)-n+1)]\n        total_ngrams.update(ngrams)\n    if sum(total_ngrams.values()) == 0:\n        return 0\n    return len(total_ngrams) \u002F sum(total_ngrams.values())\n\ndef dataset_diversity_report(examples):\n    \"\"\"Generate diversity metrics for a training dataset.\"\"\"\n    responses = [ex[\"messages\"][-1][\"content\"] for ex in examples]\n    inputs = [m[\"content\"] for ex in examples for m in ex[\"messages\"] if m[\"role\"] == \"user\"]\n\n    report = {\n        \"total_examples\": len(examples),\n        \"avg_response_length\": sum(len(r.split()) for r in responses) \u002F len(responses),\n        \"avg_input_length\": sum(len(i.split()) for i in inputs) \u002F len(inputs),\n        \"distinct_1\": distinct_n(responses, 1),\n        \"distinct_2\": distinct_n(responses, 2),\n        \"distinct_3\": distinct_n(responses, 3),\n    }\n    return report\n```\n\n## Train\u002FEval Split\n\n```python\nimport random\n\ndef split_dataset(examples, eval_ratio=0.1, production_indices=None):\n    \"\"\"Split dataset into train\u002Feval, keeping production data in eval for ground truth.\n\n    Args:\n        examples: List of training examples\n        eval_ratio: Fraction of data for evaluation (default 10%)\n        production_indices: Indices of real production examples (always go to eval)\n    \"\"\"\n    production_indices = set(production_indices or [])\n    synthetic = [ex for i, ex in enumerate(examples) if i not in production_indices]\n    production = [ex for i, ex in enumerate(examples) if i in production_indices]\n\n    # Production data goes to eval (ground truth)\n    eval_set = list(production)\n\n    # Fill remaining eval budget from synthetic\n    remaining_eval = max(0, int(len(examples) * eval_ratio) - len(eval_set))\n    random.shuffle(synthetic)\n    eval_set.extend(synthetic[:remaining_eval])\n    train_set = synthetic[remaining_eval:]\n\n    print(f\"Train: {len(train_set)}, Eval: {len(eval_set)} \"\n          f\"({len(production)} production + {len(eval_set)-len(production)} synthetic)\")\n    return train_set, eval_set\n```\n\n## Common Issues\n\n| Issue | Cause | Fix |\n|-------|-------|-----|\n| `JSONDecodeError` | Trailing commas or malformed JSON | Run `validate_jsonl()` and fix flagged lines |\n| Tokenizer mismatch | Data tokenized for wrong model | Always use target model's tokenizer for length checks |\n| Training loss doesn't decrease | Data too noisy or contradictory | Filter low-quality examples, check for duplicates |\n| Model repeats training data | Overfitting on small dataset | Add more diverse examples, reduce epochs |\n| Data leakage | Eval examples appear in training | Use `split_dataset()` with `production_indices` |\n| Encoding errors | Non-UTF-8 characters | `text.encode('utf-8', errors='replace').decode('utf-8')` |\n| Examples too long | Exceeds model context | Truncate or split long conversations, check tokenizer limits |\n\n## Quick Start Checklist\n\n1. **Identify data source**: Production logs (A), frontier distillation (B), or synthetic (C)\n2. **Format to JSONL**: Standard chat format with messages array\n3. **Validate**: Run `validate_jsonl()` on the output file\n4. **Deduplicate**: Run MinHash dedup with 0.8 threshold\n5. **Check diversity**: Run `dataset_diversity_report()`, aim for distinct-2 > 0.5\n6. **Split**: 90\u002F10 train\u002Feval, production data in eval set\n7. **Count tokens**: Verify no examples exceed model's context window\n8. **Proceed to training**: Load `tinker` or `unsloth` skill for next step\n","# Training Data Pipeline\n\n## When to Use This Skill\n\nUse this skill when you need to:\n- **Format production logs** into SFT training data (API logs, user corrections, accept\u002Freject signals)\n- **Distill from frontier models** using batch APIs (OpenAI, Anthropic) to label production inputs\n- **Bootstrap synthetic data** when fewer than 1000 real examples exist\n- **Validate data quality** before training (dedup, schema check, diversity metrics)\n- **Split data** into train\u002Feval sets with production data reserved for evaluation\n\n### Three Data Paths\n\n| Path | When to Use | Data Source | Cost |\n|------|-------------|-------------|------|\n| A) Production data | Have API logs or user feedback | Your own production systems | Free (already collected) |\n| B) Frontier distillation | Have production inputs but no labels | OpenAI\u002FAnthropic batch APIs | ~50% of real-time API cost |\n| C) Synthetic bootstrap | \u003C 1000 real examples | Frontier model generation | Varies by volume |\n\n**Always prefer Path A** — production data is the moat competitors can't replicate.\n\n## JSONL Chat Format\n\nAll training platforms (Tinker, Unsloth, TRL, Axolotl) accept this standard chat format:\n\n```jsonl\n{\"messages\": [{\"role\": \"system\", \"content\": \"You are a helpful assistant.\"}, {\"role\": \"user\", \"content\": \"What is 2+2?\"}, {\"role\": \"assistant\", \"content\": \"4\"}]}\n{\"messages\": [{\"role\": \"user\", \"content\": \"Translate to French: Hello\"}, {\"role\": \"assistant\", \"content\": \"Bonjour\"}]}\n```\n\n### Format Rules\n- One JSON object per line, no trailing commas\n- `messages` array with `role` and `content` fields\n- Roles: `system` (optional, first only), `user`, `assistant` (alternating)\n- Multi-turn: alternate user\u002Fassistant pairs within a single messages array\n- UTF-8 encoding, no BOM\n- `assistant` messages are the training targets — everything else is context\n\n### Platform-Specific Notes\n\n**Tinker**: Standard chat format above. Max 32K tokens per example. System message optional.\n\n**Unsloth\u002FTRL**: Same format. Also accepts `{\"prompt\": \"...\", \"completion\": \"...\"}` for simple pairs. Chat format preferred for multi-turn.\n\n**Axolotl**: Supports multiple formats via config. Recommend `chat_template` type with standard JSONL.\n\n## Path A: Production Data Collection\n\n### From API Logs\n\nIf you log API requests\u002Fresponses, convert them directly:\n\n```python\nimport json\n\ndef api_log_to_training(log_entry):\n    \"\"\"Convert an API request\u002Fresponse log to training format.\"\"\"\n    messages = []\n\n    # Add system prompt if present\n    if log_entry.get(\"system_prompt\"):\n        messages.append({\n            \"role\": \"system\",\n            \"content\": log_entry[\"system_prompt\"]\n        })\n\n    # Add the user's input\n    messages.append({\n        \"role\": \"user\",\n        \"content\": log_entry[\"user_input\"]\n    })\n\n    # Add the response (use corrected version if available)\n    response = log_entry.get(\"corrected_response\") or log_entry[\"api_response\"]\n    messages.append({\n        \"role\": \"assistant\",\n        \"content\": response\n    })\n\n    return {\"messages\": messages}\n\n# Process logs\nwith open(\"api_logs.jsonl\") as f, open(\"training_data.jsonl\", \"w\") as out:\n    for line in f:\n        log = json.loads(line)\n        example = api_log_to_training(log)\n        out.write(json.dumps(example) + \"\\n\")\n```\n\n### From User Corrections\n\nUser corrections (edits to model output) are the highest-quality training signal:\n\n```python\ndef correction_to_training(original_input, corrected_output, system_prompt=None):\n    \"\"\"Convert a user correction into a training example.\n    The corrected output becomes the training target.\"\"\"\n    messages = []\n    if system_prompt:\n        messages.append({\"role\": \"system\", \"content\": system_prompt})\n    messages.append({\"role\": \"user\", \"content\": original_input})\n    messages.append({\"role\": \"assistant\", \"content\": corrected_output})\n    return {\"messages\": messages}\n```\n\n### From Accept\u002FReject Signals\n\nIf users accept or reject model outputs, use accepted outputs as positive examples:\n\n```python\ndef filter_accepted(logs):\n    \"\"\"Keep only examples where user accepted the output.\"\"\"\n    accepted = []\n    for log in logs:\n        if log.get(\"user_action\") == \"accepted\":\n            accepted.append({\n                \"messages\": [\n                    {\"role\": \"user\", \"content\": log[\"input\"]},\n                    {\"role\": \"assistant\", \"content\": log[\"output\"]}\n                ]\n            })\n    return accepted\n```\n\n## Path B: Frontier Distillation\n\nUse frontier models to label your production inputs. Best when you have real inputs but no gold labels.\n\n### OpenAI Batch API (50% discount)\n\n```python\nimport json\n\ndef create_batch_file(inputs, system_prompt, model=\"gpt-4o\"):\n    \"\"\"Create a batch file for OpenAI Batch API.\"\"\"\n    requests = []\n    for i, user_input in enumerate(inputs):\n        requests.append({\n            \"custom_id\": f\"request-{i}\",\n            \"method\": \"POST\",\n            \"url\": \"\u002Fv1\u002Fchat\u002Fcompletions\",\n            \"body\": {\n                \"model\": model,\n                \"messages\": [\n                    {\"role\": \"system\", \"content\": system_prompt},\n                    {\"role\": \"user\", \"content\": user_input}\n                ],\n                \"max_tokens\": 4096\n            }\n        })\n\n    with open(\"batch_input.jsonl\", \"w\") as f:\n        for req in requests:\n            f.write(json.dumps(req) + \"\\n\")\n    return \"batch_input.jsonl\"\n\n# Submit batch\n# openai api batches create -i batch_input.jsonl -e \u002Fv1\u002Fchat\u002Fcompletions -c 24h\n```\n\n### Anthropic Batch API\n\n```python\nimport anthropic\n\nclient = anthropic.Anthropic()\n\ndef create_anthropic_batch(inputs, system_prompt, model=\"claude-sonnet-4-5-20250929\"):\n    \"\"\"Create batch request for Anthropic Message Batches API.\"\"\"\n    requests = []\n    for i, user_input in enumerate(inputs):\n        requests.append({\n            \"custom_id\": f\"request-{i}\",\n            \"params\": {\n                \"model\": model,\n                \"max_tokens\": 4096,\n                \"system\": system_prompt,\n                \"messages\": [\n                    {\"role\": \"user\", \"content\": user_input}\n                ]\n            }\n        })\n\n    batch = client.messages.batches.create(requests=requests)\n    return batch.id\n```\n\n### Processing Batch Results\n\n```python\ndef batch_results_to_training(results_file, inputs, system_prompt=None):\n    \"\"\"Convert batch API results into training JSONL.\"\"\"\n    training = []\n    with open(results_file) as f:\n        for line in f:\n            result = json.loads(line)\n            idx = int(result[\"custom_id\"].split(\"-\")[1])\n            messages = []\n            if system_prompt:\n                messages.append({\"role\": \"system\", \"content\": system_prompt})\n            messages.append({\"role\": \"user\", \"content\": inputs[idx]})\n            # Extract assistant response from batch result\n            content = result[\"response\"][\"body\"][\"choices\"][0][\"message\"][\"content\"]\n            messages.append({\"role\": \"assistant\", \"content\": content})\n            training.append({\"messages\": messages})\n\n    with open(\"distilled_training.jsonl\", \"w\") as f:\n        for example in training:\n            f.write(json.dumps(example) + \"\\n\")\n    return len(training)\n```\n\n## Path C: Synthetic Bootstrapping\n\nGenerate training data from scratch when you have \u003C 1000 real examples. Use as a starting point, then replace with production data as it accumulates.\n\n### Seed Prompt Strategy\n\n```python\nimport openai\n\nclient = openai.OpenAI()\n\ndef generate_synthetic_examples(task_description, seed_examples, n=500, model=\"gpt-4o\"):\n    \"\"\"Generate diverse synthetic training examples from seed examples.\"\"\"\n\n    meta_prompt = f\"\"\"You are generating training data for an LLM that will be fine-tuned for:\n{task_description}\n\nHere are {len(seed_examples)} real examples of the desired behavior:\n{json.dumps(seed_examples[:5], indent=2)}\n\nGenerate a NEW, diverse example. The input should cover a different scenario than\nthe seeds. The output should match the quality and style of the examples above.\n\nReturn JSON: {{\"input\": \"...\", \"output\": \"...\"}}\"\"\"\n\n    examples = []\n    for i in range(n):\n        response = client.chat.completions.create(\n            model=model,\n            messages=[{\"role\": \"user\", \"content\": meta_prompt}],\n            response_format={\"type\": \"json_object\"},\n            temperature=0.9,  # High temp for diversity\n        )\n        example = json.loads(response.choices[0].message.content)\n        examples.append({\n            \"messages\": [\n                {\"role\": \"user\", \"content\": example[\"input\"]},\n                {\"role\": \"assistant\", \"content\": example[\"output\"]}\n            ]\n        })\n    return examples\n```\n\n### Diversity Strategies\n- Vary temperature (0.7-1.0) across generation batches\n- Use different frontier models (GPT-4o, Claude, Gemini) to reduce model-specific bias\n- Seed with representative prompts from different categories\u002Fdifficulty levels\n- Include edge cases and adversarial examples explicitly in seed prompts\n\n## Data Quality Validation\n\n### Schema Validation\n\n```python\ndef validate_jsonl(filepath):\n    \"\"\"Validate JSONL training file format.\"\"\"\n    errors = []\n    valid = 0\n    with open(filepath) as f:\n        for i, line in enumerate(f, 1):\n            try:\n                obj = json.loads(line)\n            except json.JSONDecodeError as e:\n                errors.append(f\"Line {i}: Invalid JSON — {e}\")\n                continue\n\n            if \"messages\" not in obj:\n                errors.append(f\"Line {i}: Missing 'messages' key\")\n                continue\n\n            msgs = obj[\"messages\"]\n            if not isinstance(msgs, list) or len(msgs) \u003C 2:\n                errors.append(f\"Line {i}: 'messages' must be a list with >= 2 entries\")\n                continue\n\n            # Check roles\n            has_user = any(m.get(\"role\") == \"user\" for m in msgs)\n            has_assistant = any(m.get(\"role\") == \"assistant\" for m in msgs)\n            if not has_user or not has_assistant:\n                errors.append(f\"Line {i}: Must have at least one user and one assistant message\")\n                continue\n\n            for j, msg in enumerate(msgs):\n                if \"role\" not in msg or \"content\" not in msg:\n                    errors.append(f\"Line {i}, message {j}: Missing 'role' or 'content'\")\n                elif msg[\"role\"] not in (\"system\", \"user\", \"assistant\"):\n                    errors.append(f\"Line {i}, message {j}: Invalid role '{msg['role']}'\")\n\n            valid += 1\n\n    return {\"valid\": valid, \"errors\": errors, \"total\": valid + len(errors)}\n```\n\n### Deduplication (MinHash)\n\n```python\nfrom datasketch import MinHash, MinHashLSH\n\ndef deduplicate_dataset(examples, threshold=0.8):\n    \"\"\"Remove near-duplicate examples using MinHash LSH.\"\"\"\n    lsh = MinHashLSH(threshold=threshold, num_perm=128)\n    unique = []\n\n    for i, ex in enumerate(examples):\n        # Hash the assistant's response (training target)\n        text = ex[\"messages\"][-1][\"content\"]\n        m = MinHash(num_perm=128)\n        for word in text.lower().split():\n            m.update(word.encode(\"utf-8\"))\n\n        key = f\"doc-{i}\"\n        if not lsh.query(m):\n            lsh.insert(key, m)\n            unique.append(ex)\n\n    removed = len(examples) - len(unique)\n    print(f\"Removed {removed} duplicates ({removed\u002Flen(examples)*100:.1f}%)\")\n    return unique\n```\n\n### Diversity Metrics\n\n```python\nfrom collections import Counter\n\ndef distinct_n(texts, n=2):\n    \"\"\"Calculate distinct-n metric (ratio of unique n-grams to total n-grams).\"\"\"\n    total_ngrams = Counter()\n    for text in texts:\n        words = text.lower().split()\n        ngrams = [tuple(words[i:i+n]) for i in range(len(words)-n+1)]\n        total_ngrams.update(ngrams)\n    if sum(total_ngrams.values()) == 0:\n        return 0\n    return len(total_ngrams) \u002F sum(total_ngrams.values())\n\ndef dataset_diversity_report(examples):\n    \"\"\"Generate diversity metrics for a training dataset.\"\"\"\n    responses = [ex[\"messages\"][-1][\"content\"] for ex in examples]\n    inputs = [m[\"content\"] for ex in examples for m in ex[\"messages\"] if m[\"role\"] == \"user\"]\n\n    report = {\n        \"total_examples\": len(examples),\n        \"avg_response_length\": sum(len(r.split()) for r in responses) \u002F len(responses),\n        \"avg_input_length\": sum(len(i.split()) for i in inputs) \u002F len(inputs),\n        \"distinct_1\": distinct_n(responses, 1),\n        \"distinct_2\": distinct_n(responses, 2),\n        \"distinct_3\": distinct_n(responses, 3),\n    }\n    return report\n```\n\n## Train\u002FEval Split\n\n```python\nimport random\n\ndef split_dataset(examples, eval_ratio=0.1, production_indices=None):\n    \"\"\"Split dataset into train\u002Feval, keeping production data in eval for ground truth.\n\n    Args:\n        examples: List of training examples\n        eval_ratio: Fraction of data for evaluation (default 10%)\n        production_indices: Indices of real production examples (always go to eval)\n    \"\"\"\n    production_indices = set(production_indices or [])\n    synthetic = [ex for i, ex in enumerate(examples) if i not in production_indices]\n    production = [ex for i, ex in enumerate(examples) if i in production_indices]\n\n    # Production data goes to eval (ground truth)\n    eval_set = list(production)\n\n    # Fill remaining eval budget from synthetic\n    remaining_eval = max(0, int(len(examples) * eval_ratio) - len(eval_set))\n    random.shuffle(synthetic)\n    eval_set.extend(synthetic[:remaining_eval])\n    train_set = synthetic[remaining_eval:]\n\n    print(f\"Train: {len(train_set)}, Eval: {len(eval_set)} \"\n          f\"({len(production)} production + {len(eval_set)-len(production)} synthetic)\")\n    return train_set, eval_set\n```\n\n## Common Issues\n\n| Issue | Cause | Fix |\n|-------|-------|-----|\n| `JSONDecodeError` | Trailing commas or malformed JSON | Run `validate_jsonl()` and fix flagged lines |\n| Tokenizer mismatch | Data tokenized for wrong model | Always use target model's tokenizer for length checks |\n| Training loss doesn't decrease | Data too noisy or contradictory | Filter low-quality examples, check for duplicates |\n| Model repeats training data | Overfitting on small dataset | Add more diverse examples, reduce epochs |\n| Data leakage | Eval examples appear in training | Use `split_dataset()` with `production_indices` |\n| Encoding errors | Non-UTF-8 characters | `text.encode('utf-8', errors='replace').decode('utf-8')` |\n| Examples too long | Exceeds model context | Truncate or split long conversations, check tokenizer limits |\n\n## Quick Start Checklist\n\n1. **Identify data source**: Production logs (A), frontier distillation (B), or synthetic (C)\n2. **Format to JSONL**: Standard chat format with messages array\n3. **Validate**: Run `validate_jsonl()` on the output file\n4. **Deduplicate**: Run MinHash dedup with 0.8 threshold\n5. **Check diversity**: Run `dataset_diversity_report()`, aim for distinct-2 > 0.5\n6. **Split**: 90\u002F10 train\u002Feval, production data in eval set\n7. **Count tokens**: Verify no examples exceed model's context window\n8. **Proceed to training**: Load `tinker` or `unsloth` skill for next step\n",{"name":5,"tags":50,"author":58,"license":15,"version":59,"category":60,"description":7,"dependencies":61},[26,51,52,53,54,55,56,57],"Data Pipeline","Fine-Tuning","Distillation","Synthetic Data","Production Data","JSONL","Data Quality","Synthetic Sciences","1.0.0","ml-training",[62,63,43],"datasets","transformers","main","4aa39efd1cdab0b4657ecee16c92a92011a5b10f","openscience\u002Fbackend\u002Fcli\u002Fskills\u002Fml-training\u002Ftraining-data-pipeline","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub","https:\u002F\u002Fgithub.com\u002Ffadrienne","2026-10-28 00:43:09",{"source":8,"skillName":5,"commands":71,"prompt":81,"zipAvailable":82},[72,75,78],{"tool":73,"command":74},"npx","npx skills add fadrienne\u002FMyGitHub --skill training-data-pipeline",{"tool":76,"command":77},"bunx","bunx skills add fadrienne\u002FMyGitHub --skill training-data-pipeline",{"tool":79,"command":80},"pnpm","pnpm dlx skills add fadrienne\u002FMyGitHub --skill training-data-pipeline","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fopenscience\u002Fbackend\u002Fcli\u002Fskills\u002Fml-training\u002Ftraining-data-pipeline\u002FSKILL.md",true,[84,90,94],{"path":85,"resourceType":86,"resourceTypeDesc":87,"sizeBytes":88,"sourceHtmlUrl":89},"references\u002Fdata-quality.md","REFERENCE","参考文档",4194,"https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fopenscience\u002Fbackend\u002Fcli\u002Fskills\u002Fml-training\u002Ftraining-data-pipeline\u002Freferences\u002Fdata-quality.md",{"path":91,"resourceType":86,"resourceTypeDesc":87,"sizeBytes":92,"sourceHtmlUrl":93},"references\u002Ffrontier-distillation.md",4450,"https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fopenscience\u002Fbackend\u002Fcli\u002Fskills\u002Fml-training\u002Ftraining-data-pipeline\u002Freferences\u002Ffrontier-distillation.md",{"path":95,"resourceType":86,"resourceTypeDesc":87,"sizeBytes":96,"sourceHtmlUrl":97},"references\u002Fproduction-data-formatting.md",3857,"https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fopenscience\u002Fbackend\u002Fcli\u002Fskills\u002Fml-training\u002Ftraining-data-pipeline\u002Freferences\u002Fproduction-data-formatting.md",[99,130,154,186,212,239],{"skillNo":100,"name":101,"nameZh":6,"description":102,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":103,"categories":104,"keywords":110,"publishedAt":45},"01M7167B28PK20HP7R7ZED2DY0","statistical-power","Sample size calculation, power analysis, and effect size conversion for scientific studies. Use BEFORE data collection (a priori) to determine required N, or after (post-hoc) to compute achieved power. Covers parametric tests (t-test, ANOVA, chi-square, proportions, correlation, regression), simulation-based power for complex\u002Fclustered\u002Fnested designs, and power curves. Use this skill when someone asks 'how many samples do I need', 'what is my power', 'what effect size can I detect', or any variant. Pairs with \u002Fexperimental-design (choose design first, then compute power) and \u002Fstatistical-analysis (analyze data after collection).","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fscientific-agent-skills\u002Fskills\u002Fstatistical-power\u002FSKILL.md",[105],{"categoryNo":106,"categoryName":107,"taxonomyType":108,"taxonomyTypeDesc":109},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[111,112,113,114,115,116,117,118,42,119,120,121,122,123,124,125,126,127,128,129],"power","statistical power","effect","scipy","regression","choose","design","uv","Bash","collection","analysis","calculation","ANOVA","conversion","analyze","t-test","chi-square","simulation","sample",{"skillNo":131,"name":132,"nameZh":6,"description":133,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":134,"categories":135,"keywords":139,"publishedAt":45},"01M71677EBCM8EPCE9BZCK8505","statistical-analysis","Guided statistical analysis with test selection and reporting. Use when you need help choosing appropriate tests for your data, assumption checking, power analysis, and APA-formatted results. Best for academic research reporting, test selection guidance. For implementing specific models programmatically use statsmodels.","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fscientific-agent-skills\u002Fskills\u002Fstatistical-analysis\u002FSKILL.md",[136],{"categoryNo":137,"categoryName":138,"taxonomyType":108,"taxonomyTypeDesc":109},"01JBSEEDCAT000000000000008","测试与质量",[121,140,111,141,142,143,42,144,145,28,41,146,36,147,148,149,150,151,152,153],"statistical analysis","academic research","implementing","selection","you","reporting","tests","with","test","research","results","statsmodels","models","help",{"skillNo":155,"name":156,"nameZh":6,"description":157,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":158,"categories":159,"keywords":167,"publishedAt":45},"01M71673R93XAHFH81DBNVNG7H","scientific-writing","Core skill for the deep research and writing tool. Write scientific manuscripts in full paragraphs (never bullet points). Use two-stage process with (1) section outlines with key points using research-lookup then (2) convert to flowing prose. IMRAD structure, citations (APA\u002FAMA\u002FVancouver), figures\u002Ftables, reporting guidelines (CONSORT\u002FSTROBE\u002FPRISMA), for research papers and journal submissions.","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fscientific-agent-skills\u002Fskills\u002Fscientific-writing\u002FSKILL.md",[160,163,164],{"categoryNo":161,"categoryName":162,"taxonomyType":108,"taxonomyTypeDesc":109},"01JBSEEDCAT000000000000018","写作与文案",{"categoryNo":106,"categoryName":107,"taxonomyType":108,"taxonomyTypeDesc":109},{"categoryNo":165,"categoryName":166,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000036","内容创作者",[168,169,170,171,172,173,174,175,176,177,119,178,179,180,181,182,183,184,145,185],"writing","scientific","scientific writing","Citations","deep","deep research","never","papers","journal","false","api","prisma","structure","openrouter","Outlines","core skill","read","features",{"skillNo":187,"name":188,"nameZh":6,"description":189,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":190,"categories":191,"keywords":196,"publishedAt":45},"01M7166ZPKXBF4EBPHVTJPQBB1","scientific-visualization","Meta-skill for publication-ready figures. Use when creating journal submission figures requiring multi-panel layouts, significance annotations, error bars, colorblind-safe palettes, and specific journal formatting (Nature, Science, Cell). Orchestrates matplotlib\u002Fseaborn\u002Fplotly with publication styles. For quick exploration use seaborn or plotly directly.","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fscientific-agent-skills\u002Fskills\u002Fscientific-visualization\u002FSKILL.md",[192,195],{"categoryNo":193,"categoryName":194,"taxonomyType":108,"taxonomyTypeDesc":109},"01JBSEEDCAT000000000000013","分析与可视化",{"categoryNo":106,"categoryName":107,"taxonomyType":108,"taxonomyTypeDesc":109},[197,169,198,199,200,201,202,203,204,176,42,205,206,207,37,208,209,210,41,211],"visualization","scientific visualization","error","ready","quick","exploration","meta-skill","Meta","safe","plotly","annotations","Publication","figures","panel","submission",{"skillNo":213,"name":214,"nameZh":6,"description":215,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":216,"categories":217,"keywords":224,"publishedAt":45},"01M7166VXJWQSSC3RQZP60WBGR","research-code","Debug, profile, review, and restructure Python research and analysis code. Use when analysis scripts crash, produce wrong results, run too slowly, or need a correctness review. Covers: reading tracebacks and diagnosing root causes, hypothesis-driven debugging, reviewing code for common statistical\u002Fnumerical errors, profiling data pipelines, improving reproducibility (seeds, environments, determinism), and structuring analysis scripts for sharing. Distinct from \u002Fexploratory-data-analysis (which understands data) and \u002Fstatistical-analysis (which chooses and runs tests) — this skill focuses on the code itself.","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fscientific-agent-skills\u002Fskills\u002Fresearch-code\u002FSKILL.md",[218,221],{"categoryNo":219,"categoryName":220,"taxonomyType":108,"taxonomyTypeDesc":109},"01JBSEEDCAT000000000000021","资料检索与研究",{"categoryNo":222,"categoryName":223,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000030","开发工程师",[225,226,149,227,228,229,230,118,231,42,119,121,232,233,234,235,236,237,184,238],"code","research code","which","memory","data-analysis","errors","debugging","common","review","data pipelines","run","scripts","line","write",{"skillNo":240,"name":241,"nameZh":6,"description":242,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":243,"licenseFileUrl":6,"sourceHtmlUrl":244,"categories":245,"keywords":252,"publishedAt":45},"01M7166CF5HSR7SVZYZFWTDX55","markdown-mermaid-writing","Comprehensive markdown and Mermaid diagram writing skill. Use when creating any scientific document, report, analysis, or visualization. Establishes text-based diagrams as the default documentation standard with full style guides (markdown + mermaid), 24 diagram type references, and 9 document templates.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fscientific-agent-skills\u002Fskills\u002Fmarkdown-mermaid-writing\u002FSKILL.md",[246,249,250,251],{"categoryNo":247,"categoryName":248,"taxonomyType":108,"taxonomyTypeDesc":109},"01JBSEEDCAT000000000000002","文档处理",{"categoryNo":161,"categoryName":162,"taxonomyType":108,"taxonomyTypeDesc":109},{"categoryNo":106,"categoryName":107,"taxonomyType":108,"taxonomyTypeDesc":109},{"categoryNo":165,"categoryName":166,"taxonomyType":21,"taxonomyTypeDesc":22},[168,253,254,255,256,257,197,258,259,42,119,121,260,261,262,263,264,265,169,184],"markdown mermaid writing","mermaid","markdown","community","diagrams","project","default","team","guides","text","documentation","references","diagram"]