[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6QESA2ET0QP5NVTT04DZ8YB":3,"skill-same:01M6QESA2ET0QP5NVTT04DZ8YB":97},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":50,"commitSha":51,"skillDirectory":52,"forksCount":14,"repositoryHtmlUrl":53,"repositoryHomepageUrl":54,"ownerHtmlUrl":55,"ownerBlogUrl":56,"ownerDescription":57,"updatedAt":58,"install":59,"resources":72},"01M6QESA2ET0QP5NVTT04DZ8YB","grpo-rl-training",null,"Guides GRPO (Group Relative Policy Optimization) fine-tuning of language models with the TRL library, including GRPOTrainer configuration, composing multiple reward functions (correctness, format, length, style), dataset prep in chat format, Unsloth setup, LoRA merging, and monitoring reward, reward_std and KL. Use when training a model to follow a strict output format such as XML or JSON. Use when teaching verifiable tasks like math or code with objective correctness rewards. Use when improving chain-of-thought reasoning with custom reward functions. Use when debugging flat rewards, mode collapse, or OOM in GRPO runs. Do not use for plain supervised fine-tuning (use SFT) or when high-quality preference pairs exist (use DPO or PPO).","Tashima-Tarsh\u002FDisha6.6","Tashima-Tarsh","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F269906498?v=4","01M6PBH98X03XERH0HFT4PFB1P","NITISH KUMAR","01M6PBH9DA0M6KM3Z12P848F0K",0,"MIT","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fkal-grpo-rl-training\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"training","GRPO","grpo rl training","rl","custom","policy","for","mode","guides","structured output","as","do","use","learning","PPO","RLHF","Post-Training","Fine-Tuning","monitoring","library","2026-11-10 10:12:30",false,"---\nname: grpo-rl-training\ndescription: Guides GRPO (Group Relative Policy Optimization) fine-tuning of language models with the TRL library, including GRPOTrainer configuration, composing multiple reward functions (correctness, format, length, style), dataset prep in chat format, Unsloth setup, LoRA merging, and monitoring reward, reward_std and KL. Use when training a model to follow a strict output format such as XML or JSON. Use when teaching verifiable tasks like math or code with objective correctness rewards. Use when improving chain-of-thought reasoning with custom reward functions. Use when debugging flat rewards, mode collapse, or OOM in GRPO runs. Do not use for plain supervised fine-tuning (use SFT) or when high-quality preference pairs exist (use DPO or PPO).\nlicense: MIT\nmetadata:\n  version: 1.0.0\n  category: ml-training\n  maintainer: Kalaris Labs\n  tags: Post-Training, Reinforcement Learning, GRPO, TRL, RLHF, Reward Modeling, Reasoning, DPO, PPO, Structured Output\n  dependencies: transformers>=4.47.0, trl>=0.14.0, datasets>=3.2.0, peft>=0.14.0, torch\n---\n\n# GRPO\u002FRL Training with TRL\n\nExpert-level guidance for implementing Group Relative Policy Optimization (GRPO) using the Transformer Reinforcement Learning (TRL) library. This skill provides battle-tested patterns, critical insights, and production-ready workflows for fine-tuning language models with custom reward functions.\n\n## When to Use This Skill\n\nUse GRPO training when you need to:\n- **Enforce specific output formats** (e.g., XML tags, JSON, structured reasoning)\n- **Teach verifiable tasks** with objective correctness metrics (math, coding, fact-checking)\n- **Improve reasoning capabilities** by rewarding chain-of-thought patterns\n- **Align models to domain-specific behaviors** without labeled preference data\n- **Optimize for multiple objectives** simultaneously (format + correctness + style)\n\n**Do NOT use GRPO for:**\n- Simple supervised fine-tuning tasks (use SFT instead)\n- Tasks without clear reward signals\n- When you already have high-quality preference pairs (use DPO\u002FPPO instead)\n\n---\n\n## Core Concepts\n\n### 1. GRPO Algorithm Fundamentals\n\n**Key Mechanism:**\n- Generates **multiple completions** for each prompt (group size: 4-16)\n- Compares completions within each group using reward functions\n- Updates policy to favor higher-rewarded responses relative to the group\n\n**Critical Difference from PPO:**\n- No separate reward model needed\n- More sample-efficient (learns from within-group comparisons)\n- Simpler to implement and debug\n\n**Mathematical Intuition:**\n```\nFor each prompt p:\n  1. Generate N completions: {c₁, c₂, ..., cₙ}\n  2. Compute rewards: {r₁, r₂, ..., rₙ}\n  3. Learn to increase probability of high-reward completions\n     relative to low-reward ones in the same group\n```\n\n### 2. Reward Function Design Philosophy\n\n**Golden Rules:**\n1. **Compose multiple reward functions** - Each handles one aspect (format, correctness, style)\n2. **Scale rewards appropriately** - Higher weight = stronger signal\n3. **Use incremental rewards** - Partial credit for partial compliance\n4. **Test rewards independently** - Debug each reward function in isolation\n\n**Reward Function Types:**\n\n| Type | Use Case | Example Weight |\n|------|----------|----------------|\n| **Correctness** | Verifiable tasks (math, code) | 2.0 (highest) |\n| **Format** | Strict structure enforcement | 0.5-1.0 |\n| **Length** | Encourage verbosity\u002Fconciseness | 0.1-0.5 |\n| **Style** | Penalize unwanted patterns | -0.5 to 0.5 |\n\n---\n\n## Implementation Workflow\n\n### Step 1: Dataset Preparation\n\n**Critical Requirements:**\n- Prompts in chat format (list of dicts with 'role' and 'content')\n- Include system prompts to set expectations\n- For verifiable tasks, include ground truth answers as additional columns\n\n**Example Structure:**\n```python\nfrom datasets import load_dataset, Dataset\n\nSYSTEM_PROMPT = \"\"\"\nRespond in the following format:\n\u003Creasoning>\n[Your step-by-step thinking]\n\u003C\u002Freasoning>\n\u003Canswer>\n[Final answer]\n\u003C\u002Fanswer>\n\"\"\"\n\ndef prepare_dataset(raw_data):\n    \"\"\"\n    Transform raw data into GRPO-compatible format.\n\n    Returns: Dataset with columns:\n    - 'prompt': List[Dict] with role\u002Fcontent (system + user messages)\n    - 'answer': str (ground truth, optional but recommended)\n    \"\"\"\n    return raw_data.map(lambda x: {\n        'prompt': [\n            {'role': 'system', 'content': SYSTEM_PROMPT},\n            {'role': 'user', 'content': x['question']}\n        ],\n        'answer': extract_answer(x['raw_answer'])\n    })\n```\n\n**Pro Tips:**\n- Use one-shot or few-shot examples in system prompt for complex formats\n- Keep prompts concise (max_prompt_length: 256-512 tokens)\n- Validate data quality before training (garbage in = garbage out)\n\n### Step 2: Reward Function Implementation\n\n**Template Structure:**\n```python\ndef reward_function_name(\n    prompts,        # List[List[Dict]]: Original prompts\n    completions,    # List[List[Dict]]: Model generations\n    answer=None,    # Optional: Ground truth from dataset\n    **kwargs        # Additional dataset columns\n) -> list[float]:\n    \"\"\"\n    Evaluate completions and return rewards.\n\n    Returns: List of floats (one per completion)\n    \"\"\"\n    # Extract completion text\n    responses = [comp[0]['content'] for comp in completions]\n\n    # Compute rewards\n    rewards = []\n    for response in responses:\n        score = compute_score(response)\n        rewards.append(score)\n\n    return rewards\n```\n\n**Example 1: Correctness Reward (Math\u002FCoding)**\n```python\ndef correctness_reward(prompts, completions, answer, **kwargs):\n    \"\"\"Reward correct answers with high score.\"\"\"\n    responses = [comp[0]['content'] for comp in completions]\n    extracted = [extract_final_answer(r) for r in responses]\n    return [2.0 if ans == gt else 0.0\n            for ans, gt in zip(extracted, answer)]\n```\n\n**Example 2: Format Reward (Structured Output)**\n```python\nimport re\n\ndef format_reward(completions, **kwargs):\n    \"\"\"Reward XML-like structured format.\"\"\"\n    pattern = r'\u003Creasoning>.*?\u003C\u002Freasoning>\\s*\u003Canswer>.*?\u003C\u002Fanswer>'\n    responses = [comp[0]['content'] for comp in completions]\n    return [1.0 if re.search(pattern, r, re.DOTALL) else 0.0\n            for r in responses]\n```\n\n**Example 3: Incremental Format Reward (Partial Credit)**\n```python\ndef incremental_format_reward(completions, **kwargs):\n    \"\"\"Award partial credit for format compliance.\"\"\"\n    responses = [comp[0]['content'] for comp in completions]\n    rewards = []\n\n    for r in responses:\n        score = 0.0\n        if '\u003Creasoning>' in r:\n            score += 0.25\n        if '\u003C\u002Freasoning>' in r:\n            score += 0.25\n        if '\u003Canswer>' in r:\n            score += 0.25\n        if '\u003C\u002Fanswer>' in r:\n            score += 0.25\n        # Penalize extra text after closing tag\n        if r.count('\u003C\u002Fanswer>') == 1:\n            extra_text = r.split('\u003C\u002Fanswer>')[-1].strip()\n            score -= len(extra_text) * 0.001\n        rewards.append(score)\n\n    return rewards\n```\n\n**Critical Insight:**\nCombine 3-5 reward functions for robust training. Order matters less than diversity of signals.\n\n### Step 3: Training Configuration\n\n**Memory-Optimized Config (Small GPU)**\n```python\nfrom trl import GRPOConfig\n\ntraining_args = GRPOConfig(\n    output_dir=\"outputs\u002Fgrpo-model\",\n\n    # Learning rate\n    learning_rate=5e-6,          # Lower = more stable\n    adam_beta1=0.9,\n    adam_beta2=0.99,\n    weight_decay=0.1,\n    warmup_ratio=0.1,\n    lr_scheduler_type='cosine',\n\n    # Batch settings\n    per_device_train_batch_size=1,\n    gradient_accumulation_steps=4,  # Effective batch = 4\n\n    # GRPO-specific\n    num_generations=8,            # Group size: 8-16 recommended\n    max_prompt_length=256,\n    max_completion_length=512,\n\n    # Training duration\n    num_train_epochs=1,\n    max_steps=None,               # Or set fixed steps (e.g., 500)\n\n    # Optimization\n    bf16=True,                    # Faster on A100\u002FH100\n    optim=\"adamw_8bit\",          # Memory-efficient optimizer\n    max_grad_norm=0.1,\n\n    # Logging\n    logging_steps=1,\n    save_steps=100,\n    report_to=\"wandb\",            # Or \"none\" for no logging\n)\n```\n\n**High-Performance Config (Large GPU)**\n```python\ntraining_args = GRPOConfig(\n    output_dir=\"outputs\u002Fgrpo-model\",\n    learning_rate=1e-5,\n    per_device_train_batch_size=4,\n    gradient_accumulation_steps=2,\n    num_generations=16,           # Larger groups = better signal\n    max_prompt_length=512,\n    max_completion_length=1024,\n    num_train_epochs=1,\n    bf16=True,\n    use_vllm=True,                # Fast generation with vLLM\n    logging_steps=10,\n)\n```\n\n**Critical Hyperparameters:**\n\n| Parameter | Impact | Tuning Advice |\n|-----------|--------|---------------|\n| `num_generations` | Group size for comparison | Start with 8, increase to 16 if GPU allows |\n| `learning_rate` | Convergence speed\u002Fstability | 5e-6 (safe), 1e-5 (faster, riskier) |\n| `max_completion_length` | Output verbosity | Match your task (512 for reasoning, 256 for short answers) |\n| `gradient_accumulation_steps` | Effective batch size | Increase if GPU memory limited |\n\n### Step 4: Model Setup and Training\n\n**Standard Setup (Transformers)**\n```python\nimport torch\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom peft import LoraConfig\nfrom trl import GRPOTrainer\n\n# Load model\nmodel_name = \"Qwen\u002FQwen2.5-1.5B-Instruct\"\nmodel = AutoModelForCausalLM.from_pretrained(\n    model_name,\n    torch_dtype=torch.bfloat16,\n    attn_implementation=\"flash_attention_2\",  # 2-3x faster\n    device_map=\"auto\"\n)\n\ntokenizer = AutoTokenizer.from_pretrained(model_name)\ntokenizer.pad_token = tokenizer.eos_token\n\n# Optional: LoRA for parameter-efficient training\npeft_config = LoraConfig(\n    r=16,                         # Rank (higher = more capacity)\n    lora_alpha=32,               # Scaling factor (typically 2*r)\n    target_modules=[\n        \"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\",\n        \"gate_proj\", \"up_proj\", \"down_proj\"\n    ],\n    task_type=\"CAUSAL_LM\",\n    lora_dropout=0.05,\n)\n\n# Initialize trainer\ntrainer = GRPOTrainer(\n    model=model,\n    processing_class=tokenizer,\n    reward_funcs=[\n        incremental_format_reward,\n        format_reward,\n        correctness_reward,\n    ],\n    args=training_args,\n    train_dataset=dataset,\n    peft_config=peft_config,      # Remove for full fine-tuning\n)\n\n# Train\ntrainer.train()\n\n# Save\ntrainer.save_model(\"final_model\")\n```\n\n**Unsloth Setup (2-3x Faster)**\n```python\nfrom unsloth import FastLanguageModel\n\nmodel, tokenizer = FastLanguageModel.from_pretrained(\n    model_name=\"google\u002Fgemma-3-1b-it\",\n    max_seq_length=1024,\n    load_in_4bit=True,\n    fast_inference=True,\n    max_lora_rank=32,\n)\n\nmodel = FastLanguageModel.get_peft_model(\n    model,\n    r=32,\n    target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\",\n                    \"gate_proj\", \"up_proj\", \"down_proj\"],\n    lora_alpha=32,\n    use_gradient_checkpointing=\"unsloth\",\n)\n\n# Rest is identical to standard setup\ntrainer = GRPOTrainer(model=model, ...)\ntrainer.train()\n```\n\n---\n\n## Critical Training Insights\n\nDetails, code examples and parameter tables: [references\u002Fcritical-training-insights.md](references\u002Fcritical-training-insights.md). Read it when this step applies.\n\n## Advanced Patterns\n\nDetails, code examples and parameter tables: [references\u002Fadvanced-patterns.md](references\u002Fadvanced-patterns.md). Read it when this step applies.\n\n## Deployment and Inference\n\n### Save and Merge LoRA\n```python\n# Merge LoRA adapters into base model\nif hasattr(trainer.model, 'merge_and_unload'):\n    merged_model = trainer.model.merge_and_unload()\n    merged_model.save_pretrained(\"production_model\")\n    tokenizer.save_pretrained(\"production_model\")\n```\n\n### Inference Example\n```python\nfrom transformers import pipeline\n\ngenerator = pipeline(\n    \"text-generation\",\n    model=\"production_model\",\n    tokenizer=tokenizer\n)\n\nresult = generator(\n    [\n        {'role': 'system', 'content': SYSTEM_PROMPT},\n        {'role': 'user', 'content': \"What is 15 + 27?\"}\n    ],\n    max_new_tokens=256,\n    do_sample=True,\n    temperature=0.7,\n    top_p=0.9\n)\nprint(result[0]['generated_text'])\n```\n\n---\n\n## Best Practices Checklist\n\n**Before Training:**\n- [ ] Validate dataset format (prompts as List[Dict])\n- [ ] Test reward functions on sample data\n- [ ] Calculate expected max_prompt_length from data\n- [ ] Choose appropriate num_generations based on GPU memory\n- [ ] Set up logging (wandb recommended)\n\n**During Training:**\n- [ ] Monitor reward progression (should increase)\n- [ ] Check reward_std (should stay > 0.1)\n- [ ] Watch for OOM errors (reduce batch size if needed)\n- [ ] Sample generations every 50-100 steps\n- [ ] Validate format compliance on holdout set\n\n**After Training:**\n- [ ] Merge LoRA weights if using PEFT\n- [ ] Test on diverse prompts\n- [ ] Compare to baseline model\n- [ ] Document reward weights and hyperparameters\n- [ ] Save reproducibility config\n\n---\n\n## Troubleshooting Guide\n\n### Debugging Workflow\n1. **Isolate reward functions** - Test each independently\n2. **Check data distribution** - Ensure diversity in prompts\n3. **Reduce complexity** - Start with single reward, add gradually\n4. **Monitor generations** - Print samples every N steps\n5. **Validate extraction logic** - Ensure answer parsing works\n\n### Quick Fixes\n```python\n# Debug reward function\ndef debug_reward(completions, **kwargs):\n    responses = [comp[0]['content'] for comp in completions]\n    for i, r in enumerate(responses[:2]):  # Print first 2\n        print(f\"Response {i}: {r[:200]}...\")\n    return [1.0] * len(responses)  # Dummy rewards\n\n# Test without training\ntrainer = GRPOTrainer(..., reward_funcs=[debug_reward])\ntrainer.generate_completions(dataset[:1])  # Generate without updating\n```\n\n---\n\n## References and Resources\n\n**Official Documentation:**\n- TRL GRPO Trainer: https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftrl\u002Fgrpo_trainer\n- DeepSeek R1 Paper: https:\u002F\u002Farxiv.org\u002Fabs\u002F2501.12948\n- Unsloth Docs: https:\u002F\u002Fdocs.unsloth.ai\u002F\n\n**Example Repositories:**\n- Open R1 Implementation: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Fopen-r1\n- TRL Examples: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl\u002Ftree\u002Fmain\u002Fexamples\n\n**Recommended Reading:**\n- Progressive Disclosure Pattern for agent instructions\n- Reward shaping in RL (Ng et al.)\n- LoRA paper (Hu et al., 2021)\n\n---\n\n## Usage Instructions for Agents\n\nWhen this skill is loaded:\n\n1. **Read this entire file** before implementing GRPO training\n2. **Start with the simplest reward function** (e.g., length-based) to validate setup\n3. **Use the templates** in `templates\u002F` directory as starting points\n4. **Reference examples** in `examples\u002F` for task-specific implementations\n5. **Follow the workflow** sequentially (don't skip steps)\n6. **Debug incrementally** - add one reward function at a time\n\n**Critical Reminders:**\n- Always use multiple reward functions (3-5 is optimal)\n- Monitor reward metrics, not loss\n- Test reward functions before training\n- Start small (num_generations=4), scale up gradually\n- Save checkpoints frequently (every 100 steps)\n\nThis skill is designed for **expert-level implementation**. Beginners should start with supervised fine-tuning before attempting GRPO.\n\n## Related skills\n\n- `fine-tuning-with-trl`: Fine-tune LLMs using reinforcement learning with TRL - SFT for instruction tuning, DPO for preference alignment, PPO\u002FGRPO for reward optimi…\n- `unsloth`: Provides guidance on fine-tuning large language models with Unsloth, a library for faster, lower-memory training using LoRA and QLoRA, base…\n- `axolotl`: Provides guidance for fine-tuning large language models with Axolotl, covering YAML training configs, LoRA and QLoRA, preference training w…\n","# GRPO\u002FRL Training with TRL\n\nExpert-level guidance for implementing Group Relative Policy Optimization (GRPO) using the Transformer Reinforcement Learning (TRL) library. This skill provides battle-tested patterns, critical insights, and production-ready workflows for fine-tuning language models with custom reward functions.\n\n## When to Use This Skill\n\nUse GRPO training when you need to:\n- **Enforce specific output formats** (e.g., XML tags, JSON, structured reasoning)\n- **Teach verifiable tasks** with objective correctness metrics (math, coding, fact-checking)\n- **Improve reasoning capabilities** by rewarding chain-of-thought patterns\n- **Align models to domain-specific behaviors** without labeled preference data\n- **Optimize for multiple objectives** simultaneously (format + correctness + style)\n\n**Do NOT use GRPO for:**\n- Simple supervised fine-tuning tasks (use SFT instead)\n- Tasks without clear reward signals\n- When you already have high-quality preference pairs (use DPO\u002FPPO instead)\n\n---\n\n## Core Concepts\n\n### 1. GRPO Algorithm Fundamentals\n\n**Key Mechanism:**\n- Generates **multiple completions** for each prompt (group size: 4-16)\n- Compares completions within each group using reward functions\n- Updates policy to favor higher-rewarded responses relative to the group\n\n**Critical Difference from PPO:**\n- No separate reward model needed\n- More sample-efficient (learns from within-group comparisons)\n- Simpler to implement and debug\n\n**Mathematical Intuition:**\n```\nFor each prompt p:\n  1. Generate N completions: {c₁, c₂, ..., cₙ}\n  2. Compute rewards: {r₁, r₂, ..., rₙ}\n  3. Learn to increase probability of high-reward completions\n     relative to low-reward ones in the same group\n```\n\n### 2. Reward Function Design Philosophy\n\n**Golden Rules:**\n1. **Compose multiple reward functions** - Each handles one aspect (format, correctness, style)\n2. **Scale rewards appropriately** - Higher weight = stronger signal\n3. **Use incremental rewards** - Partial credit for partial compliance\n4. **Test rewards independently** - Debug each reward function in isolation\n\n**Reward Function Types:**\n\n| Type | Use Case | Example Weight |\n|------|----------|----------------|\n| **Correctness** | Verifiable tasks (math, code) | 2.0 (highest) |\n| **Format** | Strict structure enforcement | 0.5-1.0 |\n| **Length** | Encourage verbosity\u002Fconciseness | 0.1-0.5 |\n| **Style** | Penalize unwanted patterns | -0.5 to 0.5 |\n\n---\n\n## Implementation Workflow\n\n### Step 1: Dataset Preparation\n\n**Critical Requirements:**\n- Prompts in chat format (list of dicts with 'role' and 'content')\n- Include system prompts to set expectations\n- For verifiable tasks, include ground truth answers as additional columns\n\n**Example Structure:**\n```python\nfrom datasets import load_dataset, Dataset\n\nSYSTEM_PROMPT = \"\"\"\nRespond in the following format:\n\u003Creasoning>\n[Your step-by-step thinking]\n\u003C\u002Freasoning>\n\u003Canswer>\n[Final answer]\n\u003C\u002Fanswer>\n\"\"\"\n\ndef prepare_dataset(raw_data):\n    \"\"\"\n    Transform raw data into GRPO-compatible format.\n\n    Returns: Dataset with columns:\n    - 'prompt': List[Dict] with role\u002Fcontent (system + user messages)\n    - 'answer': str (ground truth, optional but recommended)\n    \"\"\"\n    return raw_data.map(lambda x: {\n        'prompt': [\n            {'role': 'system', 'content': SYSTEM_PROMPT},\n            {'role': 'user', 'content': x['question']}\n        ],\n        'answer': extract_answer(x['raw_answer'])\n    })\n```\n\n**Pro Tips:**\n- Use one-shot or few-shot examples in system prompt for complex formats\n- Keep prompts concise (max_prompt_length: 256-512 tokens)\n- Validate data quality before training (garbage in = garbage out)\n\n### Step 2: Reward Function Implementation\n\n**Template Structure:**\n```python\ndef reward_function_name(\n    prompts,        # List[List[Dict]]: Original prompts\n    completions,    # List[List[Dict]]: Model generations\n    answer=None,    # Optional: Ground truth from dataset\n    **kwargs        # Additional dataset columns\n) -> list[float]:\n    \"\"\"\n    Evaluate completions and return rewards.\n\n    Returns: List of floats (one per completion)\n    \"\"\"\n    # Extract completion text\n    responses = [comp[0]['content'] for comp in completions]\n\n    # Compute rewards\n    rewards = []\n    for response in responses:\n        score = compute_score(response)\n        rewards.append(score)\n\n    return rewards\n```\n\n**Example 1: Correctness Reward (Math\u002FCoding)**\n```python\ndef correctness_reward(prompts, completions, answer, **kwargs):\n    \"\"\"Reward correct answers with high score.\"\"\"\n    responses = [comp[0]['content'] for comp in completions]\n    extracted = [extract_final_answer(r) for r in responses]\n    return [2.0 if ans == gt else 0.0\n            for ans, gt in zip(extracted, answer)]\n```\n\n**Example 2: Format Reward (Structured Output)**\n```python\nimport re\n\ndef format_reward(completions, **kwargs):\n    \"\"\"Reward XML-like structured format.\"\"\"\n    pattern = r'\u003Creasoning>.*?\u003C\u002Freasoning>\\s*\u003Canswer>.*?\u003C\u002Fanswer>'\n    responses = [comp[0]['content'] for comp in completions]\n    return [1.0 if re.search(pattern, r, re.DOTALL) else 0.0\n            for r in responses]\n```\n\n**Example 3: Incremental Format Reward (Partial Credit)**\n```python\ndef incremental_format_reward(completions, **kwargs):\n    \"\"\"Award partial credit for format compliance.\"\"\"\n    responses = [comp[0]['content'] for comp in completions]\n    rewards = []\n\n    for r in responses:\n        score = 0.0\n        if '\u003Creasoning>' in r:\n            score += 0.25\n        if '\u003C\u002Freasoning>' in r:\n            score += 0.25\n        if '\u003Canswer>' in r:\n            score += 0.25\n        if '\u003C\u002Fanswer>' in r:\n            score += 0.25\n        # Penalize extra text after closing tag\n        if r.count('\u003C\u002Fanswer>') == 1:\n            extra_text = r.split('\u003C\u002Fanswer>')[-1].strip()\n            score -= len(extra_text) * 0.001\n        rewards.append(score)\n\n    return rewards\n```\n\n**Critical Insight:**\nCombine 3-5 reward functions for robust training. Order matters less than diversity of signals.\n\n### Step 3: Training Configuration\n\n**Memory-Optimized Config (Small GPU)**\n```python\nfrom trl import GRPOConfig\n\ntraining_args = GRPOConfig(\n    output_dir=\"outputs\u002Fgrpo-model\",\n\n    # Learning rate\n    learning_rate=5e-6,          # Lower = more stable\n    adam_beta1=0.9,\n    adam_beta2=0.99,\n    weight_decay=0.1,\n    warmup_ratio=0.1,\n    lr_scheduler_type='cosine',\n\n    # Batch settings\n    per_device_train_batch_size=1,\n    gradient_accumulation_steps=4,  # Effective batch = 4\n\n    # GRPO-specific\n    num_generations=8,            # Group size: 8-16 recommended\n    max_prompt_length=256,\n    max_completion_length=512,\n\n    # Training duration\n    num_train_epochs=1,\n    max_steps=None,               # Or set fixed steps (e.g., 500)\n\n    # Optimization\n    bf16=True,                    # Faster on A100\u002FH100\n    optim=\"adamw_8bit\",          # Memory-efficient optimizer\n    max_grad_norm=0.1,\n\n    # Logging\n    logging_steps=1,\n    save_steps=100,\n    report_to=\"wandb\",            # Or \"none\" for no logging\n)\n```\n\n**High-Performance Config (Large GPU)**\n```python\ntraining_args = GRPOConfig(\n    output_dir=\"outputs\u002Fgrpo-model\",\n    learning_rate=1e-5,\n    per_device_train_batch_size=4,\n    gradient_accumulation_steps=2,\n    num_generations=16,           # Larger groups = better signal\n    max_prompt_length=512,\n    max_completion_length=1024,\n    num_train_epochs=1,\n    bf16=True,\n    use_vllm=True,                # Fast generation with vLLM\n    logging_steps=10,\n)\n```\n\n**Critical Hyperparameters:**\n\n| Parameter | Impact | Tuning Advice |\n|-----------|--------|---------------|\n| `num_generations` | Group size for comparison | Start with 8, increase to 16 if GPU allows |\n| `learning_rate` | Convergence speed\u002Fstability | 5e-6 (safe), 1e-5 (faster, riskier) |\n| `max_completion_length` | Output verbosity | Match your task (512 for reasoning, 256 for short answers) |\n| `gradient_accumulation_steps` | Effective batch size | Increase if GPU memory limited |\n\n### Step 4: Model Setup and Training\n\n**Standard Setup (Transformers)**\n```python\nimport torch\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom peft import LoraConfig\nfrom trl import GRPOTrainer\n\n# Load model\nmodel_name = \"Qwen\u002FQwen2.5-1.5B-Instruct\"\nmodel = AutoModelForCausalLM.from_pretrained(\n    model_name,\n    torch_dtype=torch.bfloat16,\n    attn_implementation=\"flash_attention_2\",  # 2-3x faster\n    device_map=\"auto\"\n)\n\ntokenizer = AutoTokenizer.from_pretrained(model_name)\ntokenizer.pad_token = tokenizer.eos_token\n\n# Optional: LoRA for parameter-efficient training\npeft_config = LoraConfig(\n    r=16,                         # Rank (higher = more capacity)\n    lora_alpha=32,               # Scaling factor (typically 2*r)\n    target_modules=[\n        \"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\",\n        \"gate_proj\", \"up_proj\", \"down_proj\"\n    ],\n    task_type=\"CAUSAL_LM\",\n    lora_dropout=0.05,\n)\n\n# Initialize trainer\ntrainer = GRPOTrainer(\n    model=model,\n    processing_class=tokenizer,\n    reward_funcs=[\n        incremental_format_reward,\n        format_reward,\n        correctness_reward,\n    ],\n    args=training_args,\n    train_dataset=dataset,\n    peft_config=peft_config,      # Remove for full fine-tuning\n)\n\n# Train\ntrainer.train()\n\n# Save\ntrainer.save_model(\"final_model\")\n```\n\n**Unsloth Setup (2-3x Faster)**\n```python\nfrom unsloth import FastLanguageModel\n\nmodel, tokenizer = FastLanguageModel.from_pretrained(\n    model_name=\"google\u002Fgemma-3-1b-it\",\n    max_seq_length=1024,\n    load_in_4bit=True,\n    fast_inference=True,\n    max_lora_rank=32,\n)\n\nmodel = FastLanguageModel.get_peft_model(\n    model,\n    r=32,\n    target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\",\n                    \"gate_proj\", \"up_proj\", \"down_proj\"],\n    lora_alpha=32,\n    use_gradient_checkpointing=\"unsloth\",\n)\n\n# Rest is identical to standard setup\ntrainer = GRPOTrainer(model=model, ...)\ntrainer.train()\n```\n\n---\n\n## Critical Training Insights\n\nDetails, code examples and parameter tables: [references\u002Fcritical-training-insights.md](references\u002Fcritical-training-insights.md). Read it when this step applies.\n\n## Advanced Patterns\n\nDetails, code examples and parameter tables: [references\u002Fadvanced-patterns.md](references\u002Fadvanced-patterns.md). Read it when this step applies.\n\n## Deployment and Inference\n\n### Save and Merge LoRA\n```python\n# Merge LoRA adapters into base model\nif hasattr(trainer.model, 'merge_and_unload'):\n    merged_model = trainer.model.merge_and_unload()\n    merged_model.save_pretrained(\"production_model\")\n    tokenizer.save_pretrained(\"production_model\")\n```\n\n### Inference Example\n```python\nfrom transformers import pipeline\n\ngenerator = pipeline(\n    \"text-generation\",\n    model=\"production_model\",\n    tokenizer=tokenizer\n)\n\nresult = generator(\n    [\n        {'role': 'system', 'content': SYSTEM_PROMPT},\n        {'role': 'user', 'content': \"What is 15 + 27?\"}\n    ],\n    max_new_tokens=256,\n    do_sample=True,\n    temperature=0.7,\n    top_p=0.9\n)\nprint(result[0]['generated_text'])\n```\n\n---\n\n## Best Practices Checklist\n\n**Before Training:**\n- [ ] Validate dataset format (prompts as List[Dict])\n- [ ] Test reward functions on sample data\n- [ ] Calculate expected max_prompt_length from data\n- [ ] Choose appropriate num_generations based on GPU memory\n- [ ] Set up logging (wandb recommended)\n\n**During Training:**\n- [ ] Monitor reward progression (should increase)\n- [ ] Check reward_std (should stay > 0.1)\n- [ ] Watch for OOM errors (reduce batch size if needed)\n- [ ] Sample generations every 50-100 steps\n- [ ] Validate format compliance on holdout set\n\n**After Training:**\n- [ ] Merge LoRA weights if using PEFT\n- [ ] Test on diverse prompts\n- [ ] Compare to baseline model\n- [ ] Document reward weights and hyperparameters\n- [ ] Save reproducibility config\n\n---\n\n## Troubleshooting Guide\n\n### Debugging Workflow\n1. **Isolate reward functions** - Test each independently\n2. **Check data distribution** - Ensure diversity in prompts\n3. **Reduce complexity** - Start with single reward, add gradually\n4. **Monitor generations** - Print samples every N steps\n5. **Validate extraction logic** - Ensure answer parsing works\n\n### Quick Fixes\n```python\n# Debug reward function\ndef debug_reward(completions, **kwargs):\n    responses = [comp[0]['content'] for comp in completions]\n    for i, r in enumerate(responses[:2]):  # Print first 2\n        print(f\"Response {i}: {r[:200]}...\")\n    return [1.0] * len(responses)  # Dummy rewards\n\n# Test without training\ntrainer = GRPOTrainer(..., reward_funcs=[debug_reward])\ntrainer.generate_completions(dataset[:1])  # Generate without updating\n```\n\n---\n\n## References and Resources\n\n**Official Documentation:**\n- TRL GRPO Trainer: https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftrl\u002Fgrpo_trainer\n- DeepSeek R1 Paper: https:\u002F\u002Farxiv.org\u002Fabs\u002F2501.12948\n- Unsloth Docs: https:\u002F\u002Fdocs.unsloth.ai\u002F\n\n**Example Repositories:**\n- Open R1 Implementation: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Fopen-r1\n- TRL Examples: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl\u002Ftree\u002Fmain\u002Fexamples\n\n**Recommended Reading:**\n- Progressive Disclosure Pattern for agent instructions\n- Reward shaping in RL (Ng et al.)\n- LoRA paper (Hu et al., 2021)\n\n---\n\n## Usage Instructions for Agents\n\nWhen this skill is loaded:\n\n1. **Read this entire file** before implementing GRPO training\n2. **Start with the simplest reward function** (e.g., length-based) to validate setup\n3. **Use the templates** in `templates\u002F` directory as starting points\n4. **Reference examples** in `examples\u002F` for task-specific implementations\n5. **Follow the workflow** sequentially (don't skip steps)\n6. **Debug incrementally** - add one reward function at a time\n\n**Critical Reminders:**\n- Always use multiple reward functions (3-5 is optimal)\n- Monitor reward metrics, not loss\n- Test reward functions before training\n- Start small (num_generations=4), scale up gradually\n- Save checkpoints frequently (every 100 steps)\n\nThis skill is designed for **expert-level implementation**. Beginners should start with supervised fine-tuning before attempting GRPO.\n\n## Related skills\n\n- `fine-tuning-with-trl`: Fine-tune LLMs using reinforcement learning with TRL - SFT for instruction tuning, DPO for preference alignment, PPO\u002FGRPO for reward optimi…\n- `unsloth`: Provides guidance on fine-tuning large language models with Unsloth, a library for faster, lower-memory training using LoRA and QLoRA, base…\n- `axolotl`: Provides guidance for fine-tuning large language models with Axolotl, covering YAML training configs, LoRA and QLoRA, preference training w…\n",{"name":5,"license":15,"metadata":44,"description":7},{"tags":45,"version":46,"category":47,"maintainer":48,"dependencies":49},"Post-Training, Reinforcement Learning, GRPO, TRL, RLHF, Reward Modeling, Reasoning, DPO, PPO, Structured Output","1.0.0","ml-training","Kalaris Labs","transformers>=4.47.0, trl>=0.14.0, datasets>=3.2.0, peft>=0.14.0, torch","main","0da4485785ee599388900be50db1635489d53056",".agents\u002Fskills\u002Fkal-grpo-rl-training","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6","https:\u002F\u002Fdisha6.6.thenitishkr.in\u002F","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh","www.thenitishkr","thenitishkr | AI • Cyber-security • Research\r\n","2026-11-08 03:33:25",{"source":8,"skillName":5,"commands":60,"prompt":70,"zipAvailable":71},[61,64,67],{"tool":62,"command":63},"npx","npx skills add Tashima-Tarsh\u002FDisha6.6 --skill grpo-rl-training",{"tool":65,"command":66},"bunx","bunx skills add Tashima-Tarsh\u002FDisha6.6 --skill grpo-rl-training",{"tool":68,"command":69},"pnpm","pnpm dlx skills add Tashima-Tarsh\u002FDisha6.6 --skill grpo-rl-training","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fkal-grpo-rl-training\u002FSKILL.md",true,[73,79,85,89,93],{"path":74,"resourceType":75,"resourceTypeDesc":76,"sizeBytes":77,"sourceHtmlUrl":78},"examples\u002Freward_functions_library.py","SCRIPT","脚本",11568,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fkal-grpo-rl-training\u002Fexamples\u002Freward_functions_library.py",{"path":80,"resourceType":81,"resourceTypeDesc":82,"sizeBytes":83,"sourceHtmlUrl":84},"README.md","REFERENCE","参考文档",3539,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fkal-grpo-rl-training\u002FREADME.md",{"path":86,"resourceType":81,"resourceTypeDesc":82,"sizeBytes":87,"sourceHtmlUrl":88},"references\u002Fadvanced-patterns.md",1511,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fkal-grpo-rl-training\u002Freferences\u002Fadvanced-patterns.md",{"path":90,"resourceType":81,"resourceTypeDesc":82,"sizeBytes":91,"sourceHtmlUrl":92},"references\u002Fcritical-training-insights.md",1559,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fkal-grpo-rl-training\u002Freferences\u002Fcritical-training-insights.md",{"path":94,"resourceType":75,"resourceTypeDesc":76,"sizeBytes":95,"sourceHtmlUrl":96},"templates\u002Fbasic_grpo_training.py",6122,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fkal-grpo-rl-training\u002Ftemplates\u002Fbasic_grpo_training.py",[98,127,151,178,203,227],{"skillNo":99,"name":100,"nameZh":6,"description":101,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":102,"categories":103,"keywords":109,"publishedAt":39},"01M6QRD07D4TGNDK6VJ4EHF8BV","youtube-full","Use when the user needs YouTube transcripts, video search, channel browsing, playlist extraction, or content monitoring. Trigger phrases: 'get the transcript for', 'search YouTube for', 'what are the latest videos on', 'list this playlist', 'monitor this channel', or any request involving a YouTube URL, video ID, or @handle. Do NOT use for downloading video or audio files, YouTube engagement data (likes, comments), or private\u002Fage-restricted videos.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fyoutube-full\u002FSKILL.md",[104],{"categoryNo":105,"categoryName":106,"taxonomyType":107,"taxonomyTypeDesc":108},"01JBSEEDCAT000000000000016","图像与多媒体","SCENARIO","使用场景",[110,111,112,113,114,37,115,116,117,118,119,30,120,121,122,123,25,124,125,126],"youtube full","youtube","full","transcript","search","data","Video","Audio","list","files","get","age","trigger","URL","browsing","request","youtube transcripts",{"skillNo":128,"name":129,"nameZh":6,"description":130,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":131,"categories":132,"keywords":133,"publishedAt":39},"01M6QRCJ3112VYPA97YG7KGMC2","x-twitter-growth","X\u002FTwitter growth engine for building audience, crafting viral content, and analyzing engagement. Use when the user wants to grow on X\u002FTwitter, write tweets or threads, analyze their X profile, research competitors on X, plan a posting strategy, or optimize engagement. Complements social-content (generic multi-platform) with X-specific depth: algorithm mechanics, thread engineering, reply strategy, profile optimization, and competitive intelligence via web search.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fx-twitter-growth\u002FSKILL.md",[],[134,135,136,137,114,138,139,140,141,142,143,144,145,146,31,147,148,149,150,25],"twitter","x twitter growth","growth","research","content","engineering","plan","web search","competitors","analyze","engine","write","strategy","platform","grow","posting","competitive",{"skillNo":152,"name":153,"nameZh":6,"description":154,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":155,"categories":156,"keywords":160,"publishedAt":39},"01M6QRBN3EHQNJY42RERP7A149","write-a-skill","Create new agent skills with proper structure, progressive disclosure, and bundled resources. Use when user wants to create, write, build, or author a new skill.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fwrite-a-skill\u002FSKILL.md",[157],{"categoryNo":158,"categoryName":159,"taxonomyType":107,"taxonomyTypeDesc":108},"01JBSEEDCAT000000000000023","Skill与智能体开发",[161,145,162,163,164,165,166,167,31,168,169,50,170,171,172,173,174,175,176,177],"write a skill","github","create","https","productivity","matt pocock","direct","example","build","new skill","structure","new agent","resources","with","tree","progressive disclosure","author",{"skillNo":179,"name":180,"nameZh":6,"description":181,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":182,"categories":183,"keywords":184,"publishedAt":39},"01M6QRB6G8P33PNGAF9Y69EJ9T","wrap-up","Close out a launched Claude Managed Agent — recap every primitive the founder now owns, regenerate the single-file overview page, and suggest the next 1-2 upgrades. Use when the user says \"wrap up\", \"close this out\", \"what do I own now\", \"give me the summary\", \"recap the agent\", or when the orchestrator routes phase=wrap-up. primitives_inventory.py tables everything owned (agent, environment, session, memory, outcome, deployment); overview_page.py regenerates a self-contained .\u002Fmy-agent\u002Fagent-overview.html; upgrade_suggester.py ranks the next moves from recorded deferrals plus standing hardening steps. Companion to run-without-you; the last stop before phase=done.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fwrap-up\u002FSKILL.md",[],[185,186,187,188,189,190,191,192,193,31,194,30,195,196,197,198,199,200,201,202],"wrap up","up","wrap","gemini","deployment","run","file","hardening","closeout","stop","upgrade","founder","close","companion","you","tables","directions","phase",{"skillNo":204,"name":205,"nameZh":6,"description":206,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":207,"categories":208,"keywords":210,"publishedAt":39},"01M6QRA5AYC1RBFZ39GTWGS3N8","workflow-builder","Design and write deterministic multi-agent workflow scripts (.js files in .claude\u002Fworkflows\u002F) for Claude Code's Workflow tool. Use when a user wants to build, create, author, scaffold, or run a custom Claude Code workflow, orchestrate sub-agents (fan-out, pipeline, loop, judge-panel), or automate a repeatable multi-step task across fresh-context agents.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fworkflow-builder\u002FSKILL.md",[209],{"categoryNo":158,"categoryName":159,"taxonomyType":107,"taxonomyTypeDesc":108},[211,212,213,214,215,163,190,216,217,218,219,220,145,221,119,222,223,224,225,226],"workflow","workflow builder","builder","design","workflows","code","orchestrate","pipeline","loop","multi-agent","context","across","fresh","scripts","judge","claude code workflow",{"skillNo":228,"name":229,"nameZh":6,"description":230,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":231,"categories":232,"keywords":233,"publishedAt":39},"01M6QR7QMCZESKQA0ERW1E657M","weekly-review","Use when someone wants to run a weekly review, close open loops, audit stalled projects and commitments, get their system back to trusted, restart a lapsed review habit, or says \"\u002Fcs:weekly-review\". Walks David Allen's three-phase loop — GET CLEAR, GET CURRENT, GET CREATIVE — with deterministic scripts that inventory open loops, gate the checklist with named gaps, and score commitment health 0-100.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fweekly-review\u002FSKILL.md",[],[234,235,236,237,238,239,240,241,242,219,243,244,245,246,247,248,120,249,250,202],"review","weekly review","weekly","deterministic scripts","reflect","feeds","inventory","open","team","projects","project-management","scanner","conversation","back","project","sprint","cadence"]