[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6MA3T0Y5AH5Q15N5RFDWZ62":3,"skill-same:01M6MA3T0Y5AH5Q15N5RFDWZ62":93},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":40,"linkOnly":41,"markdownContent":42,"markdownBody":43,"frontMatter":44,"refName":51,"commitSha":52,"skillDirectory":53,"forksCount":54,"repositoryHtmlUrl":55,"repositoryHomepageUrl":56,"ownerHtmlUrl":57,"ownerBlogUrl":58,"ownerDescription":59,"updatedAt":60,"install":61,"resources":74},"01M6MA3T0Y5AH5Q15N5RFDWZ62","fine-tuning-with-trl",null,"Fine-tune LLMs using reinforcement learning with TRL - SFT for instruction tuning, DPO for preference alignment, PPO\u002FGRPO for reward optimization, and reward model training. Use when need RLHF, align model with preferences, or train from human feedback. Works with HuggingFace Transformers.","hashgraph-online\u002Fawesome-codex-plugins","hashgraph-online","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F160371962?v=4","01M6CNB6NRJVNNGN14TR01Q6D3","HOL (Hashgraph Online)","01M6JT1HSKKG9V68KQAQGBA7XP",1575,"MIT","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Ffine-tuning-with-trl\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39],"with","TRL","tuning","fine tuning","fine tuning with trl","when","llms","for","Instruction Tuning","Accelerate","human","align","alignment","feedback","model","learning","Transformers","RLHF","Post-Training","Optimization","training","2026-11-10 10:12:30",false,"---\nname: fine-tuning-with-trl\ndescription: Fine-tune LLMs using reinforcement learning with TRL - SFT for instruction tuning, DPO for preference alignment, PPO\u002FGRPO for reward optimization, and reward model training. Use when need RLHF, align model with preferences, or train from human feedback. Works with HuggingFace Transformers.\nlicense: MIT\nmetadata:\n  version: 1.0.0\n  category: ml-training\n  maintainer: Kalaris Labs\n  tags: Post-Training, TRL, Reinforcement Learning, Fine-Tuning, SFT, DPO, PPO, GRPO, RLHF, Preference Alignment, HuggingFace\n  dependencies: trl, transformers, datasets, peft, accelerate, torch\n---\n\n# TRL - Transformer Reinforcement Learning\n\n## Quick start\n\nTRL provides post-training methods for aligning language models with human preferences.\n\n**Installation**:\n```bash\npip install trl transformers datasets peft accelerate\n```\n\n**Supervised Fine-Tuning** (instruction tuning):\n```python\nfrom trl import SFTTrainer\n\ntrainer = SFTTrainer(\n    model=\"Qwen\u002FQwen2.5-0.5B\",\n    train_dataset=dataset,  # Prompt-completion pairs\n)\ntrainer.train()\n```\n\n**DPO** (align with preferences):\n```python\nfrom trl import DPOTrainer, DPOConfig\n\nconfig = DPOConfig(output_dir=\"model-dpo\", beta=0.1)\ntrainer = DPOTrainer(\n    model=model,\n    args=config,\n    train_dataset=preference_dataset,  # chosen\u002Frejected pairs\n    processing_class=tokenizer\n)\ntrainer.train()\n```\n\n## Common workflows\n\n### Workflow 1: Full RLHF pipeline (SFT → Reward Model → PPO)\n\nComplete pipeline from base model to human-aligned model.\n\nCopy this checklist:\n\n```\nRLHF Training:\n- [ ] Step 1: Supervised fine-tuning (SFT)\n- [ ] Step 2: Train reward model\n- [ ] Step 3: PPO reinforcement learning\n- [ ] Step 4: Evaluate aligned model\n```\n\n**Step 1: Supervised fine-tuning**\n\nTrain base model on instruction-following data:\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom trl import SFTTrainer, SFTConfig\nfrom datasets import load_dataset\n\n# Load model\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen\u002FQwen2.5-0.5B\")\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen\u002FQwen2.5-0.5B\")\n\n# Load instruction dataset\ndataset = load_dataset(\"trl-lib\u002FCapybara\", split=\"train\")\n\n# Configure training\ntraining_args = SFTConfig(\n    output_dir=\"Qwen2.5-0.5B-SFT\",\n    per_device_train_batch_size=4,\n    num_train_epochs=1,\n    learning_rate=2e-5,\n    logging_steps=10,\n    save_strategy=\"epoch\"\n)\n\n# Train\ntrainer = SFTTrainer(\n    model=model,\n    args=training_args,\n    train_dataset=dataset,\n    tokenizer=tokenizer\n)\ntrainer.train()\ntrainer.save_model()\n```\n\n**Step 2: Train reward model**\n\nTrain model to predict human preferences:\n\n```python\nfrom transformers import AutoModelForSequenceClassification\nfrom trl import RewardTrainer, RewardConfig\n\n# Load SFT model as base\nmodel = AutoModelForSequenceClassification.from_pretrained(\n    \"Qwen2.5-0.5B-SFT\",\n    num_labels=1  # Single reward score\n)\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen2.5-0.5B-SFT\")\n\n# Load preference data (chosen\u002Frejected pairs)\ndataset = load_dataset(\"trl-lib\u002Fultrafeedback_binarized\", split=\"train\")\n\n# Configure training\ntraining_args = RewardConfig(\n    output_dir=\"Qwen2.5-0.5B-Reward\",\n    per_device_train_batch_size=2,\n    num_train_epochs=1,\n    learning_rate=1e-5\n)\n\n# Train reward model\ntrainer = RewardTrainer(\n    model=model,\n    args=training_args,\n    processing_class=tokenizer,\n    train_dataset=dataset\n)\ntrainer.train()\ntrainer.save_model()\n```\n\n**Step 3: PPO reinforcement learning**\n\nOptimize policy using reward model:\n\n```bash\npython -m trl.scripts.ppo \\\n    --model_name_or_path Qwen2.5-0.5B-SFT \\\n    --reward_model_path Qwen2.5-0.5B-Reward \\\n    --dataset_name trl-internal-testing\u002Fdescriptiveness-sentiment-trl-style \\\n    --output_dir Qwen2.5-0.5B-PPO \\\n    --learning_rate 3e-6 \\\n    --per_device_train_batch_size 64 \\\n    --total_episodes 10000\n```\n\n**Step 4: Evaluate**\n\n```python\nfrom transformers import pipeline\n\n# Load aligned model\ngenerator = pipeline(\"text-generation\", model=\"Qwen2.5-0.5B-PPO\")\n\n# Test\nprompt = \"Explain quantum computing to a 10-year-old\"\noutput = generator(prompt, max_length=200)[0][\"generated_text\"]\nprint(output)\n```\n\n### Workflow 2: Simple preference alignment with DPO\n\nAlign model with preferences without reward model.\n\nCopy this checklist:\n\n```\nDPO Training:\n- [ ] Step 1: Prepare preference dataset\n- [ ] Step 2: Configure DPO\n- [ ] Step 3: Train with DPOTrainer\n- [ ] Step 4: Evaluate alignment\n```\n\n**Step 1: Prepare preference dataset**\n\nDataset format:\n```json\n{\n  \"prompt\": \"What is the capital of France?\",\n  \"chosen\": \"The capital of France is Paris.\",\n  \"rejected\": \"I don't know.\"\n}\n```\n\nLoad dataset:\n```python\nfrom datasets import load_dataset\n\ndataset = load_dataset(\"trl-lib\u002Fultrafeedback_binarized\", split=\"train\")\n# Or load your own\n# dataset = load_dataset(\"json\", data_files=\"preferences.json\")\n```\n\n**Step 2: Configure DPO**\n\n```python\nfrom trl import DPOConfig\n\nconfig = DPOConfig(\n    output_dir=\"Qwen2.5-0.5B-DPO\",\n    per_device_train_batch_size=4,\n    num_train_epochs=1,\n    learning_rate=5e-7,\n    beta=0.1,  # KL penalty strength\n    max_prompt_length=512,\n    max_length=1024,\n    logging_steps=10\n)\n```\n\n**Step 3: Train with DPOTrainer**\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom trl import DPOTrainer\n\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen\u002FQwen2.5-0.5B-Instruct\")\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen\u002FQwen2.5-0.5B-Instruct\")\n\ntrainer = DPOTrainer(\n    model=model,\n    args=config,\n    train_dataset=dataset,\n    processing_class=tokenizer\n)\n\ntrainer.train()\ntrainer.save_model()\n```\n\n**CLI alternative**:\n```bash\ntrl dpo \\\n    --model_name_or_path Qwen\u002FQwen2.5-0.5B-Instruct \\\n    --dataset_name argilla\u002FCapybara-Preferences \\\n    --output_dir Qwen2.5-0.5B-DPO \\\n    --per_device_train_batch_size 4 \\\n    --learning_rate 5e-7 \\\n    --beta 0.1\n```\n\n### Workflow 3: Memory-efficient online RL with GRPO\n\nTrain with reinforcement learning using minimal memory.\n\nCopy this checklist:\n\n```\nGRPO Training:\n- [ ] Step 1: Define reward function\n- [ ] Step 2: Configure GRPO\n- [ ] Step 3: Train with GRPOTrainer\n```\n\n**Step 1: Define reward function**\n\n```python\ndef reward_function(completions, **kwargs):\n    \"\"\"\n    Compute rewards for completions.\n\n    Args:\n        completions: List of generated texts\n\n    Returns:\n        List of reward scores (floats)\n    \"\"\"\n    rewards = []\n    for completion in completions:\n        # Example: reward based on length and unique words\n        score = len(completion.split())  # Favor longer responses\n        score += len(set(completion.lower().split()))  # Reward unique words\n        rewards.append(score)\n    return rewards\n```\n\nOr use a reward model:\n```python\nfrom transformers import pipeline\n\nreward_model = pipeline(\"text-classification\", model=\"reward-model-path\")\n\ndef reward_from_model(completions, prompts, **kwargs):\n    # Combine prompt + completion\n    full_texts = [p + c for p, c in zip(prompts, completions)]\n    # Get reward scores\n    results = reward_model(full_texts)\n    return [r[\"score\"] for r in results]\n```\n\n**Step 2: Configure GRPO**\n\n```python\nfrom trl import GRPOConfig\n\nconfig = GRPOConfig(\n    output_dir=\"Qwen2-GRPO\",\n    per_device_train_batch_size=4,\n    num_train_epochs=1,\n    learning_rate=1e-5,\n    num_generations=4,  # Generate 4 completions per prompt\n    max_new_tokens=128\n)\n```\n\n**Step 3: Train with GRPOTrainer**\n\n```python\nfrom datasets import load_dataset\nfrom trl import GRPOTrainer\n\n# Load prompt-only dataset\ndataset = load_dataset(\"trl-lib\u002Ftldr\", split=\"train\")\n\ntrainer = GRPOTrainer(\n    model=\"Qwen\u002FQwen2-0.5B-Instruct\",\n    reward_funcs=reward_function,  # Your reward function\n    args=config,\n    train_dataset=dataset\n)\n\ntrainer.train()\n```\n\n**CLI**:\n```bash\ntrl grpo \\\n    --model_name_or_path Qwen\u002FQwen2-0.5B-Instruct \\\n    --dataset_name trl-lib\u002Ftldr \\\n    --output_dir Qwen2-GRPO \\\n    --num_generations 4\n```\n\n## When to use vs alternatives\n\n**Use TRL when:**\n- Need to align model with human preferences\n- Have preference data (chosen\u002Frejected pairs)\n- Want to use reinforcement learning (PPO, GRPO)\n- Need reward model training\n- Doing RLHF (full pipeline)\n\n**Method selection**:\n- **SFT**: Have prompt-completion pairs, want basic instruction following\n- **DPO**: Have preferences, want simple alignment (no reward model needed)\n- **PPO**: Have reward model, need maximum control over RL\n- **GRPO**: Memory-constrained, want online RL\n- **Reward Model**: Building RLHF pipeline, need to score generations\n\n**Use alternatives instead:**\n- **HuggingFace Trainer**: Basic fine-tuning without RL\n- **Axolotl**: YAML-based training configuration\n- **LitGPT**: Educational, minimal fine-tuning\n- **Unsloth**: Fast LoRA training\n\n## Common issues\n\n**Issue: OOM during DPO training**\n\nReduce batch size and sequence length:\n```python\nconfig = DPOConfig(\n    per_device_train_batch_size=1,  # Reduce from 4\n    max_length=512,  # Reduce from 1024\n    gradient_accumulation_steps=8  # Maintain effective batch\n)\n```\n\nOr use gradient checkpointing:\n```python\nmodel.gradient_checkpointing_enable()\n```\n\n**Issue: Poor alignment quality**\n\nTune beta parameter:\n```python\n# Higher beta = more conservative (stays closer to reference)\nconfig = DPOConfig(beta=0.5)  # Default 0.1\n\n# Lower beta = more aggressive alignment\nconfig = DPOConfig(beta=0.01)\n```\n\n**Issue: Reward model not learning**\n\nCheck loss type and learning rate:\n```python\nconfig = RewardConfig(\n    learning_rate=1e-5,  # Try different LR\n    num_train_epochs=3  # Train longer\n)\n```\n\nEnsure preference dataset has clear winners:\n```python\n# Verify dataset\nprint(dataset[0])\n# Should have clear chosen > rejected\n```\n\n**Issue: PPO training unstable**\n\nAdjust KL coefficient:\n```python\nconfig = PPOConfig(\n    kl_coef=0.1,  # Increase from 0.05\n    cliprange=0.1  # Reduce from 0.2\n)\n```\n\n## Advanced topics\n\n**SFT training guide**: See [references\u002Fsft-training.md](references\u002Fsft-training.md) for dataset formats, chat templates, packing strategies, and multi-GPU training.\n\n**DPO variants**: See [references\u002Fdpo-variants.md](references\u002Fdpo-variants.md) for IPO, cDPO, RPO, and other DPO loss functions with recommended hyperparameters.\n\n**Reward modeling**: See [references\u002Freward-modeling.md](references\u002Freward-modeling.md) for outcome vs process rewards, Bradley-Terry loss, and reward model evaluation.\n\n**Online RL methods**: See [references\u002Fonline-rl.md](references\u002Fonline-rl.md) for PPO, GRPO, RLOO, and OnlineDPO with detailed configurations.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA (CUDA required)\n- **VRAM**: Depends on model and method\n  - SFT 7B: 16GB (with LoRA)\n  - DPO 7B: 24GB (stores reference model)\n  - PPO 7B: 40GB (policy + reward model)\n  - GRPO 7B: 24GB (more memory efficient)\n- **Multi-GPU**: Supported via `accelerate`\n- **Mixed precision**: BF16 recommended (A100\u002FH100)\n\n**Memory optimization**:\n- Use LoRA\u002FQLoRA for all methods\n- Enable gradient checkpointing\n- Use smaller batch sizes with gradient accumulation\n\n## Resources\n\n- Docs: https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftrl\u002F\n- GitHub: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl\n- Papers:\n  - \"Training language models to follow instructions with human feedback\" (InstructGPT, 2022)\n  - \"Direct Preference Optimization: Your Language Model is Secretly a Reward Model\" (DPO, 2023)\n  - \"Group Relative Policy Optimization\" (GRPO, 2024)\n- Examples: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl\u002Ftree\u002Fmain\u002Fexamples\u002Fscripts\n\n## Agent operating procedure\n\n1. **Check the environment.** Check GPU type, memory and driver\u002FCUDA versions (`nvidia-smi`), framework versions, and dataset location and size.\n2. **Pin down the inputs.** Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.\n3. **Run a small version first.** Do a smoke run: tiny model or subset, few steps, and confirm loss decreases and checkpoints save.\n4. **Execute the full task** using the instructions and references above.\n5. **Validate the result.** Track metrics on held-out data, compare against a baseline, and record seeds, configs and hardware.\n6. **Report.** State what was run (versions, commands, parameters), what was checked, and what is still uncertain.\n\n| If this happens | Do this |\n|---|---|\n| CUDA out-of-memory | Reduce batch size, enable gradient accumulation\u002Fcheckpointing or mixed precision, or shard the model. |\n| Loss is NaN or diverges | Lower the learning rate, check data for invalid values, and enable gradient clipping. |\n| A function, flag or endpoint in these instructions is missing in the installed version | Check the installed version's own documentation (`help()`, `--help`, official docs), adapt, and tell the user. Never invent an API. |\n| A required input, identifier or parameter is ambiguous | Ask the user, or state the assumption explicitly before running. |\n\n**Integrity rules**\n\n- Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.\n- Never claim training results without logs; estimate compute cost before launching large jobs.\n- Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.\n- Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.\n\n## Related skills\n\n- `grpo-rl-training`: Guides GRPO (Group Relative Policy Optimization) fine-tuning of language models with the TRL library, including GRPOTrainer configuration,…\n- `simpo-training`: Simple Preference Optimization for LLM alignment.\n- `verl-rl-training`: Provides guidance for training LLMs with reinforcement learning using verl (Volcano Engine RL).\n","# TRL - Transformer Reinforcement Learning\n\n## Quick start\n\nTRL provides post-training methods for aligning language models with human preferences.\n\n**Installation**:\n```bash\npip install trl transformers datasets peft accelerate\n```\n\n**Supervised Fine-Tuning** (instruction tuning):\n```python\nfrom trl import SFTTrainer\n\ntrainer = SFTTrainer(\n    model=\"Qwen\u002FQwen2.5-0.5B\",\n    train_dataset=dataset,  # Prompt-completion pairs\n)\ntrainer.train()\n```\n\n**DPO** (align with preferences):\n```python\nfrom trl import DPOTrainer, DPOConfig\n\nconfig = DPOConfig(output_dir=\"model-dpo\", beta=0.1)\ntrainer = DPOTrainer(\n    model=model,\n    args=config,\n    train_dataset=preference_dataset,  # chosen\u002Frejected pairs\n    processing_class=tokenizer\n)\ntrainer.train()\n```\n\n## Common workflows\n\n### Workflow 1: Full RLHF pipeline (SFT → Reward Model → PPO)\n\nComplete pipeline from base model to human-aligned model.\n\nCopy this checklist:\n\n```\nRLHF Training:\n- [ ] Step 1: Supervised fine-tuning (SFT)\n- [ ] Step 2: Train reward model\n- [ ] Step 3: PPO reinforcement learning\n- [ ] Step 4: Evaluate aligned model\n```\n\n**Step 1: Supervised fine-tuning**\n\nTrain base model on instruction-following data:\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom trl import SFTTrainer, SFTConfig\nfrom datasets import load_dataset\n\n# Load model\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen\u002FQwen2.5-0.5B\")\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen\u002FQwen2.5-0.5B\")\n\n# Load instruction dataset\ndataset = load_dataset(\"trl-lib\u002FCapybara\", split=\"train\")\n\n# Configure training\ntraining_args = SFTConfig(\n    output_dir=\"Qwen2.5-0.5B-SFT\",\n    per_device_train_batch_size=4,\n    num_train_epochs=1,\n    learning_rate=2e-5,\n    logging_steps=10,\n    save_strategy=\"epoch\"\n)\n\n# Train\ntrainer = SFTTrainer(\n    model=model,\n    args=training_args,\n    train_dataset=dataset,\n    tokenizer=tokenizer\n)\ntrainer.train()\ntrainer.save_model()\n```\n\n**Step 2: Train reward model**\n\nTrain model to predict human preferences:\n\n```python\nfrom transformers import AutoModelForSequenceClassification\nfrom trl import RewardTrainer, RewardConfig\n\n# Load SFT model as base\nmodel = AutoModelForSequenceClassification.from_pretrained(\n    \"Qwen2.5-0.5B-SFT\",\n    num_labels=1  # Single reward score\n)\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen2.5-0.5B-SFT\")\n\n# Load preference data (chosen\u002Frejected pairs)\ndataset = load_dataset(\"trl-lib\u002Fultrafeedback_binarized\", split=\"train\")\n\n# Configure training\ntraining_args = RewardConfig(\n    output_dir=\"Qwen2.5-0.5B-Reward\",\n    per_device_train_batch_size=2,\n    num_train_epochs=1,\n    learning_rate=1e-5\n)\n\n# Train reward model\ntrainer = RewardTrainer(\n    model=model,\n    args=training_args,\n    processing_class=tokenizer,\n    train_dataset=dataset\n)\ntrainer.train()\ntrainer.save_model()\n```\n\n**Step 3: PPO reinforcement learning**\n\nOptimize policy using reward model:\n\n```bash\npython -m trl.scripts.ppo \\\n    --model_name_or_path Qwen2.5-0.5B-SFT \\\n    --reward_model_path Qwen2.5-0.5B-Reward \\\n    --dataset_name trl-internal-testing\u002Fdescriptiveness-sentiment-trl-style \\\n    --output_dir Qwen2.5-0.5B-PPO \\\n    --learning_rate 3e-6 \\\n    --per_device_train_batch_size 64 \\\n    --total_episodes 10000\n```\n\n**Step 4: Evaluate**\n\n```python\nfrom transformers import pipeline\n\n# Load aligned model\ngenerator = pipeline(\"text-generation\", model=\"Qwen2.5-0.5B-PPO\")\n\n# Test\nprompt = \"Explain quantum computing to a 10-year-old\"\noutput = generator(prompt, max_length=200)[0][\"generated_text\"]\nprint(output)\n```\n\n### Workflow 2: Simple preference alignment with DPO\n\nAlign model with preferences without reward model.\n\nCopy this checklist:\n\n```\nDPO Training:\n- [ ] Step 1: Prepare preference dataset\n- [ ] Step 2: Configure DPO\n- [ ] Step 3: Train with DPOTrainer\n- [ ] Step 4: Evaluate alignment\n```\n\n**Step 1: Prepare preference dataset**\n\nDataset format:\n```json\n{\n  \"prompt\": \"What is the capital of France?\",\n  \"chosen\": \"The capital of France is Paris.\",\n  \"rejected\": \"I don't know.\"\n}\n```\n\nLoad dataset:\n```python\nfrom datasets import load_dataset\n\ndataset = load_dataset(\"trl-lib\u002Fultrafeedback_binarized\", split=\"train\")\n# Or load your own\n# dataset = load_dataset(\"json\", data_files=\"preferences.json\")\n```\n\n**Step 2: Configure DPO**\n\n```python\nfrom trl import DPOConfig\n\nconfig = DPOConfig(\n    output_dir=\"Qwen2.5-0.5B-DPO\",\n    per_device_train_batch_size=4,\n    num_train_epochs=1,\n    learning_rate=5e-7,\n    beta=0.1,  # KL penalty strength\n    max_prompt_length=512,\n    max_length=1024,\n    logging_steps=10\n)\n```\n\n**Step 3: Train with DPOTrainer**\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom trl import DPOTrainer\n\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen\u002FQwen2.5-0.5B-Instruct\")\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen\u002FQwen2.5-0.5B-Instruct\")\n\ntrainer = DPOTrainer(\n    model=model,\n    args=config,\n    train_dataset=dataset,\n    processing_class=tokenizer\n)\n\ntrainer.train()\ntrainer.save_model()\n```\n\n**CLI alternative**:\n```bash\ntrl dpo \\\n    --model_name_or_path Qwen\u002FQwen2.5-0.5B-Instruct \\\n    --dataset_name argilla\u002FCapybara-Preferences \\\n    --output_dir Qwen2.5-0.5B-DPO \\\n    --per_device_train_batch_size 4 \\\n    --learning_rate 5e-7 \\\n    --beta 0.1\n```\n\n### Workflow 3: Memory-efficient online RL with GRPO\n\nTrain with reinforcement learning using minimal memory.\n\nCopy this checklist:\n\n```\nGRPO Training:\n- [ ] Step 1: Define reward function\n- [ ] Step 2: Configure GRPO\n- [ ] Step 3: Train with GRPOTrainer\n```\n\n**Step 1: Define reward function**\n\n```python\ndef reward_function(completions, **kwargs):\n    \"\"\"\n    Compute rewards for completions.\n\n    Args:\n        completions: List of generated texts\n\n    Returns:\n        List of reward scores (floats)\n    \"\"\"\n    rewards = []\n    for completion in completions:\n        # Example: reward based on length and unique words\n        score = len(completion.split())  # Favor longer responses\n        score += len(set(completion.lower().split()))  # Reward unique words\n        rewards.append(score)\n    return rewards\n```\n\nOr use a reward model:\n```python\nfrom transformers import pipeline\n\nreward_model = pipeline(\"text-classification\", model=\"reward-model-path\")\n\ndef reward_from_model(completions, prompts, **kwargs):\n    # Combine prompt + completion\n    full_texts = [p + c for p, c in zip(prompts, completions)]\n    # Get reward scores\n    results = reward_model(full_texts)\n    return [r[\"score\"] for r in results]\n```\n\n**Step 2: Configure GRPO**\n\n```python\nfrom trl import GRPOConfig\n\nconfig = GRPOConfig(\n    output_dir=\"Qwen2-GRPO\",\n    per_device_train_batch_size=4,\n    num_train_epochs=1,\n    learning_rate=1e-5,\n    num_generations=4,  # Generate 4 completions per prompt\n    max_new_tokens=128\n)\n```\n\n**Step 3: Train with GRPOTrainer**\n\n```python\nfrom datasets import load_dataset\nfrom trl import GRPOTrainer\n\n# Load prompt-only dataset\ndataset = load_dataset(\"trl-lib\u002Ftldr\", split=\"train\")\n\ntrainer = GRPOTrainer(\n    model=\"Qwen\u002FQwen2-0.5B-Instruct\",\n    reward_funcs=reward_function,  # Your reward function\n    args=config,\n    train_dataset=dataset\n)\n\ntrainer.train()\n```\n\n**CLI**:\n```bash\ntrl grpo \\\n    --model_name_or_path Qwen\u002FQwen2-0.5B-Instruct \\\n    --dataset_name trl-lib\u002Ftldr \\\n    --output_dir Qwen2-GRPO \\\n    --num_generations 4\n```\n\n## When to use vs alternatives\n\n**Use TRL when:**\n- Need to align model with human preferences\n- Have preference data (chosen\u002Frejected pairs)\n- Want to use reinforcement learning (PPO, GRPO)\n- Need reward model training\n- Doing RLHF (full pipeline)\n\n**Method selection**:\n- **SFT**: Have prompt-completion pairs, want basic instruction following\n- **DPO**: Have preferences, want simple alignment (no reward model needed)\n- **PPO**: Have reward model, need maximum control over RL\n- **GRPO**: Memory-constrained, want online RL\n- **Reward Model**: Building RLHF pipeline, need to score generations\n\n**Use alternatives instead:**\n- **HuggingFace Trainer**: Basic fine-tuning without RL\n- **Axolotl**: YAML-based training configuration\n- **LitGPT**: Educational, minimal fine-tuning\n- **Unsloth**: Fast LoRA training\n\n## Common issues\n\n**Issue: OOM during DPO training**\n\nReduce batch size and sequence length:\n```python\nconfig = DPOConfig(\n    per_device_train_batch_size=1,  # Reduce from 4\n    max_length=512,  # Reduce from 1024\n    gradient_accumulation_steps=8  # Maintain effective batch\n)\n```\n\nOr use gradient checkpointing:\n```python\nmodel.gradient_checkpointing_enable()\n```\n\n**Issue: Poor alignment quality**\n\nTune beta parameter:\n```python\n# Higher beta = more conservative (stays closer to reference)\nconfig = DPOConfig(beta=0.5)  # Default 0.1\n\n# Lower beta = more aggressive alignment\nconfig = DPOConfig(beta=0.01)\n```\n\n**Issue: Reward model not learning**\n\nCheck loss type and learning rate:\n```python\nconfig = RewardConfig(\n    learning_rate=1e-5,  # Try different LR\n    num_train_epochs=3  # Train longer\n)\n```\n\nEnsure preference dataset has clear winners:\n```python\n# Verify dataset\nprint(dataset[0])\n# Should have clear chosen > rejected\n```\n\n**Issue: PPO training unstable**\n\nAdjust KL coefficient:\n```python\nconfig = PPOConfig(\n    kl_coef=0.1,  # Increase from 0.05\n    cliprange=0.1  # Reduce from 0.2\n)\n```\n\n## Advanced topics\n\n**SFT training guide**: See [references\u002Fsft-training.md](references\u002Fsft-training.md) for dataset formats, chat templates, packing strategies, and multi-GPU training.\n\n**DPO variants**: See [references\u002Fdpo-variants.md](references\u002Fdpo-variants.md) for IPO, cDPO, RPO, and other DPO loss functions with recommended hyperparameters.\n\n**Reward modeling**: See [references\u002Freward-modeling.md](references\u002Freward-modeling.md) for outcome vs process rewards, Bradley-Terry loss, and reward model evaluation.\n\n**Online RL methods**: See [references\u002Fonline-rl.md](references\u002Fonline-rl.md) for PPO, GRPO, RLOO, and OnlineDPO with detailed configurations.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA (CUDA required)\n- **VRAM**: Depends on model and method\n  - SFT 7B: 16GB (with LoRA)\n  - DPO 7B: 24GB (stores reference model)\n  - PPO 7B: 40GB (policy + reward model)\n  - GRPO 7B: 24GB (more memory efficient)\n- **Multi-GPU**: Supported via `accelerate`\n- **Mixed precision**: BF16 recommended (A100\u002FH100)\n\n**Memory optimization**:\n- Use LoRA\u002FQLoRA for all methods\n- Enable gradient checkpointing\n- Use smaller batch sizes with gradient accumulation\n\n## Resources\n\n- Docs: https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftrl\u002F\n- GitHub: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl\n- Papers:\n  - \"Training language models to follow instructions with human feedback\" (InstructGPT, 2022)\n  - \"Direct Preference Optimization: Your Language Model is Secretly a Reward Model\" (DPO, 2023)\n  - \"Group Relative Policy Optimization\" (GRPO, 2024)\n- Examples: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl\u002Ftree\u002Fmain\u002Fexamples\u002Fscripts\n\n## Agent operating procedure\n\n1. **Check the environment.** Check GPU type, memory and driver\u002FCUDA versions (`nvidia-smi`), framework versions, and dataset location and size.\n2. **Pin down the inputs.** Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.\n3. **Run a small version first.** Do a smoke run: tiny model or subset, few steps, and confirm loss decreases and checkpoints save.\n4. **Execute the full task** using the instructions and references above.\n5. **Validate the result.** Track metrics on held-out data, compare against a baseline, and record seeds, configs and hardware.\n6. **Report.** State what was run (versions, commands, parameters), what was checked, and what is still uncertain.\n\n| If this happens | Do this |\n|---|---|\n| CUDA out-of-memory | Reduce batch size, enable gradient accumulation\u002Fcheckpointing or mixed precision, or shard the model. |\n| Loss is NaN or diverges | Lower the learning rate, check data for invalid values, and enable gradient clipping. |\n| A function, flag or endpoint in these instructions is missing in the installed version | Check the installed version's own documentation (`help()`, `--help`, official docs), adapt, and tell the user. Never invent an API. |\n| A required input, identifier or parameter is ambiguous | Ask the user, or state the assumption explicitly before running. |\n\n**Integrity rules**\n\n- Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.\n- Never claim training results without logs; estimate compute cost before launching large jobs.\n- Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.\n- Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.\n\n## Related skills\n\n- `grpo-rl-training`: Guides GRPO (Group Relative Policy Optimization) fine-tuning of language models with the TRL library, including GRPOTrainer configuration,…\n- `simpo-training`: Simple Preference Optimization for LLM alignment.\n- `verl-rl-training`: Provides guidance for training LLMs with reinforcement learning using verl (Volcano Engine RL).\n",{"name":5,"license":15,"metadata":45,"description":7},{"tags":46,"version":47,"category":48,"maintainer":49,"dependencies":50},"Post-Training, TRL, Reinforcement Learning, Fine-Tuning, SFT, DPO, PPO, GRPO, RLHF, Preference Alignment, HuggingFace","1.0.0","ml-training","Kalaris Labs","trl, transformers, datasets, peft, accelerate, torch","main","2b63f856ce19c2a797929cacabd9e0fc936812fa","plugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Ffine-tuning-with-trl",491,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins","https:\u002F\u002Fhol.org\u002Fplugins\u002Fbest-codex-plugins","https:\u002F\u002Fgithub.com\u002Fhashgraph-online","https:\u002F\u002Fhol.org","Open-source standards and developer tools for interoperable AI agents: discovery, identity, communication, payments, privacy and security.","2026-11-14 14:06:13",{"source":8,"skillName":5,"commands":62,"prompt":72,"zipAvailable":73},[63,66,69],{"tool":64,"command":65},"npx","npx skills add hashgraph-online\u002Fawesome-codex-plugins --skill fine-tuning-with-trl",{"tool":67,"command":68},"bunx","bunx skills add hashgraph-online\u002Fawesome-codex-plugins --skill fine-tuning-with-trl",{"tool":70,"command":71},"pnpm","pnpm dlx skills add hashgraph-online\u002Fawesome-codex-plugins --skill fine-tuning-with-trl","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Ffine-tuning-with-trl\u002FSKILL.md",true,[75,81,85,89],{"path":76,"resourceType":77,"resourceTypeDesc":78,"sizeBytes":79,"sourceHtmlUrl":80},"references\u002Fdpo-variants.md","REFERENCE","参考文档",4297,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Ffine-tuning-with-trl\u002Freferences\u002Fdpo-variants.md",{"path":82,"resourceType":77,"resourceTypeDesc":78,"sizeBytes":83,"sourceHtmlUrl":84},"references\u002Fonline-rl.md",1971,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Ffine-tuning-with-trl\u002Freferences\u002Fonline-rl.md",{"path":86,"resourceType":77,"resourceTypeDesc":78,"sizeBytes":87,"sourceHtmlUrl":88},"references\u002Freward-modeling.md",2597,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Ffine-tuning-with-trl\u002Freferences\u002Freward-modeling.md",{"path":90,"resourceType":77,"resourceTypeDesc":78,"sizeBytes":91,"sourceHtmlUrl":92},"references\u002Fsft-training.md",3237,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Ffine-tuning-with-trl\u002Freferences\u002Fsft-training.md",[94,103,111,119,127,135],{"skillNo":95,"name":96,"nameZh":6,"description":97,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":98,"sourceHtmlUrl":99,"categories":100,"keywords":101,"publishedAt":102},"01M7DV332YQMGD8WTRJ9DZQXTB","worktable-review-with-annotations","Create, inspect, reply to, update, or resolve situated Worktable annotations. Use when feedback belongs to a Doc section, block, text range, or HTML Doc, or when the user asks to address existing review feedback and coordinate a requested artifact edit.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-review-with-annotations\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-review-with-annotations\u002FSKILL.md",[],[],"2026-11-15 14:19:17",{"skillNo":104,"name":105,"nameZh":6,"description":106,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":107,"sourceHtmlUrl":108,"categories":109,"keywords":110,"publishedAt":102},"01M7DV2YGHCZVBEB1AHYB6CGHE","worktable-find-and-synthesize-context","Find, assess, and synthesize context from Worktable Docs and Records. Use when the user asks what a workspace contains, wants workspace-backed research, needs current facts reconciled across artifacts, or requests a cited synthesis without changing workspace state.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-find-and-synthesize-context\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-find-and-synthesize-context\u002FSKILL.md",[],[],{"skillNo":112,"name":113,"nameZh":6,"description":114,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":115,"sourceHtmlUrl":116,"categories":117,"keywords":118,"publishedAt":102},"01M7DV2TQN1H11ME2SAF9MXN7V","worktable-create-or-update-docs","Create, edit, rename, or maintain narrative Worktable Docs. Use for plans, decisions, briefs, notes, research syntheses, and other durable prose when the user requests a new artifact or a safe update to an existing one.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-update-docs\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-update-docs\u002FSKILL.md",[],[],{"skillNo":120,"name":121,"nameZh":6,"description":122,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":123,"sourceHtmlUrl":124,"categories":125,"keywords":126,"publishedAt":102},"01M7DV2NE452NEA8TXATKSNAJH","worktable-create-or-manage-records","Organize repeated structured information as durable Worktable Records. Use for research datasets, inventories, directories, pipelines, trackers, and other collections whose items need identity, fields, filtering, links, validation, or individual updates.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-records\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-records\u002FSKILL.md",[],[],{"skillNo":128,"name":129,"nameZh":6,"description":130,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":131,"sourceHtmlUrl":132,"categories":133,"keywords":134,"publishedAt":102},"01M7DV2HENRBR2KSCVM064WH0Y","worktable-create-or-manage-interactive-html","Create or improve durable visual and interactive artifacts in Worktable. Use for dashboards, trackers, calculators, mini apps, slide presentations, and other requests that benefit from a polished HTML Doc.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-interactive-html\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-interactive-html\u002FSKILL.md",[],[],{"skillNo":136,"name":137,"nameZh":6,"description":138,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":139,"sourceHtmlUrl":140,"categories":141,"keywords":142,"publishedAt":102},"01M7DV2CADR5R7RQFFYPNAGR9Y","worktable-collaborate-threads","Read, answer, and hand off messages through durable Worktable threads with users or other authenticated participants. Use when the user asks to check addressed messages, reply to a participant, continue a thread, request another agent's help, wait briefly for a response, or watch for comments and messages sent to the agent.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-collaborate-threads\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-collaborate-threads\u002FSKILL.md",[],[]]