[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6MFEM3T18Z4VW6R85FEBJY5":3,"skill-same:01M6MFEM3T18Z4VW6R85FEBJY5":85},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":64,"commitSha":65,"skillDirectory":66,"forksCount":67,"repositoryHtmlUrl":68,"repositoryHomepageUrl":6,"ownerHtmlUrl":69,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":70,"install":71,"resources":84},"01M6MFEM3T18Z4VW6R85FEBJY5","fine-tuning-with-trl",null,"Fine-tune LLMs using reinforcement learning with TRL - SFT for instruction tuning, DPO for preference alignment, PPO\u002FGRPO for reward optimization, and reward model training. Use when need RLHF, align model with preferences, or train from human feedback. Works with HuggingFace Transformers.","ibragimov-oasis\u002Fvibe-coder","ibragimov-oasis","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F240665573?v=4","01M6DF78Y45R8228FHAZ7G6JXM","IBRAGIMOV MUKHAMMADJON","01M6K5ZYR8TDPAFG4R6693E2A6",3,"MIT","https:\u002F\u002Fgithub.com\u002Fibragimov-oasis\u002Fvibe-coder\u002Fblob\u002Fmain\u002Fvc-kit\u002Fcontent\u002FCOMBINED\u002FCOMBINED\u002FREPO_DOCS\u002F02-hermes-agent\u002Fskills\u002Fmlops\u002Ftraining\u002Ftrl-fine-tuning\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"with","TRL","tuning","fine tuning with trl","fine tuning","for","llms","orchestra","Accelerate","human","align","alignment","feedback","model","learning","Transformers","RLHF","Post-Training","Optimization","training","2026-11-10 10:12:30",false,"---\nname: fine-tuning-with-trl\ndescription: Fine-tune LLMs using reinforcement learning with TRL - SFT for instruction tuning, DPO for preference alignment, PPO\u002FGRPO for reward optimization, and reward model training. Use when need RLHF, align model with preferences, or train from human feedback. Works with HuggingFace Transformers.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ndependencies: [trl, transformers, datasets, peft, accelerate, torch]\nmetadata:\n  hermes:\n    tags: [Post-Training, TRL, Reinforcement Learning, Fine-Tuning, SFT, DPO, PPO, GRPO, RLHF, Preference Alignment, HuggingFace]\n\n---\n\n# TRL - Transformer Reinforcement Learning\n\n## Quick start\n\nTRL provides post-training methods for aligning language models with human preferences.\n\n**Installation**:\n```bash\npip install trl transformers datasets peft accelerate\n```\n\n**Supervised Fine-Tuning** (instruction tuning):\n```python\nfrom trl import SFTTrainer\n\ntrainer = SFTTrainer(\n    model=\"Qwen\u002FQwen2.5-0.5B\",\n    train_dataset=dataset,  # Prompt-completion pairs\n)\ntrainer.train()\n```\n\n**DPO** (align with preferences):\n```python\nfrom trl import DPOTrainer, DPOConfig\n\nconfig = DPOConfig(output_dir=\"model-dpo\", beta=0.1)\ntrainer = DPOTrainer(\n    model=model,\n    args=config,\n    train_dataset=preference_dataset,  # chosen\u002Frejected pairs\n    processing_class=tokenizer\n)\ntrainer.train()\n```\n\n## Common workflows\n\n### Workflow 1: Full RLHF pipeline (SFT → Reward Model → PPO)\n\nComplete pipeline from base model to human-aligned model.\n\nCopy this checklist:\n\n```\nRLHF Training:\n- [ ] Step 1: Supervised fine-tuning (SFT)\n- [ ] Step 2: Train reward model\n- [ ] Step 3: PPO reinforcement learning\n- [ ] Step 4: Evaluate aligned model\n```\n\n**Step 1: Supervised fine-tuning**\n\nTrain base model on instruction-following data:\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom trl import SFTTrainer, SFTConfig\nfrom datasets import load_dataset\n\n# Load model\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen\u002FQwen2.5-0.5B\")\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen\u002FQwen2.5-0.5B\")\n\n# Load instruction dataset\ndataset = load_dataset(\"trl-lib\u002FCapybara\", split=\"train\")\n\n# Configure training\ntraining_args = SFTConfig(\n    output_dir=\"Qwen2.5-0.5B-SFT\",\n    per_device_train_batch_size=4,\n    num_train_epochs=1,\n    learning_rate=2e-5,\n    logging_steps=10,\n    save_strategy=\"epoch\"\n)\n\n# Train\ntrainer = SFTTrainer(\n    model=model,\n    args=training_args,\n    train_dataset=dataset,\n    tokenizer=tokenizer\n)\ntrainer.train()\ntrainer.save_model()\n```\n\n**Step 2: Train reward model**\n\nTrain model to predict human preferences:\n\n```python\nfrom transformers import AutoModelForSequenceClassification\nfrom trl import RewardTrainer, RewardConfig\n\n# Load SFT model as base\nmodel = AutoModelForSequenceClassification.from_pretrained(\n    \"Qwen2.5-0.5B-SFT\",\n    num_labels=1  # Single reward score\n)\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen2.5-0.5B-SFT\")\n\n# Load preference data (chosen\u002Frejected pairs)\ndataset = load_dataset(\"trl-lib\u002Fultrafeedback_binarized\", split=\"train\")\n\n# Configure training\ntraining_args = RewardConfig(\n    output_dir=\"Qwen2.5-0.5B-Reward\",\n    per_device_train_batch_size=2,\n    num_train_epochs=1,\n    learning_rate=1e-5\n)\n\n# Train reward model\ntrainer = RewardTrainer(\n    model=model,\n    args=training_args,\n    processing_class=tokenizer,\n    train_dataset=dataset\n)\ntrainer.train()\ntrainer.save_model()\n```\n\n**Step 3: PPO reinforcement learning**\n\nOptimize policy using reward model:\n\n```bash\npython -m trl.scripts.ppo \\\n    --model_name_or_path Qwen2.5-0.5B-SFT \\\n    --reward_model_path Qwen2.5-0.5B-Reward \\\n    --dataset_name trl-internal-testing\u002Fdescriptiveness-sentiment-trl-style \\\n    --output_dir Qwen2.5-0.5B-PPO \\\n    --learning_rate 3e-6 \\\n    --per_device_train_batch_size 64 \\\n    --total_episodes 10000\n```\n\n**Step 4: Evaluate**\n\n```python\nfrom transformers import pipeline\n\n# Load aligned model\ngenerator = pipeline(\"text-generation\", model=\"Qwen2.5-0.5B-PPO\")\n\n# Test\nprompt = \"Explain quantum computing to a 10-year-old\"\noutput = generator(prompt, max_length=200)[0][\"generated_text\"]\nprint(output)\n```\n\n### Workflow 2: Simple preference alignment with DPO\n\nAlign model with preferences without reward model.\n\nCopy this checklist:\n\n```\nDPO Training:\n- [ ] Step 1: Prepare preference dataset\n- [ ] Step 2: Configure DPO\n- [ ] Step 3: Train with DPOTrainer\n- [ ] Step 4: Evaluate alignment\n```\n\n**Step 1: Prepare preference dataset**\n\nDataset format:\n```json\n{\n  \"prompt\": \"What is the capital of France?\",\n  \"chosen\": \"The capital of France is Paris.\",\n  \"rejected\": \"I don't know.\"\n}\n```\n\nLoad dataset:\n```python\nfrom datasets import load_dataset\n\ndataset = load_dataset(\"trl-lib\u002Fultrafeedback_binarized\", split=\"train\")\n# Or load your own\n# dataset = load_dataset(\"json\", data_files=\"preferences.json\")\n```\n\n**Step 2: Configure DPO**\n\n```python\nfrom trl import DPOConfig\n\nconfig = DPOConfig(\n    output_dir=\"Qwen2.5-0.5B-DPO\",\n    per_device_train_batch_size=4,\n    num_train_epochs=1,\n    learning_rate=5e-7,\n    beta=0.1,  # KL penalty strength\n    max_prompt_length=512,\n    max_length=1024,\n    logging_steps=10\n)\n```\n\n**Step 3: Train with DPOTrainer**\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom trl import DPOTrainer\n\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen\u002FQwen2.5-0.5B-Instruct\")\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen\u002FQwen2.5-0.5B-Instruct\")\n\ntrainer = DPOTrainer(\n    model=model,\n    args=config,\n    train_dataset=dataset,\n    processing_class=tokenizer\n)\n\ntrainer.train()\ntrainer.save_model()\n```\n\n**CLI alternative**:\n```bash\ntrl dpo \\\n    --model_name_or_path Qwen\u002FQwen2.5-0.5B-Instruct \\\n    --dataset_name argilla\u002FCapybara-Preferences \\\n    --output_dir Qwen2.5-0.5B-DPO \\\n    --per_device_train_batch_size 4 \\\n    --learning_rate 5e-7 \\\n    --beta 0.1\n```\n\n### Workflow 3: Memory-efficient online RL with GRPO\n\nTrain with reinforcement learning using minimal memory.\n\nCopy this checklist:\n\n```\nGRPO Training:\n- [ ] Step 1: Define reward function\n- [ ] Step 2: Configure GRPO\n- [ ] Step 3: Train with GRPOTrainer\n```\n\n**Step 1: Define reward function**\n\n```python\ndef reward_function(completions, **kwargs):\n    \"\"\"\n    Compute rewards for completions.\n\n    Args:\n        completions: List of generated texts\n\n    Returns:\n        List of reward scores (floats)\n    \"\"\"\n    rewards = []\n    for completion in completions:\n        # Example: reward based on length and unique words\n        score = len(completion.split())  # Favor longer responses\n        score += len(set(completion.lower().split()))  # Reward unique words\n        rewards.append(score)\n    return rewards\n```\n\nOr use a reward model:\n```python\nfrom transformers import pipeline\n\nreward_model = pipeline(\"text-classification\", model=\"reward-model-path\")\n\ndef reward_from_model(completions, prompts, **kwargs):\n    # Combine prompt + completion\n    full_texts = [p + c for p, c in zip(prompts, completions)]\n    # Get reward scores\n    results = reward_model(full_texts)\n    return [r[\"score\"] for r in results]\n```\n\n**Step 2: Configure GRPO**\n\n```python\nfrom trl import GRPOConfig\n\nconfig = GRPOConfig(\n    output_dir=\"Qwen2-GRPO\",\n    per_device_train_batch_size=4,\n    num_train_epochs=1,\n    learning_rate=1e-5,\n    num_generations=4,  # Generate 4 completions per prompt\n    max_new_tokens=128\n)\n```\n\n**Step 3: Train with GRPOTrainer**\n\n```python\nfrom datasets import load_dataset\nfrom trl import GRPOTrainer\n\n# Load prompt-only dataset\ndataset = load_dataset(\"trl-lib\u002Ftldr\", split=\"train\")\n\ntrainer = GRPOTrainer(\n    model=\"Qwen\u002FQwen2-0.5B-Instruct\",\n    reward_funcs=reward_function,  # Your reward function\n    args=config,\n    train_dataset=dataset\n)\n\ntrainer.train()\n```\n\n**CLI**:\n```bash\ntrl grpo \\\n    --model_name_or_path Qwen\u002FQwen2-0.5B-Instruct \\\n    --dataset_name trl-lib\u002Ftldr \\\n    --output_dir Qwen2-GRPO \\\n    --num_generations 4\n```\n\n## When to use vs alternatives\n\n**Use TRL when:**\n- Need to align model with human preferences\n- Have preference data (chosen\u002Frejected pairs)\n- Want to use reinforcement learning (PPO, GRPO)\n- Need reward model training\n- Doing RLHF (full pipeline)\n\n**Method selection**:\n- **SFT**: Have prompt-completion pairs, want basic instruction following\n- **DPO**: Have preferences, want simple alignment (no reward model needed)\n- **PPO**: Have reward model, need maximum control over RL\n- **GRPO**: Memory-constrained, want online RL\n- **Reward Model**: Building RLHF pipeline, need to score generations\n\n**Use alternatives instead:**\n- **HuggingFace Trainer**: Basic fine-tuning without RL\n- **Axolotl**: YAML-based training configuration\n- **LitGPT**: Educational, minimal fine-tuning\n- **Unsloth**: Fast LoRA training\n\n## Common issues\n\n**Issue: OOM during DPO training**\n\nReduce batch size and sequence length:\n```python\nconfig = DPOConfig(\n    per_device_train_batch_size=1,  # Reduce from 4\n    max_length=512,  # Reduce from 1024\n    gradient_accumulation_steps=8  # Maintain effective batch\n)\n```\n\nOr use gradient checkpointing:\n```python\nmodel.gradient_checkpointing_enable()\n```\n\n**Issue: Poor alignment quality**\n\nTune beta parameter:\n```python\n# Higher beta = more conservative (stays closer to reference)\nconfig = DPOConfig(beta=0.5)  # Default 0.1\n\n# Lower beta = more aggressive alignment\nconfig = DPOConfig(beta=0.01)\n```\n\n**Issue: Reward model not learning**\n\nCheck loss type and learning rate:\n```python\nconfig = RewardConfig(\n    learning_rate=1e-5,  # Try different LR\n    num_train_epochs=3  # Train longer\n)\n```\n\nEnsure preference dataset has clear winners:\n```python\n# Verify dataset\nprint(dataset[0])\n# Should have clear chosen > rejected\n```\n\n**Issue: PPO training unstable**\n\nAdjust KL coefficient:\n```python\nconfig = PPOConfig(\n    kl_coef=0.1,  # Increase from 0.05\n    cliprange=0.1  # Reduce from 0.2\n)\n```\n\n## Advanced topics\n\n**SFT training guide**: See [references\u002Fsft-training.md](references\u002Fsft-training.md) for dataset formats, chat templates, packing strategies, and multi-GPU training.\n\n**DPO variants**: See [references\u002Fdpo-variants.md](references\u002Fdpo-variants.md) for IPO, cDPO, RPO, and other DPO loss functions with recommended hyperparameters.\n\n**Reward modeling**: See [references\u002Freward-modeling.md](references\u002Freward-modeling.md) for outcome vs process rewards, Bradley-Terry loss, and reward model evaluation.\n\n**Online RL methods**: See [references\u002Fonline-rl.md](references\u002Fonline-rl.md) for PPO, GRPO, RLOO, and OnlineDPO with detailed configurations.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA (CUDA required)\n- **VRAM**: Depends on model and method\n  - SFT 7B: 16GB (with LoRA)\n  - DPO 7B: 24GB (stores reference model)\n  - PPO 7B: 40GB (policy + reward model)\n  - GRPO 7B: 24GB (more memory efficient)\n- **Multi-GPU**: Supported via `accelerate`\n- **Mixed precision**: BF16 recommended (A100\u002FH100)\n\n**Memory optimization**:\n- Use LoRA\u002FQLoRA for all methods\n- Enable gradient checkpointing\n- Use smaller batch sizes with gradient accumulation\n\n## Resources\n\n- Docs: https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftrl\u002F\n- GitHub: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl\n- Papers:\n  - \"Training language models to follow instructions with human feedback\" (InstructGPT, 2022)\n  - \"Direct Preference Optimization: Your Language Model is Secretly a Reward Model\" (DPO, 2023)\n  - \"Group Relative Policy Optimization\" (GRPO, 2024)\n- Examples: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl\u002Ftree\u002Fmain\u002Fexamples\u002Fscripts\n\n## ? Связи\n\n- [[000 - Map of Maps]] — REPO_DOCS\n- [[000 - Map of Maps]] — Map of Maps\n\n","# TRL - Transformer Reinforcement Learning\n\n## Quick start\n\nTRL provides post-training methods for aligning language models with human preferences.\n\n**Installation**:\n```bash\npip install trl transformers datasets peft accelerate\n```\n\n**Supervised Fine-Tuning** (instruction tuning):\n```python\nfrom trl import SFTTrainer\n\ntrainer = SFTTrainer(\n    model=\"Qwen\u002FQwen2.5-0.5B\",\n    train_dataset=dataset,  # Prompt-completion pairs\n)\ntrainer.train()\n```\n\n**DPO** (align with preferences):\n```python\nfrom trl import DPOTrainer, DPOConfig\n\nconfig = DPOConfig(output_dir=\"model-dpo\", beta=0.1)\ntrainer = DPOTrainer(\n    model=model,\n    args=config,\n    train_dataset=preference_dataset,  # chosen\u002Frejected pairs\n    processing_class=tokenizer\n)\ntrainer.train()\n```\n\n## Common workflows\n\n### Workflow 1: Full RLHF pipeline (SFT → Reward Model → PPO)\n\nComplete pipeline from base model to human-aligned model.\n\nCopy this checklist:\n\n```\nRLHF Training:\n- [ ] Step 1: Supervised fine-tuning (SFT)\n- [ ] Step 2: Train reward model\n- [ ] Step 3: PPO reinforcement learning\n- [ ] Step 4: Evaluate aligned model\n```\n\n**Step 1: Supervised fine-tuning**\n\nTrain base model on instruction-following data:\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom trl import SFTTrainer, SFTConfig\nfrom datasets import load_dataset\n\n# Load model\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen\u002FQwen2.5-0.5B\")\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen\u002FQwen2.5-0.5B\")\n\n# Load instruction dataset\ndataset = load_dataset(\"trl-lib\u002FCapybara\", split=\"train\")\n\n# Configure training\ntraining_args = SFTConfig(\n    output_dir=\"Qwen2.5-0.5B-SFT\",\n    per_device_train_batch_size=4,\n    num_train_epochs=1,\n    learning_rate=2e-5,\n    logging_steps=10,\n    save_strategy=\"epoch\"\n)\n\n# Train\ntrainer = SFTTrainer(\n    model=model,\n    args=training_args,\n    train_dataset=dataset,\n    tokenizer=tokenizer\n)\ntrainer.train()\ntrainer.save_model()\n```\n\n**Step 2: Train reward model**\n\nTrain model to predict human preferences:\n\n```python\nfrom transformers import AutoModelForSequenceClassification\nfrom trl import RewardTrainer, RewardConfig\n\n# Load SFT model as base\nmodel = AutoModelForSequenceClassification.from_pretrained(\n    \"Qwen2.5-0.5B-SFT\",\n    num_labels=1  # Single reward score\n)\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen2.5-0.5B-SFT\")\n\n# Load preference data (chosen\u002Frejected pairs)\ndataset = load_dataset(\"trl-lib\u002Fultrafeedback_binarized\", split=\"train\")\n\n# Configure training\ntraining_args = RewardConfig(\n    output_dir=\"Qwen2.5-0.5B-Reward\",\n    per_device_train_batch_size=2,\n    num_train_epochs=1,\n    learning_rate=1e-5\n)\n\n# Train reward model\ntrainer = RewardTrainer(\n    model=model,\n    args=training_args,\n    processing_class=tokenizer,\n    train_dataset=dataset\n)\ntrainer.train()\ntrainer.save_model()\n```\n\n**Step 3: PPO reinforcement learning**\n\nOptimize policy using reward model:\n\n```bash\npython -m trl.scripts.ppo \\\n    --model_name_or_path Qwen2.5-0.5B-SFT \\\n    --reward_model_path Qwen2.5-0.5B-Reward \\\n    --dataset_name trl-internal-testing\u002Fdescriptiveness-sentiment-trl-style \\\n    --output_dir Qwen2.5-0.5B-PPO \\\n    --learning_rate 3e-6 \\\n    --per_device_train_batch_size 64 \\\n    --total_episodes 10000\n```\n\n**Step 4: Evaluate**\n\n```python\nfrom transformers import pipeline\n\n# Load aligned model\ngenerator = pipeline(\"text-generation\", model=\"Qwen2.5-0.5B-PPO\")\n\n# Test\nprompt = \"Explain quantum computing to a 10-year-old\"\noutput = generator(prompt, max_length=200)[0][\"generated_text\"]\nprint(output)\n```\n\n### Workflow 2: Simple preference alignment with DPO\n\nAlign model with preferences without reward model.\n\nCopy this checklist:\n\n```\nDPO Training:\n- [ ] Step 1: Prepare preference dataset\n- [ ] Step 2: Configure DPO\n- [ ] Step 3: Train with DPOTrainer\n- [ ] Step 4: Evaluate alignment\n```\n\n**Step 1: Prepare preference dataset**\n\nDataset format:\n```json\n{\n  \"prompt\": \"What is the capital of France?\",\n  \"chosen\": \"The capital of France is Paris.\",\n  \"rejected\": \"I don't know.\"\n}\n```\n\nLoad dataset:\n```python\nfrom datasets import load_dataset\n\ndataset = load_dataset(\"trl-lib\u002Fultrafeedback_binarized\", split=\"train\")\n# Or load your own\n# dataset = load_dataset(\"json\", data_files=\"preferences.json\")\n```\n\n**Step 2: Configure DPO**\n\n```python\nfrom trl import DPOConfig\n\nconfig = DPOConfig(\n    output_dir=\"Qwen2.5-0.5B-DPO\",\n    per_device_train_batch_size=4,\n    num_train_epochs=1,\n    learning_rate=5e-7,\n    beta=0.1,  # KL penalty strength\n    max_prompt_length=512,\n    max_length=1024,\n    logging_steps=10\n)\n```\n\n**Step 3: Train with DPOTrainer**\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom trl import DPOTrainer\n\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen\u002FQwen2.5-0.5B-Instruct\")\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen\u002FQwen2.5-0.5B-Instruct\")\n\ntrainer = DPOTrainer(\n    model=model,\n    args=config,\n    train_dataset=dataset,\n    processing_class=tokenizer\n)\n\ntrainer.train()\ntrainer.save_model()\n```\n\n**CLI alternative**:\n```bash\ntrl dpo \\\n    --model_name_or_path Qwen\u002FQwen2.5-0.5B-Instruct \\\n    --dataset_name argilla\u002FCapybara-Preferences \\\n    --output_dir Qwen2.5-0.5B-DPO \\\n    --per_device_train_batch_size 4 \\\n    --learning_rate 5e-7 \\\n    --beta 0.1\n```\n\n### Workflow 3: Memory-efficient online RL with GRPO\n\nTrain with reinforcement learning using minimal memory.\n\nCopy this checklist:\n\n```\nGRPO Training:\n- [ ] Step 1: Define reward function\n- [ ] Step 2: Configure GRPO\n- [ ] Step 3: Train with GRPOTrainer\n```\n\n**Step 1: Define reward function**\n\n```python\ndef reward_function(completions, **kwargs):\n    \"\"\"\n    Compute rewards for completions.\n\n    Args:\n        completions: List of generated texts\n\n    Returns:\n        List of reward scores (floats)\n    \"\"\"\n    rewards = []\n    for completion in completions:\n        # Example: reward based on length and unique words\n        score = len(completion.split())  # Favor longer responses\n        score += len(set(completion.lower().split()))  # Reward unique words\n        rewards.append(score)\n    return rewards\n```\n\nOr use a reward model:\n```python\nfrom transformers import pipeline\n\nreward_model = pipeline(\"text-classification\", model=\"reward-model-path\")\n\ndef reward_from_model(completions, prompts, **kwargs):\n    # Combine prompt + completion\n    full_texts = [p + c for p, c in zip(prompts, completions)]\n    # Get reward scores\n    results = reward_model(full_texts)\n    return [r[\"score\"] for r in results]\n```\n\n**Step 2: Configure GRPO**\n\n```python\nfrom trl import GRPOConfig\n\nconfig = GRPOConfig(\n    output_dir=\"Qwen2-GRPO\",\n    per_device_train_batch_size=4,\n    num_train_epochs=1,\n    learning_rate=1e-5,\n    num_generations=4,  # Generate 4 completions per prompt\n    max_new_tokens=128\n)\n```\n\n**Step 3: Train with GRPOTrainer**\n\n```python\nfrom datasets import load_dataset\nfrom trl import GRPOTrainer\n\n# Load prompt-only dataset\ndataset = load_dataset(\"trl-lib\u002Ftldr\", split=\"train\")\n\ntrainer = GRPOTrainer(\n    model=\"Qwen\u002FQwen2-0.5B-Instruct\",\n    reward_funcs=reward_function,  # Your reward function\n    args=config,\n    train_dataset=dataset\n)\n\ntrainer.train()\n```\n\n**CLI**:\n```bash\ntrl grpo \\\n    --model_name_or_path Qwen\u002FQwen2-0.5B-Instruct \\\n    --dataset_name trl-lib\u002Ftldr \\\n    --output_dir Qwen2-GRPO \\\n    --num_generations 4\n```\n\n## When to use vs alternatives\n\n**Use TRL when:**\n- Need to align model with human preferences\n- Have preference data (chosen\u002Frejected pairs)\n- Want to use reinforcement learning (PPO, GRPO)\n- Need reward model training\n- Doing RLHF (full pipeline)\n\n**Method selection**:\n- **SFT**: Have prompt-completion pairs, want basic instruction following\n- **DPO**: Have preferences, want simple alignment (no reward model needed)\n- **PPO**: Have reward model, need maximum control over RL\n- **GRPO**: Memory-constrained, want online RL\n- **Reward Model**: Building RLHF pipeline, need to score generations\n\n**Use alternatives instead:**\n- **HuggingFace Trainer**: Basic fine-tuning without RL\n- **Axolotl**: YAML-based training configuration\n- **LitGPT**: Educational, minimal fine-tuning\n- **Unsloth**: Fast LoRA training\n\n## Common issues\n\n**Issue: OOM during DPO training**\n\nReduce batch size and sequence length:\n```python\nconfig = DPOConfig(\n    per_device_train_batch_size=1,  # Reduce from 4\n    max_length=512,  # Reduce from 1024\n    gradient_accumulation_steps=8  # Maintain effective batch\n)\n```\n\nOr use gradient checkpointing:\n```python\nmodel.gradient_checkpointing_enable()\n```\n\n**Issue: Poor alignment quality**\n\nTune beta parameter:\n```python\n# Higher beta = more conservative (stays closer to reference)\nconfig = DPOConfig(beta=0.5)  # Default 0.1\n\n# Lower beta = more aggressive alignment\nconfig = DPOConfig(beta=0.01)\n```\n\n**Issue: Reward model not learning**\n\nCheck loss type and learning rate:\n```python\nconfig = RewardConfig(\n    learning_rate=1e-5,  # Try different LR\n    num_train_epochs=3  # Train longer\n)\n```\n\nEnsure preference dataset has clear winners:\n```python\n# Verify dataset\nprint(dataset[0])\n# Should have clear chosen > rejected\n```\n\n**Issue: PPO training unstable**\n\nAdjust KL coefficient:\n```python\nconfig = PPOConfig(\n    kl_coef=0.1,  # Increase from 0.05\n    cliprange=0.1  # Reduce from 0.2\n)\n```\n\n## Advanced topics\n\n**SFT training guide**: See [references\u002Fsft-training.md](references\u002Fsft-training.md) for dataset formats, chat templates, packing strategies, and multi-GPU training.\n\n**DPO variants**: See [references\u002Fdpo-variants.md](references\u002Fdpo-variants.md) for IPO, cDPO, RPO, and other DPO loss functions with recommended hyperparameters.\n\n**Reward modeling**: See [references\u002Freward-modeling.md](references\u002Freward-modeling.md) for outcome vs process rewards, Bradley-Terry loss, and reward model evaluation.\n\n**Online RL methods**: See [references\u002Fonline-rl.md](references\u002Fonline-rl.md) for PPO, GRPO, RLOO, and OnlineDPO with detailed configurations.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA (CUDA required)\n- **VRAM**: Depends on model and method\n  - SFT 7B: 16GB (with LoRA)\n  - DPO 7B: 24GB (stores reference model)\n  - PPO 7B: 40GB (policy + reward model)\n  - GRPO 7B: 24GB (more memory efficient)\n- **Multi-GPU**: Supported via `accelerate`\n- **Mixed precision**: BF16 recommended (A100\u002FH100)\n\n**Memory optimization**:\n- Use LoRA\u002FQLoRA for all methods\n- Enable gradient checkpointing\n- Use smaller batch sizes with gradient accumulation\n\n## Resources\n\n- Docs: https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftrl\u002F\n- GitHub: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl\n- Papers:\n  - \"Training language models to follow instructions with human feedback\" (InstructGPT, 2022)\n  - \"Direct Preference Optimization: Your Language Model is Secretly a Reward Model\" (DPO, 2023)\n  - \"Group Relative Policy Optimization\" (GRPO, 2024)\n- Examples: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl\u002Ftree\u002Fmain\u002Fexamples\u002Fscripts\n\n## ? Связи\n\n- [[000 - Map of Maps]] — REPO_DOCS\n- [[000 - Map of Maps]] — Map of Maps\n\n",{"name":5,"author":44,"license":15,"version":45,"metadata":46,"description":7,"dependencies":57},"Orchestra Research","1.0.0",{"hermes":47},{"tags":48},[36,20,49,50,51,52,53,54,35,55,56],"Reinforcement Learning","Fine-Tuning","SFT","DPO","PPO","GRPO","Preference Alignment","HuggingFace",[58,59,60,61,62,63],"trl","transformers","datasets","peft","accelerate","torch","main","bdd9f958d7bb1a0f19b8ad8da020b0bab0f53eb1","vc-kit\u002Fcontent\u002FCOMBINED\u002FCOMBINED\u002FREPO_DOCS\u002F02-hermes-agent\u002Fskills\u002Fmlops\u002Ftraining\u002Ftrl-fine-tuning",1,"https:\u002F\u002Fgithub.com\u002Fibragimov-oasis\u002Fvibe-coder","https:\u002F\u002Fgithub.com\u002Fibragimov-oasis","2026-06-24 16:31:18",{"source":8,"skillName":5,"commands":72,"prompt":82,"zipAvailable":83},[73,76,79],{"tool":74,"command":75},"npx","npx skills add ibragimov-oasis\u002Fvibe-coder --skill fine-tuning-with-trl",{"tool":77,"command":78},"bunx","bunx skills add ibragimov-oasis\u002Fvibe-coder --skill fine-tuning-with-trl",{"tool":80,"command":81},"pnpm","pnpm dlx skills add ibragimov-oasis\u002Fvibe-coder --skill fine-tuning-with-trl","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fibragimov-oasis\u002Fvibe-coder\u002Fblob\u002Fmain\u002Fvc-kit\u002Fcontent\u002FCOMBINED\u002FCOMBINED\u002FREPO_DOCS\u002F02-hermes-agent\u002Fskills\u002Fmlops\u002Ftraining\u002Ftrl-fine-tuning\u002FSKILL.md",true,[],[86,123,150,170,196,226],{"skillNo":87,"name":88,"nameZh":6,"description":89,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":90,"categories":91,"keywords":102,"publishedAt":39},"01M6Q4Q22KKKVMJJYA411DQ4ER","ckm:ui-styling","Create beautiful, accessible user interfaces with shadcn\u002Fui components (built on Radix UI + Tailwind), Tailwind CSS utility-first styling, and canvas-based visual designs. Use when building user interfaces, implementing design systems, creating responsive layouts, adding accessible components (dialogs, dropdowns, forms, tables), customizing themes and colors, implementing dark mode, generating visual designs and posters, or establishing consistent styling patterns across applications.","https:\u002F\u002Fgithub.com\u002Fibragimov-oasis\u002Fvibe-coder\u002Fblob\u002Fmain\u002Fvc-kit\u002Fcontent\u002FCOMBINED\u002FCOMBINED\u002Fui-design\u002Fui-rules\u002Fui-ux-pro-max\u002Fclaude\u002Fskills\u002Fui-styling\u002FSKILL.md",[92,97],{"categoryNo":93,"categoryName":94,"taxonomyType":95,"taxonomyTypeDesc":96},"01JBSEEDCAT000000000000015","视觉设计","SCENARIO","使用场景",{"categoryNo":98,"categoryName":99,"taxonomyType":100,"taxonomyTypeDesc":101},"01JBSEEDCAT000000000000032","设计师","PROFESSION","适用职业",[103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122],"ui styling","ui","styling","ckm:ui styling","ckm","design","css","canvas","components","component","create","design systems","Tailwind","across","themes","utility","mode","implementing","artifact\u002Fui-component","source\u002Fui-design",{"skillNo":124,"name":125,"nameZh":6,"description":126,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":127,"categories":128,"keywords":137,"publishedAt":39},"01M6Q4PQ2NX4NDVWTEHA3MQ2R0","ckm:design","Comprehensive design skill: brand identity, design tokens, UI styling, logo generation (55 styles, Gemini AI), corporate identity program (50 deliverables, CIP mockups), HTML presentations (Chart.js), banner design (22 styles, social\u002Fads\u002Fweb\u002Fprint), icon design (15 styles, SVG, Gemini 3.1 Pro), social photos (HTML→screenshot, multi-platform). Actions: design logo, create CIP, generate mockups, build slides, design banner, generate icon, create social photos, social media images, brand identity, design system. Platforms: Facebook, Twitter, LinkedIn, YouTube, Instagram, Pinterest, TikTok, Threads, Google Ads.","https:\u002F\u002Fgithub.com\u002Fibragimov-oasis\u002Fvibe-coder\u002Fblob\u002Fmain\u002Fvc-kit\u002Fcontent\u002FCOMBINED\u002FCOMBINED\u002Fui-design\u002Fui-rules\u002Fui-ux-pro-max\u002Fclaude\u002Fskills\u002Fdesign\u002FSKILL.md",[129,132,133,136],{"categoryNo":130,"categoryName":131,"taxonomyType":95,"taxonomyTypeDesc":96},"01JBSEEDCAT000000000000003","演示文稿",{"categoryNo":93,"categoryName":94,"taxonomyType":95,"taxonomyTypeDesc":96},{"categoryNo":134,"categoryName":135,"taxonomyType":95,"taxonomyTypeDesc":96},"01JBSEEDCAT000000000000023","Skill与智能体开发",{"categoryNo":98,"categoryName":99,"taxonomyType":100,"taxonomyTypeDesc":101},[108,125,107,138,112,103,139,113,140,141,142,143,144,145,146,147,148,149,121,122],"google","presentations","generate","screenshot","actions","Media","logo","ads","youtube","tiktok","artifact","chart",{"skillNo":151,"name":152,"nameZh":6,"description":153,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":154,"categories":155,"keywords":158,"publishedAt":39},"01M6Q4PJE27GEA3M0CECEBCZKS","ckm:design-system","Token architecture, component specifications, and slide generation. Three-layer tokens (primitive→semantic→component), CSS variables, spacing\u002Ftypography scales, component specs, strategic slide creation. Use for design tokens, systematic design, brand-compliant presentations.","https:\u002F\u002Fgithub.com\u002Fibragimov-oasis\u002Fvibe-coder\u002Fblob\u002Fmain\u002Fvc-kit\u002Fcontent\u002FCOMBINED\u002FCOMBINED\u002Fui-design\u002Fui-rules\u002Fui-ux-pro-max\u002Fclaude\u002Fskills\u002Fdesign-system\u002FSKILL.md",[156,157],{"categoryNo":93,"categoryName":94,"taxonomyType":95,"taxonomyTypeDesc":96},{"categoryNo":98,"categoryName":99,"taxonomyType":100,"taxonomyTypeDesc":101},[108,159,160,161,125,107,109,112,162,139,148,163,164,165,166,167,168,169,121,122],"design system","system","ckm:design system","architecture","token","creation","systematic","css variables","Variables","spacing","layer",{"skillNo":171,"name":172,"nameZh":6,"description":173,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":174,"categories":175,"keywords":181,"publishedAt":39},"01M6Q4P90JXFYMH2S634QM6SQJ","ckm:banner-design","Design banners for social media, ads, website heroes, creative assets, and print. Multiple art direction options with AI-generated visuals. Actions: design, create, generate banner. Platforms: Facebook, Twitter\u002FX, LinkedIn, YouTube, Instagram, Google Display, website hero, print. Styles: minimalist, gradient, bold typography, photo-based, illustrated, geometric, retro, glassmorphism, 3D, neon, duotone, editorial, collage. Uses ui-ux-pro-max, frontend-design, ai-artist, ai-multimodal skills.","https:\u002F\u002Fgithub.com\u002Fibragimov-oasis\u002Fvibe-coder\u002Fblob\u002Fmain\u002Fvc-kit\u002Fcontent\u002FCOMBINED\u002FCOMBINED\u002Fui-design\u002Fui-rules\u002Fui-ux-pro-max\u002Fclaude\u002Fskills\u002Fbanner-design\u002FSKILL.md",[176,179,180],{"categoryNo":177,"categoryName":178,"taxonomyType":95,"taxonomyTypeDesc":96},"01JBSEEDCAT000000000000009","前端开发",{"categoryNo":93,"categoryName":94,"taxonomyType":95,"taxonomyTypeDesc":96},{"categoryNo":98,"categoryName":99,"taxonomyType":100,"taxonomyTypeDesc":101},[108,182,183,184,107,185,142,143,186,187,188,189,190,191,192,193,121,122,194,195],"banner design","ckm:banner design","banner","frontend","3D","generated","Art","social","dimensions","neon","direction","display","frontend-design","hero",{"skillNo":197,"name":198,"nameZh":6,"description":199,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":200,"categories":201,"keywords":208,"publishedAt":39},"01M6Q2RZXBJ9TBE413GXCYQNDB","seo","Comprehensive SEO analysis for any website or business type. Performs full site audits, single-page deep analysis, technical SEO checks (crawlability, indexability, Core Web Vitals with INP), schema markup detection\u002Fvalidation\u002Fgeneration, content quality assessment (E-E-A-T framework per Dec 2025 update extending to all competitive queries), image optimization, sitemap analysis, and Generative Engine Optimization (GEO) for AI Overviews, ChatGPT, and Perplexity citations. Analyzes AI crawler accessibility (GPTBot, ClaudeBot, PerplexityBot), llms.txt compliance, brand mention signals, and passage-level citability. Industry detection for SaaS, e-commerce, local business, publishers, agencies. Triggers on: \"SEO\", \"audit\", \"schema\", \"Core Web Vitals\", \"sitemap\", \"E-E-A-T\", \"AI Overviews\", \"GEO\", \"technical SEO\", \"content quality\", \"page speed\", \"structured data\".","https:\u002F\u002Fgithub.com\u002Fibragimov-oasis\u002Fvibe-coder\u002Fblob\u002Fmain\u002Fvc-kit\u002Fcontent\u002FCOMBINED\u002FCOMBINED\u002Fskills\u002Fskills-seo\u002Fseo\u002Fclaude-seo\u002Fskills\u002Fseo\u002FSKILL.md",[202,205],{"categoryNo":203,"categoryName":204,"taxonomyType":95,"taxonomyTypeDesc":96},"01JBSEEDCAT000000000000019","营销推广",{"categoryNo":206,"categoryName":207,"taxonomyType":100,"taxonomyTypeDesc":101},"01JBSEEDCAT000000000000035","市场营销",[198,209,25,210,211,212,213,148,214,215,216,217,218,219,220,221,222,223,224,225],"image","assessment","deep analysis","Citations","analysis","local","deep","crawlability","seo analysis","2025","commerce","URL","domain\u002Fseo","source\u002Fskills-seo","vitals","generative","publishers",{"skillNo":227,"name":228,"nameZh":6,"description":229,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":230,"categories":231,"keywords":234,"publishedAt":39},"01M6Q2RVE7VGFCFQKBK468NHVY","seo-technical","Technical SEO audit across 9 categories: crawlability, indexability, security, URL structure, mobile, Core Web Vitals, structured data, JavaScript rendering, and IndexNow protocol. Use when user says \"technical SEO\", \"crawl issues\", \"robots.txt\", \"Core Web Vitals\", \"site speed\", or \"security headers\".","https:\u002F\u002Fgithub.com\u002Fibragimov-oasis\u002Fvibe-coder\u002Fblob\u002Fmain\u002Fvc-kit\u002Fcontent\u002FCOMBINED\u002FCOMBINED\u002Fskills\u002Fskills-seo\u002Fseo\u002Fclaude-seo\u002Fskills\u002Fseo-technical\u002FSKILL.md",[232,233],{"categoryNo":203,"categoryName":204,"taxonomyType":95,"taxonomyTypeDesc":96},{"categoryNo":206,"categoryName":207,"taxonomyType":100,"taxonomyTypeDesc":101},[198,235,236,237,238,239,240,241,242,148,116,216,243,244,220,245,246,221,222,223],"technical","seo technical","issues","audit","data","mobile","crawl","structure","rendering","technical seo audit","headers","categories"]