[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6MBC0233XK6WRSFGEQ37SRT":3,"skill-same:01M6MBC0233XK6WRSFGEQ37SRT":84},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":50,"commitSha":51,"skillDirectory":52,"forksCount":53,"repositoryHtmlUrl":54,"repositoryHomepageUrl":55,"ownerHtmlUrl":56,"ownerBlogUrl":57,"ownerDescription":58,"updatedAt":59,"install":60,"resources":73},"01M6MBC0233XK6WRSFGEQ37SRT","verl-rl-training",null,"Provides guidance for training LLMs with reinforcement learning using verl (Volcano Engine RL). Use when implementing RLHF, GRPO, PPO, or other RL algorithms for LLM post-training at scale with flexible infrastructure backends.","hashgraph-online\u002Fawesome-codex-plugins","hashgraph-online","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F160371962?v=4","01M6CNB6NRJVNNGN14TR01Q6D3","HOL (Hashgraph Online)","01M6JT1HSKKG9V68KQAQGBA7XP",1575,"MIT","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fverl-rl-training\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"training","verl rl training","verl","rl","engine","for","implementing","at","post","algorithms","use","distributed","learning","vLLM","Ray","PPO","RLHF","Post-Training","GRPO","infrastructure","2026-11-10 10:12:30",false,"---\nname: verl-rl-training\ndescription: Provides guidance for training LLMs with reinforcement learning using verl (Volcano Engine RL). Use when implementing RLHF, GRPO, PPO, or other RL algorithms for LLM post-training at scale with flexible infrastructure backends.\nlicense: MIT\nmetadata:\n  version: 1.0.0\n  category: ml-training\n  maintainer: Kalaris Labs\n  tags: Reinforcement Learning, RLHF, GRPO, PPO, Post-Training, Distributed Training\n  dependencies: verl>=0.3.0, torch>=2.0.0, ray>=2.41.0, vllm>=0.8.2, transformers>=4.40.0\n---\n\n# verl: Volcano Engine Reinforcement Learning for LLMs\n\nverl is a flexible, efficient, and production-ready RL training library for large language models from ByteDance's Seed team. It implements the HybridFlow framework (EuroSys 2025) and powers models like Doubao-1.5-pro achieving O1-level performance on math benchmarks.\n\n## When to Use verl\n\n**Choose verl when you need:**\n- Production-ready RL training at scale (tested up to 671B parameters)\n- Flexibility to swap backends (FSDP ↔ Megatron-LM ↔ vLLM ↔ SGLang)\n- Support for multiple RL algorithms (PPO, GRPO, RLOO, REINFORCE++, DAPO)\n- Multi-turn rollout with tool calling for agentic workflows\n- Vision-language model RL training\n\n**Consider alternatives when:**\n- You need Megatron-native training → use **slime** or **miles**\n- You want PyTorch-native abstractions with Monarch → use **torchforge**\n- You only need simple SFT\u002FDPO → use **TRL** or **Axolotl**\n\n## Key Features\n\n- **Training backends**: FSDP, FSDP2, Megatron-LM\n- **Rollout engines**: vLLM, SGLang, HuggingFace Transformers\n- **Algorithms**: PPO, GRPO, DAPO, RLOO, ReMax, REINFORCE++, SPIN, SPPO\n- **Models**: Qwen-3, Llama-3.1, DeepSeek, Gemma-2 (0.5B to 671B)\n- **Advanced**: LoRA RL, sequence parallelism, expert parallelism, multi-turn tools\n\n## Installation\n\n```bash\n# Option 1: pip install\npip install verl[vllm]  # or verl[sglang] for SGLang backend\n\n# Option 2: Docker (recommended for production)\ndocker pull verlai\u002Fverl:vllm011.latest\n\n# Option 3: From source\ngit clone https:\u002F\u002Fgithub.com\u002Fvolcengine\u002Fverl.git\ncd verl && pip install -e .[vllm,math]\n```\n\n## Quick Start: GRPO Training\n\n```bash\npython3 -m verl.trainer.main_ppo \\\n    algorithm.adv_estimator=grpo \\\n    data.train_files=~\u002Fdata\u002Fgsm8k\u002Ftrain.parquet \\\n    actor_rollout_ref.model.path=Qwen\u002FQwen2.5-7B \\\n    actor_rollout_ref.rollout.n=8 \\\n    actor_rollout_ref.actor.use_kl_loss=True \\\n    trainer.n_gpus_per_node=8\n```\n\n## Core Architecture\n\nverl uses a **HybridFlow** programming model separating control flow from computation:\n\n```\n┌─────────────────────────────────────────────────────────┐\n│ Single-Process Controller (Ray)                         │\n│ - Orchestrates: rollout → reward → train → sync        │\n└─────────────────────┬───────────────────────────────────┘\n                      │\n┌─────────────────────▼───────────────────────────────────┐\n│ Multi-Process Workers                                   │\n│ ├── ActorRolloutRefWorker (policy + generation)        │\n│ ├── CriticWorker (value estimation, PPO only)          │\n│ └── RewardManager (model-based or rule-based rewards)  │\n└─────────────────────────────────────────────────────────┘\n```\n\n---\n\n## Workflow 1: Math Reasoning with GRPO\n\nUse this workflow for training reasoning models on math tasks like GSM8K or MATH.\n\n### Prerequisites Checklist\n- [ ] GPU cluster with 8+ GPUs (H100 recommended)\n- [ ] Dataset in parquet format with `prompt` and `reward_model` columns\n- [ ] Base model from HuggingFace Hub\n\n### Step 1: Prepare Dataset\n\n```python\nimport pandas as pd\n\ndata = [\n    {\n        \"prompt\": [{\"role\": \"user\", \"content\": \"What is 15 + 27?\"}],\n        \"reward_model\": {\"ground_truth\": \"42\"}\n    },\n    # ... more examples\n]\ndf = pd.DataFrame(data)\ndf.to_parquet(\"train.parquet\")\n```\n\n### Step 2: Define Reward Function\n\n```python\n# reward_function.py\nimport re\n\ndef compute_reward(responses, ground_truths):\n    rewards = []\n    for response, gt in zip(responses, ground_truths):\n        # Extract answer from response\n        match = re.search(r'\\\\boxed{([^}]+)}', response)\n        if match and match.group(1).strip() == gt.strip():\n            rewards.append(1.0)\n        else:\n            rewards.append(0.0)\n    return rewards\n```\n\n### Step 3: Create Training Config\n\n```yaml\n# config\u002Fgrpo_math.yaml\nalgorithm:\n  adv_estimator: grpo\n  gamma: 1.0\n  lam: 1.0\n\ndata:\n  train_files: \u002Fpath\u002Fto\u002Ftrain.parquet\n  val_files: \u002Fpath\u002Fto\u002Fval.parquet\n  train_batch_size: 256\n  max_prompt_length: 512\n  max_response_length: 2048\n\nactor_rollout_ref:\n  model:\n    path: Qwen\u002FQwen2.5-7B-Instruct\n  actor:\n    use_kl_loss: true\n    kl_loss_coef: 0.001\n    ppo_mini_batch_size: 64\n  rollout:\n    name: vllm\n    n: 8  # samples per prompt\n    temperature: 0.7\n    top_p: 0.95\n\ntrainer:\n  total_epochs: 3\n  n_gpus_per_node: 8\n  save_freq: 100\n```\n\n### Step 4: Launch Training\n\n```bash\npython3 -m verl.trainer.main_ppo \\\n    --config-path config \\\n    --config-name grpo_math \\\n    trainer.experiment_name=grpo_math_qwen7b\n```\n\n### Step 5: Monitor and Validate\n- [ ] Check WandB\u002FTensorBoard for loss curves\n- [ ] Verify reward is increasing over steps\n- [ ] Run evaluation on held-out test set\n\n---\n\n## Workflow 2: PPO with Critic Model\n\nUse this workflow when you need value-based advantage estimation (GAE).\n\n### Key Differences from GRPO\n- Requires separate critic model\n- Uses Generalized Advantage Estimation (GAE)\n- Better for tasks with dense rewards\n\n### Configuration\n\n```yaml\nalgorithm:\n  adv_estimator: gae  # Use GAE instead of GRPO\n  gamma: 0.99\n  lam: 0.95\n\ncritic:\n  model:\n    path: Qwen\u002FQwen2.5-7B-Instruct  # Can be same or different from actor\n  ppo_mini_batch_size: 64\n\nactor_rollout_ref:\n  actor:\n    use_kl_loss: true\n    kl_loss_coef: 0.02\n    clip_ratio: 0.2  # PPO clipping\n```\n\n### Launch with Critic\n\n```bash\npython3 -m verl.trainer.main_ppo \\\n    algorithm.adv_estimator=gae \\\n    critic.model.path=Qwen\u002FQwen2.5-7B-Instruct \\\n    trainer.n_gpus_per_node=8\n```\n\n---\n\n## Workflow 3: Large-Scale Training with Megatron\n\nUse this workflow for models >70B parameters or when you need expert parallelism.\n\n### Prerequisites\n- [ ] Install Megatron-LM bridge: `pip install mbridge`\n- [ ] Convert model to Megatron format\n- [ ] Multi-node cluster with NVLink\u002FInfiniBand\n\n### Configuration for 70B+ Models\n\n```yaml\nactor_rollout_ref:\n  model:\n    path: \u002Fpath\u002Fto\u002Fmegatron\u002Fcheckpoint\n    backend: megatron\n  actor:\n    strategy: megatron\n    tensor_model_parallel_size: 8\n    pipeline_model_parallel_size: 2\n  rollout:\n    name: vllm\n    tensor_parallel_size: 8\n```\n\n### Launch Multi-Node\n\n```bash\n# On head node\nray start --head --port=6379\n\n# On worker nodes\nray start --address='head_ip:6379'\n\n# Launch training\npython3 -m verl.trainer.main_ppo \\\n    trainer.nnodes=4 \\\n    trainer.n_gpus_per_node=8\n```\n\n---\n\n## Configuration Reference\n\n### Algorithm Selection\n\n| Algorithm | `adv_estimator` | Use Case |\n|-----------|-----------------|----------|\n| GRPO | `grpo` | Critic-free, math\u002Freasoning |\n| PPO\u002FGAE | `gae` | Dense rewards, value estimation |\n| REINFORCE++ | `reinforce_plus_plus` | Variance reduction |\n| RLOO | `rloo` | Leave-one-out baseline |\n| ReMax | `remax` | Maximum reward baseline |\n| OPO | `opo` | Optimal policy optimization |\n\n### Key Parameters\n\n```yaml\n# Rollout parameters\nactor_rollout_ref.rollout.n: 8              # Samples per prompt\nactor_rollout_ref.rollout.temperature: 0.7  # Sampling temperature\nactor_rollout_ref.rollout.top_p: 0.95       # Nucleus sampling\n\n# Training parameters\nactor_rollout_ref.actor.lr: 1e-6            # Learning rate\nactor_rollout_ref.actor.ppo_mini_batch_size: 64\nactor_rollout_ref.actor.clip_ratio: 0.2     # PPO clip range\n\n# KL control\nactor_rollout_ref.actor.use_kl_loss: true\nactor_rollout_ref.actor.kl_loss_coef: 0.001\nalgorithm.kl_ctrl.target_kl: 0.1            # For adaptive KL control\n```\n\n---\n\n## Common Issues and Solutions\n\n### Issue: OOM During Rollout\n\n**Symptoms**: CUDA out of memory during generation phase\n\n**Solutions**:\n```yaml\n# Reduce batch size\nactor_rollout_ref.rollout.log_prob_micro_batch_size: 4\n\n# Enable gradient checkpointing\nactor_rollout_ref.model.enable_gradient_checkpointing: true\n\n# Use FSDP2 with CPU offloading\nactor_rollout_ref.actor.strategy: fsdp2\nactor_rollout_ref.actor.fsdp_config.offload_policy: true\n```\n\n### Issue: Training Instability\n\n**Symptoms**: Loss spikes, reward collapse\n\n**Solutions**:\n```yaml\n# Reduce learning rate\nactor_rollout_ref.actor.lr: 5e-7\n\n# Increase KL penalty\nactor_rollout_ref.actor.kl_loss_coef: 0.01\n\n# Enable gradient clipping\nactor_rollout_ref.actor.max_grad_norm: 1.0\n```\n\n### Issue: Slow Weight Sync\n\n**Symptoms**: Long pauses between rollout and training\n\n**Solutions**:\n```bash\n# Use FSDP2 for faster resharding\nactor_rollout_ref.actor.strategy=fsdp2\n\n# Enable async weight transfer\ntrainer.async_weight_update=true\n```\n\n### Issue: vLLM Version Mismatch\n\n**Symptoms**: Import errors or generation failures\n\n**Solution**: Use compatible versions:\n```bash\npip install vllm>=0.8.5,\u003C=0.12.0\n# Avoid vLLM 0.7.x (known bugs)\n```\n\n---\n\n## Advanced Topics\n\n### Multi-Turn Tool Calling\n\nSee references\u002Fmulti-turn.md for agentic workflows with tool use.\n\n### Vision-Language Models\n\n```yaml\nactor_rollout_ref:\n  model:\n    path: Qwen\u002FQwen2.5-VL-7B-Instruct\n  rollout:\n    name: vllm\n    enable_vision: true\n```\n\n### LoRA Training\n\n```yaml\nactor_rollout_ref:\n  actor:\n    lora:\n      enabled: true\n      r: 16\n      alpha: 32\n      target_modules: [\"q_proj\", \"v_proj\"]\n```\n\n---\n\n## Resources\n\n- **Documentation**: https:\u002F\u002Fverl.readthedocs.io\u002F\n- **Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2409.19256\n- **GitHub**: https:\u002F\u002Fgithub.com\u002Fvolcengine\u002Fverl\n- **Recipes**: https:\u002F\u002Fgithub.com\u002Fverl-project\u002Fverl-recipe (DAPO, GSPO, etc.)\n- **Community**: Slack at verl-project\n\n## Agent operating procedure\n\n1. **Check the environment.** Check GPU type, memory and driver\u002FCUDA versions (`nvidia-smi`), framework versions, and dataset location and size.\n2. **Pin down the inputs.** Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.\n3. **Run a small version first.** Do a smoke run: tiny model or subset, few steps, and confirm loss decreases and checkpoints save.\n4. **Execute the full task** using the instructions and references above.\n5. **Validate the result.** Track metrics on held-out data, compare against a baseline, and record seeds, configs and hardware.\n6. **Report.** State what was run (versions, commands, parameters), what was checked, and what is still uncertain.\n\n| If this happens | Do this |\n|---|---|\n| CUDA out-of-memory | Reduce batch size, enable gradient accumulation\u002Fcheckpointing or mixed precision, or shard the model. |\n| Loss is NaN or diverges | Lower the learning rate, check data for invalid values, and enable gradient clipping. |\n| A function, flag or endpoint in these instructions is missing in the installed version | Check the installed version's own documentation (`help()`, `--help`, official docs), adapt, and tell the user. Never invent an API. |\n| A required input, identifier or parameter is ambiguous | Ask the user, or state the assumption explicitly before running. |\n\n**Integrity rules**\n\n- Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.\n- Never claim training results without logs; estimate compute cost before launching large jobs.\n- Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.\n- Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.\n\n## Related skills\n\n- `fine-tuning-with-trl`: Fine-tune LLMs using reinforcement learning with TRL - SFT for instruction tuning, DPO for preference alignment, PPO\u002FGRPO for reward optimi…\n- `slime-rl-training`: Provides guidance for LLM post-training with RL using slime, a Megatron+SGLang framework.\n- `openrlhf-training`: High-performance RLHF framework with Ray+vLLM acceleration.\n","# verl: Volcano Engine Reinforcement Learning for LLMs\n\nverl is a flexible, efficient, and production-ready RL training library for large language models from ByteDance's Seed team. It implements the HybridFlow framework (EuroSys 2025) and powers models like Doubao-1.5-pro achieving O1-level performance on math benchmarks.\n\n## When to Use verl\n\n**Choose verl when you need:**\n- Production-ready RL training at scale (tested up to 671B parameters)\n- Flexibility to swap backends (FSDP ↔ Megatron-LM ↔ vLLM ↔ SGLang)\n- Support for multiple RL algorithms (PPO, GRPO, RLOO, REINFORCE++, DAPO)\n- Multi-turn rollout with tool calling for agentic workflows\n- Vision-language model RL training\n\n**Consider alternatives when:**\n- You need Megatron-native training → use **slime** or **miles**\n- You want PyTorch-native abstractions with Monarch → use **torchforge**\n- You only need simple SFT\u002FDPO → use **TRL** or **Axolotl**\n\n## Key Features\n\n- **Training backends**: FSDP, FSDP2, Megatron-LM\n- **Rollout engines**: vLLM, SGLang, HuggingFace Transformers\n- **Algorithms**: PPO, GRPO, DAPO, RLOO, ReMax, REINFORCE++, SPIN, SPPO\n- **Models**: Qwen-3, Llama-3.1, DeepSeek, Gemma-2 (0.5B to 671B)\n- **Advanced**: LoRA RL, sequence parallelism, expert parallelism, multi-turn tools\n\n## Installation\n\n```bash\n# Option 1: pip install\npip install verl[vllm]  # or verl[sglang] for SGLang backend\n\n# Option 2: Docker (recommended for production)\ndocker pull verlai\u002Fverl:vllm011.latest\n\n# Option 3: From source\ngit clone https:\u002F\u002Fgithub.com\u002Fvolcengine\u002Fverl.git\ncd verl && pip install -e .[vllm,math]\n```\n\n## Quick Start: GRPO Training\n\n```bash\npython3 -m verl.trainer.main_ppo \\\n    algorithm.adv_estimator=grpo \\\n    data.train_files=~\u002Fdata\u002Fgsm8k\u002Ftrain.parquet \\\n    actor_rollout_ref.model.path=Qwen\u002FQwen2.5-7B \\\n    actor_rollout_ref.rollout.n=8 \\\n    actor_rollout_ref.actor.use_kl_loss=True \\\n    trainer.n_gpus_per_node=8\n```\n\n## Core Architecture\n\nverl uses a **HybridFlow** programming model separating control flow from computation:\n\n```\n┌─────────────────────────────────────────────────────────┐\n│ Single-Process Controller (Ray)                         │\n│ - Orchestrates: rollout → reward → train → sync        │\n└─────────────────────┬───────────────────────────────────┘\n                      │\n┌─────────────────────▼───────────────────────────────────┐\n│ Multi-Process Workers                                   │\n│ ├── ActorRolloutRefWorker (policy + generation)        │\n│ ├── CriticWorker (value estimation, PPO only)          │\n│ └── RewardManager (model-based or rule-based rewards)  │\n└─────────────────────────────────────────────────────────┘\n```\n\n---\n\n## Workflow 1: Math Reasoning with GRPO\n\nUse this workflow for training reasoning models on math tasks like GSM8K or MATH.\n\n### Prerequisites Checklist\n- [ ] GPU cluster with 8+ GPUs (H100 recommended)\n- [ ] Dataset in parquet format with `prompt` and `reward_model` columns\n- [ ] Base model from HuggingFace Hub\n\n### Step 1: Prepare Dataset\n\n```python\nimport pandas as pd\n\ndata = [\n    {\n        \"prompt\": [{\"role\": \"user\", \"content\": \"What is 15 + 27?\"}],\n        \"reward_model\": {\"ground_truth\": \"42\"}\n    },\n    # ... more examples\n]\ndf = pd.DataFrame(data)\ndf.to_parquet(\"train.parquet\")\n```\n\n### Step 2: Define Reward Function\n\n```python\n# reward_function.py\nimport re\n\ndef compute_reward(responses, ground_truths):\n    rewards = []\n    for response, gt in zip(responses, ground_truths):\n        # Extract answer from response\n        match = re.search(r'\\\\boxed{([^}]+)}', response)\n        if match and match.group(1).strip() == gt.strip():\n            rewards.append(1.0)\n        else:\n            rewards.append(0.0)\n    return rewards\n```\n\n### Step 3: Create Training Config\n\n```yaml\n# config\u002Fgrpo_math.yaml\nalgorithm:\n  adv_estimator: grpo\n  gamma: 1.0\n  lam: 1.0\n\ndata:\n  train_files: \u002Fpath\u002Fto\u002Ftrain.parquet\n  val_files: \u002Fpath\u002Fto\u002Fval.parquet\n  train_batch_size: 256\n  max_prompt_length: 512\n  max_response_length: 2048\n\nactor_rollout_ref:\n  model:\n    path: Qwen\u002FQwen2.5-7B-Instruct\n  actor:\n    use_kl_loss: true\n    kl_loss_coef: 0.001\n    ppo_mini_batch_size: 64\n  rollout:\n    name: vllm\n    n: 8  # samples per prompt\n    temperature: 0.7\n    top_p: 0.95\n\ntrainer:\n  total_epochs: 3\n  n_gpus_per_node: 8\n  save_freq: 100\n```\n\n### Step 4: Launch Training\n\n```bash\npython3 -m verl.trainer.main_ppo \\\n    --config-path config \\\n    --config-name grpo_math \\\n    trainer.experiment_name=grpo_math_qwen7b\n```\n\n### Step 5: Monitor and Validate\n- [ ] Check WandB\u002FTensorBoard for loss curves\n- [ ] Verify reward is increasing over steps\n- [ ] Run evaluation on held-out test set\n\n---\n\n## Workflow 2: PPO with Critic Model\n\nUse this workflow when you need value-based advantage estimation (GAE).\n\n### Key Differences from GRPO\n- Requires separate critic model\n- Uses Generalized Advantage Estimation (GAE)\n- Better for tasks with dense rewards\n\n### Configuration\n\n```yaml\nalgorithm:\n  adv_estimator: gae  # Use GAE instead of GRPO\n  gamma: 0.99\n  lam: 0.95\n\ncritic:\n  model:\n    path: Qwen\u002FQwen2.5-7B-Instruct  # Can be same or different from actor\n  ppo_mini_batch_size: 64\n\nactor_rollout_ref:\n  actor:\n    use_kl_loss: true\n    kl_loss_coef: 0.02\n    clip_ratio: 0.2  # PPO clipping\n```\n\n### Launch with Critic\n\n```bash\npython3 -m verl.trainer.main_ppo \\\n    algorithm.adv_estimator=gae \\\n    critic.model.path=Qwen\u002FQwen2.5-7B-Instruct \\\n    trainer.n_gpus_per_node=8\n```\n\n---\n\n## Workflow 3: Large-Scale Training with Megatron\n\nUse this workflow for models >70B parameters or when you need expert parallelism.\n\n### Prerequisites\n- [ ] Install Megatron-LM bridge: `pip install mbridge`\n- [ ] Convert model to Megatron format\n- [ ] Multi-node cluster with NVLink\u002FInfiniBand\n\n### Configuration for 70B+ Models\n\n```yaml\nactor_rollout_ref:\n  model:\n    path: \u002Fpath\u002Fto\u002Fmegatron\u002Fcheckpoint\n    backend: megatron\n  actor:\n    strategy: megatron\n    tensor_model_parallel_size: 8\n    pipeline_model_parallel_size: 2\n  rollout:\n    name: vllm\n    tensor_parallel_size: 8\n```\n\n### Launch Multi-Node\n\n```bash\n# On head node\nray start --head --port=6379\n\n# On worker nodes\nray start --address='head_ip:6379'\n\n# Launch training\npython3 -m verl.trainer.main_ppo \\\n    trainer.nnodes=4 \\\n    trainer.n_gpus_per_node=8\n```\n\n---\n\n## Configuration Reference\n\n### Algorithm Selection\n\n| Algorithm | `adv_estimator` | Use Case |\n|-----------|-----------------|----------|\n| GRPO | `grpo` | Critic-free, math\u002Freasoning |\n| PPO\u002FGAE | `gae` | Dense rewards, value estimation |\n| REINFORCE++ | `reinforce_plus_plus` | Variance reduction |\n| RLOO | `rloo` | Leave-one-out baseline |\n| ReMax | `remax` | Maximum reward baseline |\n| OPO | `opo` | Optimal policy optimization |\n\n### Key Parameters\n\n```yaml\n# Rollout parameters\nactor_rollout_ref.rollout.n: 8              # Samples per prompt\nactor_rollout_ref.rollout.temperature: 0.7  # Sampling temperature\nactor_rollout_ref.rollout.top_p: 0.95       # Nucleus sampling\n\n# Training parameters\nactor_rollout_ref.actor.lr: 1e-6            # Learning rate\nactor_rollout_ref.actor.ppo_mini_batch_size: 64\nactor_rollout_ref.actor.clip_ratio: 0.2     # PPO clip range\n\n# KL control\nactor_rollout_ref.actor.use_kl_loss: true\nactor_rollout_ref.actor.kl_loss_coef: 0.001\nalgorithm.kl_ctrl.target_kl: 0.1            # For adaptive KL control\n```\n\n---\n\n## Common Issues and Solutions\n\n### Issue: OOM During Rollout\n\n**Symptoms**: CUDA out of memory during generation phase\n\n**Solutions**:\n```yaml\n# Reduce batch size\nactor_rollout_ref.rollout.log_prob_micro_batch_size: 4\n\n# Enable gradient checkpointing\nactor_rollout_ref.model.enable_gradient_checkpointing: true\n\n# Use FSDP2 with CPU offloading\nactor_rollout_ref.actor.strategy: fsdp2\nactor_rollout_ref.actor.fsdp_config.offload_policy: true\n```\n\n### Issue: Training Instability\n\n**Symptoms**: Loss spikes, reward collapse\n\n**Solutions**:\n```yaml\n# Reduce learning rate\nactor_rollout_ref.actor.lr: 5e-7\n\n# Increase KL penalty\nactor_rollout_ref.actor.kl_loss_coef: 0.01\n\n# Enable gradient clipping\nactor_rollout_ref.actor.max_grad_norm: 1.0\n```\n\n### Issue: Slow Weight Sync\n\n**Symptoms**: Long pauses between rollout and training\n\n**Solutions**:\n```bash\n# Use FSDP2 for faster resharding\nactor_rollout_ref.actor.strategy=fsdp2\n\n# Enable async weight transfer\ntrainer.async_weight_update=true\n```\n\n### Issue: vLLM Version Mismatch\n\n**Symptoms**: Import errors or generation failures\n\n**Solution**: Use compatible versions:\n```bash\npip install vllm>=0.8.5,\u003C=0.12.0\n# Avoid vLLM 0.7.x (known bugs)\n```\n\n---\n\n## Advanced Topics\n\n### Multi-Turn Tool Calling\n\nSee references\u002Fmulti-turn.md for agentic workflows with tool use.\n\n### Vision-Language Models\n\n```yaml\nactor_rollout_ref:\n  model:\n    path: Qwen\u002FQwen2.5-VL-7B-Instruct\n  rollout:\n    name: vllm\n    enable_vision: true\n```\n\n### LoRA Training\n\n```yaml\nactor_rollout_ref:\n  actor:\n    lora:\n      enabled: true\n      r: 16\n      alpha: 32\n      target_modules: [\"q_proj\", \"v_proj\"]\n```\n\n---\n\n## Resources\n\n- **Documentation**: https:\u002F\u002Fverl.readthedocs.io\u002F\n- **Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2409.19256\n- **GitHub**: https:\u002F\u002Fgithub.com\u002Fvolcengine\u002Fverl\n- **Recipes**: https:\u002F\u002Fgithub.com\u002Fverl-project\u002Fverl-recipe (DAPO, GSPO, etc.)\n- **Community**: Slack at verl-project\n\n## Agent operating procedure\n\n1. **Check the environment.** Check GPU type, memory and driver\u002FCUDA versions (`nvidia-smi`), framework versions, and dataset location and size.\n2. **Pin down the inputs.** Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.\n3. **Run a small version first.** Do a smoke run: tiny model or subset, few steps, and confirm loss decreases and checkpoints save.\n4. **Execute the full task** using the instructions and references above.\n5. **Validate the result.** Track metrics on held-out data, compare against a baseline, and record seeds, configs and hardware.\n6. **Report.** State what was run (versions, commands, parameters), what was checked, and what is still uncertain.\n\n| If this happens | Do this |\n|---|---|\n| CUDA out-of-memory | Reduce batch size, enable gradient accumulation\u002Fcheckpointing or mixed precision, or shard the model. |\n| Loss is NaN or diverges | Lower the learning rate, check data for invalid values, and enable gradient clipping. |\n| A function, flag or endpoint in these instructions is missing in the installed version | Check the installed version's own documentation (`help()`, `--help`, official docs), adapt, and tell the user. Never invent an API. |\n| A required input, identifier or parameter is ambiguous | Ask the user, or state the assumption explicitly before running. |\n\n**Integrity rules**\n\n- Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.\n- Never claim training results without logs; estimate compute cost before launching large jobs.\n- Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.\n- Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.\n\n## Related skills\n\n- `fine-tuning-with-trl`: Fine-tune LLMs using reinforcement learning with TRL - SFT for instruction tuning, DPO for preference alignment, PPO\u002FGRPO for reward optimi…\n- `slime-rl-training`: Provides guidance for LLM post-training with RL using slime, a Megatron+SGLang framework.\n- `openrlhf-training`: High-performance RLHF framework with Ray+vLLM acceleration.\n",{"name":5,"license":15,"metadata":44,"description":7},{"tags":45,"version":46,"category":47,"maintainer":48,"dependencies":49},"Reinforcement Learning, RLHF, GRPO, PPO, Post-Training, Distributed Training","1.0.0","ml-training","Kalaris Labs","verl>=0.3.0, torch>=2.0.0, ray>=2.41.0, vllm>=0.8.2, transformers>=4.40.0","main","2b63f856ce19c2a797929cacabd9e0fc936812fa","plugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fverl-rl-training",491,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins","https:\u002F\u002Fhol.org\u002Fplugins\u002Fbest-codex-plugins","https:\u002F\u002Fgithub.com\u002Fhashgraph-online","https:\u002F\u002Fhol.org","Open-source standards and developer tools for interoperable AI agents: discovery, identity, communication, payments, privacy and security.","2026-11-14 14:06:13",{"source":8,"skillName":5,"commands":61,"prompt":71,"zipAvailable":72},[62,65,68],{"tool":63,"command":64},"npx","npx skills add hashgraph-online\u002Fawesome-codex-plugins --skill verl-rl-training",{"tool":66,"command":67},"bunx","bunx skills add hashgraph-online\u002Fawesome-codex-plugins --skill verl-rl-training",{"tool":69,"command":70},"pnpm","pnpm dlx skills add hashgraph-online\u002Fawesome-codex-plugins --skill verl-rl-training","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fverl-rl-training\u002FSKILL.md",true,[74,80],{"path":75,"resourceType":76,"resourceTypeDesc":77,"sizeBytes":78,"sourceHtmlUrl":79},"references\u002Fapi-reference.md","REFERENCE","参考文档",6941,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fverl-rl-training\u002Freferences\u002Fapi-reference.md",{"path":81,"resourceType":76,"resourceTypeDesc":77,"sizeBytes":82,"sourceHtmlUrl":83},"references\u002Ftroubleshooting.md",6792,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fverl-rl-training\u002Freferences\u002Ftroubleshooting.md",[85,94,102,110,118,126],{"skillNo":86,"name":87,"nameZh":6,"description":88,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":89,"sourceHtmlUrl":90,"categories":91,"keywords":92,"publishedAt":93},"01M7DV332YQMGD8WTRJ9DZQXTB","worktable-review-with-annotations","Create, inspect, reply to, update, or resolve situated Worktable annotations. Use when feedback belongs to a Doc section, block, text range, or HTML Doc, or when the user asks to address existing review feedback and coordinate a requested artifact edit.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-review-with-annotations\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-review-with-annotations\u002FSKILL.md",[],[],"2026-11-15 14:19:17",{"skillNo":95,"name":96,"nameZh":6,"description":97,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":98,"sourceHtmlUrl":99,"categories":100,"keywords":101,"publishedAt":93},"01M7DV2YGHCZVBEB1AHYB6CGHE","worktable-find-and-synthesize-context","Find, assess, and synthesize context from Worktable Docs and Records. Use when the user asks what a workspace contains, wants workspace-backed research, needs current facts reconciled across artifacts, or requests a cited synthesis without changing workspace state.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-find-and-synthesize-context\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-find-and-synthesize-context\u002FSKILL.md",[],[],{"skillNo":103,"name":104,"nameZh":6,"description":105,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":106,"sourceHtmlUrl":107,"categories":108,"keywords":109,"publishedAt":93},"01M7DV2TQN1H11ME2SAF9MXN7V","worktable-create-or-update-docs","Create, edit, rename, or maintain narrative Worktable Docs. Use for plans, decisions, briefs, notes, research syntheses, and other durable prose when the user requests a new artifact or a safe update to an existing one.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-update-docs\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-update-docs\u002FSKILL.md",[],[],{"skillNo":111,"name":112,"nameZh":6,"description":113,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":114,"sourceHtmlUrl":115,"categories":116,"keywords":117,"publishedAt":93},"01M7DV2NE452NEA8TXATKSNAJH","worktable-create-or-manage-records","Organize repeated structured information as durable Worktable Records. Use for research datasets, inventories, directories, pipelines, trackers, and other collections whose items need identity, fields, filtering, links, validation, or individual updates.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-records\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-records\u002FSKILL.md",[],[],{"skillNo":119,"name":120,"nameZh":6,"description":121,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":122,"sourceHtmlUrl":123,"categories":124,"keywords":125,"publishedAt":93},"01M7DV2HENRBR2KSCVM064WH0Y","worktable-create-or-manage-interactive-html","Create or improve durable visual and interactive artifacts in Worktable. Use for dashboards, trackers, calculators, mini apps, slide presentations, and other requests that benefit from a polished HTML Doc.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-interactive-html\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-interactive-html\u002FSKILL.md",[],[],{"skillNo":127,"name":128,"nameZh":6,"description":129,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":130,"sourceHtmlUrl":131,"categories":132,"keywords":133,"publishedAt":93},"01M7DV2CADR5R7RQFFYPNAGR9Y","worktable-collaborate-threads","Read, answer, and hand off messages through durable Worktable threads with users or other authenticated participants. Use when the user asks to check addressed messages, reply to a participant, continue a thread, request another agent's help, wait briefly for a response, or watch for comments and messages sent to the agent.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-collaborate-threads\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-collaborate-threads\u002FSKILL.md",[],[]]