[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M73AAD7CJ6WAZE92J28Q021M":3,"skill-same:01M73AAD7CJ6WAZE92J28Q021M":86},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":46,"commitSha":47,"skillDirectory":48,"forksCount":49,"repositoryHtmlUrl":50,"repositoryHomepageUrl":6,"ownerHtmlUrl":51,"ownerBlogUrl":52,"ownerDescription":6,"updatedAt":53,"install":54,"resources":67},"01M73AAD7CJ6WAZE92J28Q021M","openrlhf-training",null,"High-performance RLHF framework with Ray+vLLM acceleration. Use for PPO, GRPO, RLOO, DPO training of large models (7B-70B+). Built on Ray, vLLM, ZeRO-3. 2× faster than DeepSpeedChat with distributed architecture and GPU resource sharing.","qhjqhj00\u002Fresearch-skills-pool","qhjqhj00","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F29684092?v=4","01M72AE0ZGXVM5C57N512DGQG8","Tommy Chien","01M72AE107ESJMHT7QC5SXX144",3,"MIT","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--06-post-training__openrlhf\u002FSKILL.md",[],[],"2026-11-11 15:11:14",false,"---\nname: openrlhf-training\ndescription: High-performance RLHF framework with Ray+vLLM acceleration. Use for PPO, GRPO, RLOO, DPO training of large models (7B-70B+). Built on Ray, vLLM, ZeRO-3. 2× faster than DeepSpeedChat with distributed architecture and GPU resource sharing.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Post-Training, OpenRLHF, RLHF, PPO, GRPO, RLOO, DPO, Ray, vLLM, Distributed Training, Large Models, ZeRO-3]\ndependencies: [openrlhf, ray, vllm, torch, transformers, deepspeed]\n---\n\n# OpenRLHF - High-Performance RLHF Training\n\n## Quick start\n\nOpenRLHF is a Ray-based RLHF framework optimized for distributed training with vLLM inference acceleration.\n\n**Installation**:\n```bash\n# Launch Docker container\ndocker run --runtime=nvidia -it --rm --shm-size=\"10g\" --cap-add=SYS_ADMIN \\\n  -v $PWD:\u002Fopenrlhf nvcr.io\u002Fnvidia\u002Fpytorch:25.02-py3 bash\n\n# Uninstall conflicts\nsudo pip uninstall xgboost transformer_engine flash_attn pynvml -y\n\n# Install OpenRLHF with vLLM\npip install openrlhf[vllm]\n```\n\n**PPO Training** (Hybrid Engine):\n```bash\nray start --head --node-ip-address 0.0.0.0 --num-gpus 8\n\nray job submit --address=\"http:\u002F\u002F127.0.0.1:8265\" \\\n  --runtime-env-json='{\"working_dir\": \"\u002Fopenrlhf\"}' \\\n  -- python3 -m openrlhf.cli.train_ppo_ray \\\n  --ref_num_nodes 1 --ref_num_gpus_per_node 8 \\\n  --reward_num_nodes 1 --reward_num_gpus_per_node 8 \\\n  --critic_num_nodes 1 --critic_num_gpus_per_node 8 \\\n  --actor_num_nodes 1 --actor_num_gpus_per_node 8 \\\n  --vllm_num_engines 4 --vllm_tensor_parallel_size 2 \\\n  --colocate_all_models \\\n  --vllm_gpu_memory_utilization 0.5 \\\n  --pretrain OpenRLHF\u002FLlama-3-8b-sft-mixture \\\n  --reward_pretrain OpenRLHF\u002FLlama-3-8b-rm-700k \\\n  --save_path .\u002Foutput\u002Fllama3-8b-rlhf \\\n  --micro_train_batch_size 8 --train_batch_size 128 \\\n  --micro_rollout_batch_size 16 --rollout_batch_size 1024 \\\n  --max_epochs 1 --prompt_max_len 1024 --generate_max_len 1024 \\\n  --zero_stage 3 --bf16 \\\n  --actor_learning_rate 5e-7 --critic_learning_rate 9e-6 \\\n  --init_kl_coef 0.01 --normalize_reward \\\n  --gradient_checkpointing --packing_samples \\\n  --vllm_enable_sleep --deepspeed_enable_sleep\n```\n\n**GRPO Training** (Group Normalized Policy Optimization):\n```bash\n# Same command as PPO, but add:\n--advantage_estimator group_norm\n```\n\n## Common workflows\n\n### Workflow 1: Full RLHF pipeline (SFT → Reward Model → PPO)\n\n**Step 1: Train reward model** (DPO):\n```bash\ndeepspeed --module openrlhf.cli.train_rm \\\n  --save_path .\u002Foutput\u002Fllama3-8b-rm \\\n  --save_steps -1 --logging_steps 1 \\\n  --eval_steps -1 --train_batch_size 256 \\\n  --micro_train_batch_size 1 --pretrain meta-llama\u002FMeta-Llama-3-8B \\\n  --bf16 --max_epochs 1 --max_len 8192 \\\n  --zero_stage 3 --learning_rate 9e-6 \\\n  --dataset OpenRLHF\u002Fpreference_dataset_mixture2_and_safe_pku \\\n  --apply_chat_template --chosen_key chosen \\\n  --rejected_key rejected --flash_attn --gradient_checkpointing\n```\n\n**Step 2: PPO training**:\n```bash\nray start --head --node-ip-address 0.0.0.0 --num-gpus 8\n\nray job submit --address=\"http:\u002F\u002F127.0.0.1:8265\" \\\n  -- python3 -m openrlhf.cli.train_ppo_ray \\\n  --ref_num_nodes 1 --ref_num_gpus_per_node 8 \\\n  --reward_num_nodes 1 --reward_num_gpus_per_node 8 \\\n  --critic_num_nodes 1 --critic_num_gpus_per_node 8 \\\n  --actor_num_nodes 1 --actor_num_gpus_per_node 8 \\\n  --vllm_num_engines 4 --vllm_tensor_parallel_size 2 \\\n  --colocate_all_models \\\n  --pretrain OpenRLHF\u002FLlama-3-8b-sft-mixture \\\n  --reward_pretrain .\u002Foutput\u002Fllama3-8b-rm \\\n  --save_path .\u002Foutput\u002Fllama3-8b-ppo \\\n  --micro_train_batch_size 8 --train_batch_size 128 \\\n  --micro_rollout_batch_size 16 --rollout_batch_size 1024 \\\n  --max_epochs 1 --prompt_max_len 1024 --generate_max_len 1024 \\\n  --zero_stage 3 --bf16 \\\n  --actor_learning_rate 5e-7 --critic_learning_rate 9e-6 \\\n  --init_kl_coef 0.01 --normalize_reward \\\n  --vllm_enable_sleep --deepspeed_enable_sleep\n```\n\n### Workflow 2: GRPO training (no critic model needed)\n\nMemory-efficient alternative to PPO:\n\n```bash\nray job submit --address=\"http:\u002F\u002F127.0.0.1:8265\" \\\n  -- python3 -m openrlhf.cli.train_ppo_ray \\\n  --advantage_estimator group_norm \\\n  --ref_num_nodes 1 --ref_num_gpus_per_node 8 \\\n  --reward_num_nodes 1 --reward_num_gpus_per_node 8 \\\n  --actor_num_nodes 1 --actor_num_gpus_per_node 8 \\\n  --vllm_num_engines 4 --vllm_tensor_parallel_size 2 \\\n  --colocate_all_models \\\n  --pretrain OpenRLHF\u002FLlama-3-8b-sft-mixture \\\n  --reward_pretrain OpenRLHF\u002FLlama-3-8b-rm-700k \\\n  --save_path .\u002Foutput\u002Fllama3-8b-grpo \\\n  --micro_train_batch_size 8 --train_batch_size 128 \\\n  --micro_rollout_batch_size 16 --rollout_batch_size 1024 \\\n  --max_epochs 1 --bf16 \\\n  --actor_learning_rate 5e-7 \\\n  --init_kl_coef 0.01 --use_kl_loss --kl_estimator k3 \\\n  --normalize_reward --no_advantage_std_norm\n```\n\n**Key GRPO parameters**:\n- `--advantage_estimator group_norm` - Enables GRPO\n- `--use_kl_loss` - KL loss from GRPO paper\n- `--kl_estimator k3` - Loss function (k2 ≈ k1)\n- `--no_advantage_std_norm` - Disables std normalization\n\n### Workflow 3: DPO training (preference optimization)\n\nSimpler alternative without reward model:\n\n```bash\ndeepspeed --module openrlhf.cli.train_dpo \\\n  --save_path .\u002Foutput\u002Fllama3-8b-dpo \\\n  --save_steps -1 --logging_steps 1 \\\n  --eval_steps -1 --train_batch_size 256 \\\n  --micro_train_batch_size 2 --pretrain meta-llama\u002FMeta-Llama-3-8B \\\n  --bf16 --max_epochs 1 --max_len 8192 \\\n  --zero_stage 3 --learning_rate 5e-7 --beta 0.1 \\\n  --dataset OpenRLHF\u002Fpreference_dataset_mixture2_and_safe_pku \\\n  --apply_chat_template --chosen_key chosen \\\n  --rejected_key rejected --flash_attn --gradient_checkpointing\n```\n\n## When to use vs alternatives\n\n**Use OpenRLHF when**:\n- Training large models (7B-70B+) with RL\n- Need vLLM inference acceleration\n- Want distributed architecture with Ray\n- Have multi-node GPU cluster\n- Need PPO\u002FGRPO\u002FRLOO\u002FDPO in one framework\n\n**Algorithm selection**:\n- **PPO**: Maximum control, best for complex rewards\n- **GRPO**: Memory-efficient, no critic needed\n- **RLOO**: Modified PPO with per-token KL\n- **REINFORCE++**: More stable than GRPO, faster than PPO\n- **DPO**: Simplest, no reward model needed\n\n**Use alternatives instead**:\n- **TRL**: Single-node training, simpler API\n- **veRL**: ByteDance's framework for 671B models\n- **DeepSpeedChat**: Integrated with DeepSpeed ecosystem\n\n## Common issues\n\n**Issue: GPU OOM with large models**\n\nDisable model colocation:\n```bash\n# Remove --colocate_all_models flag\n# Allocate separate GPUs for each model\n--actor_num_gpus_per_node 8 \\\n--critic_num_gpus_per_node 8 \\\n--reward_num_gpus_per_node 8 \\\n--ref_num_gpus_per_node 8\n```\n\n**Issue: DeepSpeed GPU index out of range**\n\nSet environment variable:\n```bash\nexport RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES=1\n```\n\n**Issue: Training instability**\n\nUse Hybrid Engine instead of async:\n```bash\n--colocate_all_models \\\n--vllm_enable_sleep \\\n--deepspeed_enable_sleep\n```\n\nAdjust KL coefficient:\n```bash\n--init_kl_coef 0.05  # Increase from 0.01\n```\n\n**Issue: Slow generation during PPO**\n\nEnable vLLM acceleration:\n```bash\n--vllm_num_engines 4 \\\n--vllm_tensor_parallel_size 2 \\\n--vllm_gpu_memory_utilization 0.5\n```\n\n## Advanced topics\n\n**Hybrid Engine GPU sharing**: See [references\u002Fhybrid-engine.md](references\u002Fhybrid-engine.md) for vLLM sleep mode, DeepSpeed sleep mode, and optimal node allocation.\n\n**Algorithm comparison**: See [references\u002Falgorithm-comparison.md](references\u002Falgorithm-comparison.md) for PPO vs GRPO vs RLOO vs REINFORCE++ benchmarks and hyperparameters.\n\n**Multi-node setup**: See [references\u002Fmulti-node-training.md](references\u002Fmulti-node-training.md) for Ray cluster configuration and fault tolerance.\n\n**Custom reward functions**: See [references\u002Fcustom-rewards.md](references\u002Fcustom-rewards.md) for reinforced fine-tuning and agent RLHF.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA A100\u002FH100 recommended\n- **VRAM**:\n  - 7B model: 8× A100 40GB (Hybrid Engine)\n  - 70B model: 48× A100 80GB (vLLM:Actor:Critic = 1:1:1)\n- **Multi-node**: Ray cluster with InfiniBand recommended\n- **Docker**: NVIDIA PyTorch container 25.02+\n\n**Performance**:\n- 2× faster than DeepSpeedChat\n- vLLM inference acceleration\n- Hybrid Engine minimizes GPU idle time\n\n## Resources\n\n- Docs: https:\u002F\u002Fgithub.com\u002FOpenRLHF\u002FOpenRLHF\n- Paper: https:\u002F\u002Farxiv.org\u002Fabs\u002F2405.11143\n- Examples: https:\u002F\u002Fgithub.com\u002FOpenRLHF\u002FOpenRLHF\u002Ftree\u002Fmain\u002Fexamples\n- Discord: Community support\n\n\n\n","# OpenRLHF - High-Performance RLHF Training\n\n## Quick start\n\nOpenRLHF is a Ray-based RLHF framework optimized for distributed training with vLLM inference acceleration.\n\n**Installation**:\n```bash\n# Launch Docker container\ndocker run --runtime=nvidia -it --rm --shm-size=\"10g\" --cap-add=SYS_ADMIN \\\n  -v $PWD:\u002Fopenrlhf nvcr.io\u002Fnvidia\u002Fpytorch:25.02-py3 bash\n\n# Uninstall conflicts\nsudo pip uninstall xgboost transformer_engine flash_attn pynvml -y\n\n# Install OpenRLHF with vLLM\npip install openrlhf[vllm]\n```\n\n**PPO Training** (Hybrid Engine):\n```bash\nray start --head --node-ip-address 0.0.0.0 --num-gpus 8\n\nray job submit --address=\"http:\u002F\u002F127.0.0.1:8265\" \\\n  --runtime-env-json='{\"working_dir\": \"\u002Fopenrlhf\"}' \\\n  -- python3 -m openrlhf.cli.train_ppo_ray \\\n  --ref_num_nodes 1 --ref_num_gpus_per_node 8 \\\n  --reward_num_nodes 1 --reward_num_gpus_per_node 8 \\\n  --critic_num_nodes 1 --critic_num_gpus_per_node 8 \\\n  --actor_num_nodes 1 --actor_num_gpus_per_node 8 \\\n  --vllm_num_engines 4 --vllm_tensor_parallel_size 2 \\\n  --colocate_all_models \\\n  --vllm_gpu_memory_utilization 0.5 \\\n  --pretrain OpenRLHF\u002FLlama-3-8b-sft-mixture \\\n  --reward_pretrain OpenRLHF\u002FLlama-3-8b-rm-700k \\\n  --save_path .\u002Foutput\u002Fllama3-8b-rlhf \\\n  --micro_train_batch_size 8 --train_batch_size 128 \\\n  --micro_rollout_batch_size 16 --rollout_batch_size 1024 \\\n  --max_epochs 1 --prompt_max_len 1024 --generate_max_len 1024 \\\n  --zero_stage 3 --bf16 \\\n  --actor_learning_rate 5e-7 --critic_learning_rate 9e-6 \\\n  --init_kl_coef 0.01 --normalize_reward \\\n  --gradient_checkpointing --packing_samples \\\n  --vllm_enable_sleep --deepspeed_enable_sleep\n```\n\n**GRPO Training** (Group Normalized Policy Optimization):\n```bash\n# Same command as PPO, but add:\n--advantage_estimator group_norm\n```\n\n## Common workflows\n\n### Workflow 1: Full RLHF pipeline (SFT → Reward Model → PPO)\n\n**Step 1: Train reward model** (DPO):\n```bash\ndeepspeed --module openrlhf.cli.train_rm \\\n  --save_path .\u002Foutput\u002Fllama3-8b-rm \\\n  --save_steps -1 --logging_steps 1 \\\n  --eval_steps -1 --train_batch_size 256 \\\n  --micro_train_batch_size 1 --pretrain meta-llama\u002FMeta-Llama-3-8B \\\n  --bf16 --max_epochs 1 --max_len 8192 \\\n  --zero_stage 3 --learning_rate 9e-6 \\\n  --dataset OpenRLHF\u002Fpreference_dataset_mixture2_and_safe_pku \\\n  --apply_chat_template --chosen_key chosen \\\n  --rejected_key rejected --flash_attn --gradient_checkpointing\n```\n\n**Step 2: PPO training**:\n```bash\nray start --head --node-ip-address 0.0.0.0 --num-gpus 8\n\nray job submit --address=\"http:\u002F\u002F127.0.0.1:8265\" \\\n  -- python3 -m openrlhf.cli.train_ppo_ray \\\n  --ref_num_nodes 1 --ref_num_gpus_per_node 8 \\\n  --reward_num_nodes 1 --reward_num_gpus_per_node 8 \\\n  --critic_num_nodes 1 --critic_num_gpus_per_node 8 \\\n  --actor_num_nodes 1 --actor_num_gpus_per_node 8 \\\n  --vllm_num_engines 4 --vllm_tensor_parallel_size 2 \\\n  --colocate_all_models \\\n  --pretrain OpenRLHF\u002FLlama-3-8b-sft-mixture \\\n  --reward_pretrain .\u002Foutput\u002Fllama3-8b-rm \\\n  --save_path .\u002Foutput\u002Fllama3-8b-ppo \\\n  --micro_train_batch_size 8 --train_batch_size 128 \\\n  --micro_rollout_batch_size 16 --rollout_batch_size 1024 \\\n  --max_epochs 1 --prompt_max_len 1024 --generate_max_len 1024 \\\n  --zero_stage 3 --bf16 \\\n  --actor_learning_rate 5e-7 --critic_learning_rate 9e-6 \\\n  --init_kl_coef 0.01 --normalize_reward \\\n  --vllm_enable_sleep --deepspeed_enable_sleep\n```\n\n### Workflow 2: GRPO training (no critic model needed)\n\nMemory-efficient alternative to PPO:\n\n```bash\nray job submit --address=\"http:\u002F\u002F127.0.0.1:8265\" \\\n  -- python3 -m openrlhf.cli.train_ppo_ray \\\n  --advantage_estimator group_norm \\\n  --ref_num_nodes 1 --ref_num_gpus_per_node 8 \\\n  --reward_num_nodes 1 --reward_num_gpus_per_node 8 \\\n  --actor_num_nodes 1 --actor_num_gpus_per_node 8 \\\n  --vllm_num_engines 4 --vllm_tensor_parallel_size 2 \\\n  --colocate_all_models \\\n  --pretrain OpenRLHF\u002FLlama-3-8b-sft-mixture \\\n  --reward_pretrain OpenRLHF\u002FLlama-3-8b-rm-700k \\\n  --save_path .\u002Foutput\u002Fllama3-8b-grpo \\\n  --micro_train_batch_size 8 --train_batch_size 128 \\\n  --micro_rollout_batch_size 16 --rollout_batch_size 1024 \\\n  --max_epochs 1 --bf16 \\\n  --actor_learning_rate 5e-7 \\\n  --init_kl_coef 0.01 --use_kl_loss --kl_estimator k3 \\\n  --normalize_reward --no_advantage_std_norm\n```\n\n**Key GRPO parameters**:\n- `--advantage_estimator group_norm` - Enables GRPO\n- `--use_kl_loss` - KL loss from GRPO paper\n- `--kl_estimator k3` - Loss function (k2 ≈ k1)\n- `--no_advantage_std_norm` - Disables std normalization\n\n### Workflow 3: DPO training (preference optimization)\n\nSimpler alternative without reward model:\n\n```bash\ndeepspeed --module openrlhf.cli.train_dpo \\\n  --save_path .\u002Foutput\u002Fllama3-8b-dpo \\\n  --save_steps -1 --logging_steps 1 \\\n  --eval_steps -1 --train_batch_size 256 \\\n  --micro_train_batch_size 2 --pretrain meta-llama\u002FMeta-Llama-3-8B \\\n  --bf16 --max_epochs 1 --max_len 8192 \\\n  --zero_stage 3 --learning_rate 5e-7 --beta 0.1 \\\n  --dataset OpenRLHF\u002Fpreference_dataset_mixture2_and_safe_pku \\\n  --apply_chat_template --chosen_key chosen \\\n  --rejected_key rejected --flash_attn --gradient_checkpointing\n```\n\n## When to use vs alternatives\n\n**Use OpenRLHF when**:\n- Training large models (7B-70B+) with RL\n- Need vLLM inference acceleration\n- Want distributed architecture with Ray\n- Have multi-node GPU cluster\n- Need PPO\u002FGRPO\u002FRLOO\u002FDPO in one framework\n\n**Algorithm selection**:\n- **PPO**: Maximum control, best for complex rewards\n- **GRPO**: Memory-efficient, no critic needed\n- **RLOO**: Modified PPO with per-token KL\n- **REINFORCE++**: More stable than GRPO, faster than PPO\n- **DPO**: Simplest, no reward model needed\n\n**Use alternatives instead**:\n- **TRL**: Single-node training, simpler API\n- **veRL**: ByteDance's framework for 671B models\n- **DeepSpeedChat**: Integrated with DeepSpeed ecosystem\n\n## Common issues\n\n**Issue: GPU OOM with large models**\n\nDisable model colocation:\n```bash\n# Remove --colocate_all_models flag\n# Allocate separate GPUs for each model\n--actor_num_gpus_per_node 8 \\\n--critic_num_gpus_per_node 8 \\\n--reward_num_gpus_per_node 8 \\\n--ref_num_gpus_per_node 8\n```\n\n**Issue: DeepSpeed GPU index out of range**\n\nSet environment variable:\n```bash\nexport RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES=1\n```\n\n**Issue: Training instability**\n\nUse Hybrid Engine instead of async:\n```bash\n--colocate_all_models \\\n--vllm_enable_sleep \\\n--deepspeed_enable_sleep\n```\n\nAdjust KL coefficient:\n```bash\n--init_kl_coef 0.05  # Increase from 0.01\n```\n\n**Issue: Slow generation during PPO**\n\nEnable vLLM acceleration:\n```bash\n--vllm_num_engines 4 \\\n--vllm_tensor_parallel_size 2 \\\n--vllm_gpu_memory_utilization 0.5\n```\n\n## Advanced topics\n\n**Hybrid Engine GPU sharing**: See [references\u002Fhybrid-engine.md](references\u002Fhybrid-engine.md) for vLLM sleep mode, DeepSpeed sleep mode, and optimal node allocation.\n\n**Algorithm comparison**: See [references\u002Falgorithm-comparison.md](references\u002Falgorithm-comparison.md) for PPO vs GRPO vs RLOO vs REINFORCE++ benchmarks and hyperparameters.\n\n**Multi-node setup**: See [references\u002Fmulti-node-training.md](references\u002Fmulti-node-training.md) for Ray cluster configuration and fault tolerance.\n\n**Custom reward functions**: See [references\u002Fcustom-rewards.md](references\u002Fcustom-rewards.md) for reinforced fine-tuning and agent RLHF.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA A100\u002FH100 recommended\n- **VRAM**:\n  - 7B model: 8× A100 40GB (Hybrid Engine)\n  - 70B model: 48× A100 80GB (vLLM:Actor:Critic = 1:1:1)\n- **Multi-node**: Ray cluster with InfiniBand recommended\n- **Docker**: NVIDIA PyTorch container 25.02+\n\n**Performance**:\n- 2× faster than DeepSpeedChat\n- vLLM inference acceleration\n- Hybrid Engine minimizes GPU idle time\n\n## Resources\n\n- Docs: https:\u002F\u002Fgithub.com\u002FOpenRLHF\u002FOpenRLHF\n- Paper: https:\u002F\u002Farxiv.org\u002Fabs\u002F2405.11143\n- Examples: https:\u002F\u002Fgithub.com\u002FOpenRLHF\u002FOpenRLHF\u002Ftree\u002Fmain\u002Fexamples\n- Discord: Community support\n\n\n\n",{"name":5,"tags":24,"author":37,"license":15,"version":38,"description":7,"dependencies":39},[25,26,27,28,29,30,31,32,33,34,35,36],"Post-Training","OpenRLHF","RLHF","PPO","GRPO","RLOO","DPO","Ray","vLLM","Distributed Training","Large Models","ZeRO-3","Orchestra Research","1.0.0",[40,41,42,43,44,45],"openrlhf","ray","vllm","torch","transformers","deepspeed","master","c247302c836de6438d746b45670990eb71691166","skills_collected\u002FOrchestra-Research--AI-research-SKILLs--06-post-training__openrlhf",1,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool","https:\u002F\u002Fgithub.com\u002Fqhjqhj00","chien.io","2026-05-11 15:07:59",{"source":8,"skillName":5,"commands":55,"prompt":65,"zipAvailable":66},[56,59,62],{"tool":57,"command":58},"npx","npx skills add qhjqhj00\u002Fresearch-skills-pool --skill openrlhf-training",{"tool":60,"command":61},"bunx","bunx skills add qhjqhj00\u002Fresearch-skills-pool --skill openrlhf-training",{"tool":63,"command":64},"pnpm","pnpm dlx skills add qhjqhj00\u002Fresearch-skills-pool --skill openrlhf-training","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--06-post-training__openrlhf\u002FSKILL.md",true,[68,74,78,82],{"path":69,"resourceType":70,"resourceTypeDesc":71,"sizeBytes":72,"sourceHtmlUrl":73},"references\u002Falgorithm-comparison.md","REFERENCE","参考文档",9783,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--06-post-training__openrlhf\u002Freferences\u002Falgorithm-comparison.md",{"path":75,"resourceType":70,"resourceTypeDesc":71,"sizeBytes":76,"sourceHtmlUrl":77},"references\u002Fcustom-rewards.md",15865,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--06-post-training__openrlhf\u002Freferences\u002Fcustom-rewards.md",{"path":79,"resourceType":70,"resourceTypeDesc":71,"sizeBytes":80,"sourceHtmlUrl":81},"references\u002Fhybrid-engine.md",7266,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--06-post-training__openrlhf\u002Freferences\u002Fhybrid-engine.md",{"path":83,"resourceType":70,"resourceTypeDesc":71,"sizeBytes":84,"sourceHtmlUrl":85},"references\u002Fmulti-node-training.md",11099,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--06-post-training__openrlhf\u002Freferences\u002Fmulti-node-training.md",[87,94,101,108,115,122],{"skillNo":88,"name":89,"nameZh":6,"description":90,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":91,"categories":92,"keywords":93,"publishedAt":19},"01M73JJB6EQPFPPVQK2Y9N67Q0","transformer-lens-interpretability","Provides guidance for mechanistic interpretability research using TransformerLens to inspect and manipulate transformer internals via HookPoints and activation caching. Use when reverse-engineering model algorithms, studying attention patterns, or performing activation patching experiments.","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Ftop100-runs\u002F130-transformer-lens\u002FSKILL.md",[],[],{"skillNo":95,"name":96,"nameZh":6,"description":97,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":98,"categories":99,"keywords":100,"publishedAt":19},"01M73JHTANRRWEE9HP98TPEHAY","wiki-vitepress","Packages generated wiki Markdown into a VitePress static site with dark theme, dark-mode Mermaid diagrams with click-to-zoom, and production build output. Use when the user wants to create a browsable website from generated wiki pages.","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Ftop100-runs\u002F126-wiki-vitepress\u002FSKILL.md",[],[],{"skillNo":102,"name":103,"nameZh":6,"description":104,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":105,"categories":106,"keywords":107,"publishedAt":19},"01M73JHJBMVFW46F1Y8K3SBF43","phoenix-observability","Open-source AI observability platform for LLM tracing, evaluation, and monitoring. Use when debugging LLM applications with detailed traces, running evaluations on datasets, or monitoring production AI systems with real-time insights.","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Ftop100-runs\u002F124-phoenix\u002FSKILL.md",[],[],{"skillNo":109,"name":110,"nameZh":6,"description":111,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":112,"categories":113,"keywords":114,"publishedAt":19},"01M73JHFPA3A2806PMP42ZN86Q","frontend-ui-dark-ts","Build dark-themed React applications using Tailwind CSS with custom theming, glassmorphism effects, and Framer Motion animations. Use when creating dashboards, admin panels, or data-rich interfaces with a refined dark aesthetic.","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Ftop100-runs\u002F123-frontend-ui-dark-ts\u002FSKILL.md",[],[],{"skillNo":116,"name":117,"nameZh":6,"description":118,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":119,"categories":120,"keywords":121,"publishedAt":19},"01M73JFVBVK57NRJS49TT2VDCT","tensorboard","Visualize training metrics, debug models with histograms, compare experiments, visualize model graphs, and profile performance with TensorBoard - Google's ML visualization toolkit","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Ftop100-runs\u002F113-tensorboard\u002FSKILL.md",[],[],{"skillNo":123,"name":124,"nameZh":6,"description":125,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":126,"licenseFileUrl":6,"sourceHtmlUrl":127,"categories":128,"keywords":129,"publishedAt":19},"01M73JFF6PK6ZCN41C89J7HS2B","markdown-mermaid-writing","Comprehensive markdown and Mermaid diagram writing skill. Use when creating any scientific document, report, analysis, or visualization. Establishes text-based diagrams as the default documentation standard with full style guides (markdown + mermaid), 24 diagram type references, and 9 document templates.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Ftop100-runs\u002F110-markdown-mermaid-writing\u002FSKILL.md",[],[]]