[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6KQN0YZPJ8SRTNQX4PCFP9Z":3,"skill-same:01M6KQN0YZPJ8SRTNQX4PCFP9Z":95},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":38,"linkOnly":39,"markdownContent":40,"markdownBody":41,"frontMatter":42,"refName":57,"commitSha":58,"skillDirectory":59,"forksCount":13,"repositoryHtmlUrl":60,"repositoryHomepageUrl":6,"ownerHtmlUrl":61,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":62,"install":63,"resources":76},"01M6KQN0YZPJ8SRTNQX4PCFP9Z","openrlhf-training",null,"High-performance RLHF framework with Ray+vLLM acceleration. Use for PPO, GRPO, RLOO, DPO training of large models (7B-70B+). Built on Ray, vLLM, ZeRO-3. 2× faster than DeepSpeedChat with distributed architecture and GPU resource sharing.","endman100\u002Fskill-router-nexus","endman100","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F31997019?v=4","01M6KPDX17BYDWW4RRB0TNGGMX","01M6KPDX64Y2QBEWVFG5W5PWPF",0,"MIT","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Training\u002Fopenrlhf\u002FSKILL.md",[],[18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37],"training","openrlhf training","OpenRLHF","PPO","for","orchestra","high-performance","use","Transformers","vLLM","Ray","RLOO","performance","RLHF","Post-Training","ZeRO","Distributed Training","DeepSpeed","framework","architecture","2026-11-10 10:12:30",false,"---\nname: openrlhf-training\ndescription: High-performance RLHF framework with Ray+vLLM acceleration. Use for PPO, GRPO, RLOO, DPO training of large models (7B-70B+). Built on Ray, vLLM, ZeRO-3. 2× faster than DeepSpeedChat with distributed architecture and GPU resource sharing.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Post-Training, OpenRLHF, RLHF, PPO, GRPO, RLOO, DPO, Ray, vLLM, Distributed Training, Large Models, ZeRO-3]\ndependencies: [openrlhf, ray, vllm, torch, transformers, deepspeed]\n---\n\n# OpenRLHF - High-Performance RLHF Training\n\n## Quick start\n\nOpenRLHF is a Ray-based RLHF framework optimized for distributed training with vLLM inference acceleration.\n\n**Installation**:\n```bash\n# Launch Docker container\ndocker run --runtime=nvidia -it --rm --shm-size=\"10g\" --cap-add=SYS_ADMIN \\\n  -v $PWD:\u002Fopenrlhf nvcr.io\u002Fnvidia\u002Fpytorch:25.02-py3 bash\n\n# Uninstall conflicts\nsudo pip uninstall xgboost transformer_engine flash_attn pynvml -y\n\n# Install OpenRLHF with vLLM\npip install openrlhf[vllm]\n```\n\n**PPO Training** (Hybrid Engine):\n```bash\nray start --head --node-ip-address 0.0.0.0 --num-gpus 8\n\nray job submit --address=\"http:\u002F\u002F127.0.0.1:8265\" \\\n  --runtime-env-json='{\"working_dir\": \"\u002Fopenrlhf\"}' \\\n  -- python3 -m openrlhf.cli.train_ppo_ray \\\n  --ref_num_nodes 1 --ref_num_gpus_per_node 8 \\\n  --reward_num_nodes 1 --reward_num_gpus_per_node 8 \\\n  --critic_num_nodes 1 --critic_num_gpus_per_node 8 \\\n  --actor_num_nodes 1 --actor_num_gpus_per_node 8 \\\n  --vllm_num_engines 4 --vllm_tensor_parallel_size 2 \\\n  --colocate_all_models \\\n  --vllm_gpu_memory_utilization 0.5 \\\n  --pretrain OpenRLHF\u002FLlama-3-8b-sft-mixture \\\n  --reward_pretrain OpenRLHF\u002FLlama-3-8b-rm-700k \\\n  --save_path .\u002Foutput\u002Fllama3-8b-rlhf \\\n  --micro_train_batch_size 8 --train_batch_size 128 \\\n  --micro_rollout_batch_size 16 --rollout_batch_size 1024 \\\n  --max_epochs 1 --prompt_max_len 1024 --generate_max_len 1024 \\\n  --zero_stage 3 --bf16 \\\n  --actor_learning_rate 5e-7 --critic_learning_rate 9e-6 \\\n  --init_kl_coef 0.01 --normalize_reward \\\n  --gradient_checkpointing --packing_samples \\\n  --vllm_enable_sleep --deepspeed_enable_sleep\n```\n\n**GRPO Training** (Group Normalized Policy Optimization):\n```bash\n# Same command as PPO, but add:\n--advantage_estimator group_norm\n```\n\n## Common workflows\n\n### Workflow 1: Full RLHF pipeline (SFT → Reward Model → PPO)\n\n**Step 1: Train reward model** (DPO):\n```bash\ndeepspeed --module openrlhf.cli.train_rm \\\n  --save_path .\u002Foutput\u002Fllama3-8b-rm \\\n  --save_steps -1 --logging_steps 1 \\\n  --eval_steps -1 --train_batch_size 256 \\\n  --micro_train_batch_size 1 --pretrain meta-llama\u002FMeta-Llama-3-8B \\\n  --bf16 --max_epochs 1 --max_len 8192 \\\n  --zero_stage 3 --learning_rate 9e-6 \\\n  --dataset OpenRLHF\u002Fpreference_dataset_mixture2_and_safe_pku \\\n  --apply_chat_template --chosen_key chosen \\\n  --rejected_key rejected --flash_attn --gradient_checkpointing\n```\n\n**Step 2: PPO training**:\n```bash\nray start --head --node-ip-address 0.0.0.0 --num-gpus 8\n\nray job submit --address=\"http:\u002F\u002F127.0.0.1:8265\" \\\n  -- python3 -m openrlhf.cli.train_ppo_ray \\\n  --ref_num_nodes 1 --ref_num_gpus_per_node 8 \\\n  --reward_num_nodes 1 --reward_num_gpus_per_node 8 \\\n  --critic_num_nodes 1 --critic_num_gpus_per_node 8 \\\n  --actor_num_nodes 1 --actor_num_gpus_per_node 8 \\\n  --vllm_num_engines 4 --vllm_tensor_parallel_size 2 \\\n  --colocate_all_models \\\n  --pretrain OpenRLHF\u002FLlama-3-8b-sft-mixture \\\n  --reward_pretrain .\u002Foutput\u002Fllama3-8b-rm \\\n  --save_path .\u002Foutput\u002Fllama3-8b-ppo \\\n  --micro_train_batch_size 8 --train_batch_size 128 \\\n  --micro_rollout_batch_size 16 --rollout_batch_size 1024 \\\n  --max_epochs 1 --prompt_max_len 1024 --generate_max_len 1024 \\\n  --zero_stage 3 --bf16 \\\n  --actor_learning_rate 5e-7 --critic_learning_rate 9e-6 \\\n  --init_kl_coef 0.01 --normalize_reward \\\n  --vllm_enable_sleep --deepspeed_enable_sleep\n```\n\n### Workflow 2: GRPO training (no critic model needed)\n\nMemory-efficient alternative to PPO:\n\n```bash\nray job submit --address=\"http:\u002F\u002F127.0.0.1:8265\" \\\n  -- python3 -m openrlhf.cli.train_ppo_ray \\\n  --advantage_estimator group_norm \\\n  --ref_num_nodes 1 --ref_num_gpus_per_node 8 \\\n  --reward_num_nodes 1 --reward_num_gpus_per_node 8 \\\n  --actor_num_nodes 1 --actor_num_gpus_per_node 8 \\\n  --vllm_num_engines 4 --vllm_tensor_parallel_size 2 \\\n  --colocate_all_models \\\n  --pretrain OpenRLHF\u002FLlama-3-8b-sft-mixture \\\n  --reward_pretrain OpenRLHF\u002FLlama-3-8b-rm-700k \\\n  --save_path .\u002Foutput\u002Fllama3-8b-grpo \\\n  --micro_train_batch_size 8 --train_batch_size 128 \\\n  --micro_rollout_batch_size 16 --rollout_batch_size 1024 \\\n  --max_epochs 1 --bf16 \\\n  --actor_learning_rate 5e-7 \\\n  --init_kl_coef 0.01 --use_kl_loss --kl_estimator k3 \\\n  --normalize_reward --no_advantage_std_norm\n```\n\n**Key GRPO parameters**:\n- `--advantage_estimator group_norm` - Enables GRPO\n- `--use_kl_loss` - KL loss from GRPO paper\n- `--kl_estimator k3` - Loss function (k2 ≈ k1)\n- `--no_advantage_std_norm` - Disables std normalization\n\n### Workflow 3: DPO training (preference optimization)\n\nSimpler alternative without reward model:\n\n```bash\ndeepspeed --module openrlhf.cli.train_dpo \\\n  --save_path .\u002Foutput\u002Fllama3-8b-dpo \\\n  --save_steps -1 --logging_steps 1 \\\n  --eval_steps -1 --train_batch_size 256 \\\n  --micro_train_batch_size 2 --pretrain meta-llama\u002FMeta-Llama-3-8B \\\n  --bf16 --max_epochs 1 --max_len 8192 \\\n  --zero_stage 3 --learning_rate 5e-7 --beta 0.1 \\\n  --dataset OpenRLHF\u002Fpreference_dataset_mixture2_and_safe_pku \\\n  --apply_chat_template --chosen_key chosen \\\n  --rejected_key rejected --flash_attn --gradient_checkpointing\n```\n\n## When to use vs alternatives\n\n**Use OpenRLHF when**:\n- Training large models (7B-70B+) with RL\n- Need vLLM inference acceleration\n- Want distributed architecture with Ray\n- Have multi-node GPU cluster\n- Need PPO\u002FGRPO\u002FRLOO\u002FDPO in one framework\n\n**Algorithm selection**:\n- **PPO**: Maximum control, best for complex rewards\n- **GRPO**: Memory-efficient, no critic needed\n- **RLOO**: Modified PPO with per-token KL\n- **REINFORCE++**: More stable than GRPO, faster than PPO\n- **DPO**: Simplest, no reward model needed\n\n**Use alternatives instead**:\n- **TRL**: Single-node training, simpler API\n- **veRL**: ByteDance's framework for 671B models\n- **DeepSpeedChat**: Integrated with DeepSpeed ecosystem\n\n## Common issues\n\n**Issue: GPU OOM with large models**\n\nDisable model colocation:\n```bash\n# Remove --colocate_all_models flag\n# Allocate separate GPUs for each model\n--actor_num_gpus_per_node 8 \\\n--critic_num_gpus_per_node 8 \\\n--reward_num_gpus_per_node 8 \\\n--ref_num_gpus_per_node 8\n```\n\n**Issue: DeepSpeed GPU index out of range**\n\nSet environment variable:\n```bash\nexport RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES=1\n```\n\n**Issue: Training instability**\n\nUse Hybrid Engine instead of async:\n```bash\n--colocate_all_models \\\n--vllm_enable_sleep \\\n--deepspeed_enable_sleep\n```\n\nAdjust KL coefficient:\n```bash\n--init_kl_coef 0.05  # Increase from 0.01\n```\n\n**Issue: Slow generation during PPO**\n\nEnable vLLM acceleration:\n```bash\n--vllm_num_engines 4 \\\n--vllm_tensor_parallel_size 2 \\\n--vllm_gpu_memory_utilization 0.5\n```\n\n## Advanced topics\n\n**Hybrid Engine GPU sharing**: See [references\u002Fhybrid-engine.md](references\u002Fhybrid-engine.md) for vLLM sleep mode, DeepSpeed sleep mode, and optimal node allocation.\n\n**Algorithm comparison**: See [references\u002Falgorithm-comparison.md](references\u002Falgorithm-comparison.md) for PPO vs GRPO vs RLOO vs REINFORCE++ benchmarks and hyperparameters.\n\n**Multi-node setup**: See [references\u002Fmulti-node-training.md](references\u002Fmulti-node-training.md) for Ray cluster configuration and fault tolerance.\n\n**Custom reward functions**: See [references\u002Fcustom-rewards.md](references\u002Fcustom-rewards.md) for reinforced fine-tuning and agent RLHF.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA A100\u002FH100 recommended\n- **VRAM**:\n  - 7B model: 8× A100 40GB (Hybrid Engine)\n  - 70B model: 48× A100 80GB (vLLM:Actor:Critic = 1:1:1)\n- **Multi-node**: Ray cluster with InfiniBand recommended\n- **Docker**: NVIDIA PyTorch container 25.02+\n\n**Performance**:\n- 2× faster than DeepSpeedChat\n- vLLM inference acceleration\n- Hybrid Engine minimizes GPU idle time\n\n## Resources\n\n- Docs: https:\u002F\u002Fgithub.com\u002FOpenRLHF\u002FOpenRLHF\n- Paper: https:\u002F\u002Farxiv.org\u002Fabs\u002F2405.11143\n- Examples: https:\u002F\u002Fgithub.com\u002FOpenRLHF\u002FOpenRLHF\u002Ftree\u002Fmain\u002Fexamples\n- Discord: Community support\n\n\n\n","# OpenRLHF - High-Performance RLHF Training\n\n## Quick start\n\nOpenRLHF is a Ray-based RLHF framework optimized for distributed training with vLLM inference acceleration.\n\n**Installation**:\n```bash\n# Launch Docker container\ndocker run --runtime=nvidia -it --rm --shm-size=\"10g\" --cap-add=SYS_ADMIN \\\n  -v $PWD:\u002Fopenrlhf nvcr.io\u002Fnvidia\u002Fpytorch:25.02-py3 bash\n\n# Uninstall conflicts\nsudo pip uninstall xgboost transformer_engine flash_attn pynvml -y\n\n# Install OpenRLHF with vLLM\npip install openrlhf[vllm]\n```\n\n**PPO Training** (Hybrid Engine):\n```bash\nray start --head --node-ip-address 0.0.0.0 --num-gpus 8\n\nray job submit --address=\"http:\u002F\u002F127.0.0.1:8265\" \\\n  --runtime-env-json='{\"working_dir\": \"\u002Fopenrlhf\"}' \\\n  -- python3 -m openrlhf.cli.train_ppo_ray \\\n  --ref_num_nodes 1 --ref_num_gpus_per_node 8 \\\n  --reward_num_nodes 1 --reward_num_gpus_per_node 8 \\\n  --critic_num_nodes 1 --critic_num_gpus_per_node 8 \\\n  --actor_num_nodes 1 --actor_num_gpus_per_node 8 \\\n  --vllm_num_engines 4 --vllm_tensor_parallel_size 2 \\\n  --colocate_all_models \\\n  --vllm_gpu_memory_utilization 0.5 \\\n  --pretrain OpenRLHF\u002FLlama-3-8b-sft-mixture \\\n  --reward_pretrain OpenRLHF\u002FLlama-3-8b-rm-700k \\\n  --save_path .\u002Foutput\u002Fllama3-8b-rlhf \\\n  --micro_train_batch_size 8 --train_batch_size 128 \\\n  --micro_rollout_batch_size 16 --rollout_batch_size 1024 \\\n  --max_epochs 1 --prompt_max_len 1024 --generate_max_len 1024 \\\n  --zero_stage 3 --bf16 \\\n  --actor_learning_rate 5e-7 --critic_learning_rate 9e-6 \\\n  --init_kl_coef 0.01 --normalize_reward \\\n  --gradient_checkpointing --packing_samples \\\n  --vllm_enable_sleep --deepspeed_enable_sleep\n```\n\n**GRPO Training** (Group Normalized Policy Optimization):\n```bash\n# Same command as PPO, but add:\n--advantage_estimator group_norm\n```\n\n## Common workflows\n\n### Workflow 1: Full RLHF pipeline (SFT → Reward Model → PPO)\n\n**Step 1: Train reward model** (DPO):\n```bash\ndeepspeed --module openrlhf.cli.train_rm \\\n  --save_path .\u002Foutput\u002Fllama3-8b-rm \\\n  --save_steps -1 --logging_steps 1 \\\n  --eval_steps -1 --train_batch_size 256 \\\n  --micro_train_batch_size 1 --pretrain meta-llama\u002FMeta-Llama-3-8B \\\n  --bf16 --max_epochs 1 --max_len 8192 \\\n  --zero_stage 3 --learning_rate 9e-6 \\\n  --dataset OpenRLHF\u002Fpreference_dataset_mixture2_and_safe_pku \\\n  --apply_chat_template --chosen_key chosen \\\n  --rejected_key rejected --flash_attn --gradient_checkpointing\n```\n\n**Step 2: PPO training**:\n```bash\nray start --head --node-ip-address 0.0.0.0 --num-gpus 8\n\nray job submit --address=\"http:\u002F\u002F127.0.0.1:8265\" \\\n  -- python3 -m openrlhf.cli.train_ppo_ray \\\n  --ref_num_nodes 1 --ref_num_gpus_per_node 8 \\\n  --reward_num_nodes 1 --reward_num_gpus_per_node 8 \\\n  --critic_num_nodes 1 --critic_num_gpus_per_node 8 \\\n  --actor_num_nodes 1 --actor_num_gpus_per_node 8 \\\n  --vllm_num_engines 4 --vllm_tensor_parallel_size 2 \\\n  --colocate_all_models \\\n  --pretrain OpenRLHF\u002FLlama-3-8b-sft-mixture \\\n  --reward_pretrain .\u002Foutput\u002Fllama3-8b-rm \\\n  --save_path .\u002Foutput\u002Fllama3-8b-ppo \\\n  --micro_train_batch_size 8 --train_batch_size 128 \\\n  --micro_rollout_batch_size 16 --rollout_batch_size 1024 \\\n  --max_epochs 1 --prompt_max_len 1024 --generate_max_len 1024 \\\n  --zero_stage 3 --bf16 \\\n  --actor_learning_rate 5e-7 --critic_learning_rate 9e-6 \\\n  --init_kl_coef 0.01 --normalize_reward \\\n  --vllm_enable_sleep --deepspeed_enable_sleep\n```\n\n### Workflow 2: GRPO training (no critic model needed)\n\nMemory-efficient alternative to PPO:\n\n```bash\nray job submit --address=\"http:\u002F\u002F127.0.0.1:8265\" \\\n  -- python3 -m openrlhf.cli.train_ppo_ray \\\n  --advantage_estimator group_norm \\\n  --ref_num_nodes 1 --ref_num_gpus_per_node 8 \\\n  --reward_num_nodes 1 --reward_num_gpus_per_node 8 \\\n  --actor_num_nodes 1 --actor_num_gpus_per_node 8 \\\n  --vllm_num_engines 4 --vllm_tensor_parallel_size 2 \\\n  --colocate_all_models \\\n  --pretrain OpenRLHF\u002FLlama-3-8b-sft-mixture \\\n  --reward_pretrain OpenRLHF\u002FLlama-3-8b-rm-700k \\\n  --save_path .\u002Foutput\u002Fllama3-8b-grpo \\\n  --micro_train_batch_size 8 --train_batch_size 128 \\\n  --micro_rollout_batch_size 16 --rollout_batch_size 1024 \\\n  --max_epochs 1 --bf16 \\\n  --actor_learning_rate 5e-7 \\\n  --init_kl_coef 0.01 --use_kl_loss --kl_estimator k3 \\\n  --normalize_reward --no_advantage_std_norm\n```\n\n**Key GRPO parameters**:\n- `--advantage_estimator group_norm` - Enables GRPO\n- `--use_kl_loss` - KL loss from GRPO paper\n- `--kl_estimator k3` - Loss function (k2 ≈ k1)\n- `--no_advantage_std_norm` - Disables std normalization\n\n### Workflow 3: DPO training (preference optimization)\n\nSimpler alternative without reward model:\n\n```bash\ndeepspeed --module openrlhf.cli.train_dpo \\\n  --save_path .\u002Foutput\u002Fllama3-8b-dpo \\\n  --save_steps -1 --logging_steps 1 \\\n  --eval_steps -1 --train_batch_size 256 \\\n  --micro_train_batch_size 2 --pretrain meta-llama\u002FMeta-Llama-3-8B \\\n  --bf16 --max_epochs 1 --max_len 8192 \\\n  --zero_stage 3 --learning_rate 5e-7 --beta 0.1 \\\n  --dataset OpenRLHF\u002Fpreference_dataset_mixture2_and_safe_pku \\\n  --apply_chat_template --chosen_key chosen \\\n  --rejected_key rejected --flash_attn --gradient_checkpointing\n```\n\n## When to use vs alternatives\n\n**Use OpenRLHF when**:\n- Training large models (7B-70B+) with RL\n- Need vLLM inference acceleration\n- Want distributed architecture with Ray\n- Have multi-node GPU cluster\n- Need PPO\u002FGRPO\u002FRLOO\u002FDPO in one framework\n\n**Algorithm selection**:\n- **PPO**: Maximum control, best for complex rewards\n- **GRPO**: Memory-efficient, no critic needed\n- **RLOO**: Modified PPO with per-token KL\n- **REINFORCE++**: More stable than GRPO, faster than PPO\n- **DPO**: Simplest, no reward model needed\n\n**Use alternatives instead**:\n- **TRL**: Single-node training, simpler API\n- **veRL**: ByteDance's framework for 671B models\n- **DeepSpeedChat**: Integrated with DeepSpeed ecosystem\n\n## Common issues\n\n**Issue: GPU OOM with large models**\n\nDisable model colocation:\n```bash\n# Remove --colocate_all_models flag\n# Allocate separate GPUs for each model\n--actor_num_gpus_per_node 8 \\\n--critic_num_gpus_per_node 8 \\\n--reward_num_gpus_per_node 8 \\\n--ref_num_gpus_per_node 8\n```\n\n**Issue: DeepSpeed GPU index out of range**\n\nSet environment variable:\n```bash\nexport RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES=1\n```\n\n**Issue: Training instability**\n\nUse Hybrid Engine instead of async:\n```bash\n--colocate_all_models \\\n--vllm_enable_sleep \\\n--deepspeed_enable_sleep\n```\n\nAdjust KL coefficient:\n```bash\n--init_kl_coef 0.05  # Increase from 0.01\n```\n\n**Issue: Slow generation during PPO**\n\nEnable vLLM acceleration:\n```bash\n--vllm_num_engines 4 \\\n--vllm_tensor_parallel_size 2 \\\n--vllm_gpu_memory_utilization 0.5\n```\n\n## Advanced topics\n\n**Hybrid Engine GPU sharing**: See [references\u002Fhybrid-engine.md](references\u002Fhybrid-engine.md) for vLLM sleep mode, DeepSpeed sleep mode, and optimal node allocation.\n\n**Algorithm comparison**: See [references\u002Falgorithm-comparison.md](references\u002Falgorithm-comparison.md) for PPO vs GRPO vs RLOO vs REINFORCE++ benchmarks and hyperparameters.\n\n**Multi-node setup**: See [references\u002Fmulti-node-training.md](references\u002Fmulti-node-training.md) for Ray cluster configuration and fault tolerance.\n\n**Custom reward functions**: See [references\u002Fcustom-rewards.md](references\u002Fcustom-rewards.md) for reinforced fine-tuning and agent RLHF.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA A100\u002FH100 recommended\n- **VRAM**:\n  - 7B model: 8× A100 40GB (Hybrid Engine)\n  - 70B model: 48× A100 80GB (vLLM:Actor:Critic = 1:1:1)\n- **Multi-node**: Ray cluster with InfiniBand recommended\n- **Docker**: NVIDIA PyTorch container 25.02+\n\n**Performance**:\n- 2× faster than DeepSpeedChat\n- vLLM inference acceleration\n- Hybrid Engine minimizes GPU idle time\n\n## Resources\n\n- Docs: https:\u002F\u002Fgithub.com\u002FOpenRLHF\u002FOpenRLHF\n- Paper: https:\u002F\u002Farxiv.org\u002Fabs\u002F2405.11143\n- Examples: https:\u002F\u002Fgithub.com\u002FOpenRLHF\u002FOpenRLHF\u002Ftree\u002Fmain\u002Fexamples\n- Discord: Community support\n\n\n\n",{"name":5,"tags":43,"author":48,"license":14,"version":49,"description":7,"dependencies":50},[32,20,31,21,44,29,45,28,27,34,46,47],"GRPO","DPO","Large Models","ZeRO-3","Orchestra Research","1.0.0",[51,52,53,54,55,56],"openrlhf","ray","vllm","torch","transformers","deepspeed","main","f19b254e8975d0bfddb394686706106930d7964d","AI-Training\u002Fopenrlhf","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus","https:\u002F\u002Fgithub.com\u002Fendman100","2026-11-11 18:19:10",{"source":8,"skillName":5,"commands":64,"prompt":74,"zipAvailable":75},[65,68,71],{"tool":66,"command":67},"npx","npx skills add endman100\u002Fskill-router-nexus --skill openrlhf-training",{"tool":69,"command":70},"bunx","bunx skills add endman100\u002Fskill-router-nexus --skill openrlhf-training",{"tool":72,"command":73},"pnpm","pnpm dlx skills add endman100\u002Fskill-router-nexus --skill openrlhf-training","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Training\u002Fopenrlhf\u002FSKILL.md",true,[77,83,87,91],{"path":78,"resourceType":79,"resourceTypeDesc":80,"sizeBytes":81,"sourceHtmlUrl":82},"references\u002Falgorithm-comparison.md","REFERENCE","参考文档",9783,"https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Training\u002Fopenrlhf\u002Freferences\u002Falgorithm-comparison.md",{"path":84,"resourceType":79,"resourceTypeDesc":80,"sizeBytes":85,"sourceHtmlUrl":86},"references\u002Fcustom-rewards.md",15865,"https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Training\u002Fopenrlhf\u002Freferences\u002Fcustom-rewards.md",{"path":88,"resourceType":79,"resourceTypeDesc":80,"sizeBytes":89,"sourceHtmlUrl":90},"references\u002Fhybrid-engine.md",7266,"https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Training\u002Fopenrlhf\u002Freferences\u002Fhybrid-engine.md",{"path":92,"resourceType":79,"resourceTypeDesc":80,"sizeBytes":93,"sourceHtmlUrl":94},"references\u002Fmulti-node-training.md",11099,"https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Training\u002Fopenrlhf\u002Freferences\u002Fmulti-node-training.md",[96,120,156,179,188,212],{"skillNo":97,"name":98,"nameZh":6,"description":99,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":100,"sourceHtmlUrl":101,"categories":102,"keywords":108,"publishedAt":38},"01M6MPCT5BXVER8HQZTW3QE9AS","create-master","基于佛教经典文献，生成特定高僧大德的 AI 教学角色","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FMetaphysics-and-Spirituality\u002FMaster-skill\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FMetaphysics-and-Spirituality\u002FMaster-skill\u002FSKILL.md",[103],{"categoryNo":104,"categoryName":105,"taxonomyType":106,"taxonomyTypeDesc":107},"01JBSEEDCAT000000000000039","教师与教育","PROFESSION","适用职业",[109,110,111,112,113,114,115,116,117,118,119],"create master","master","create","生成","true","webfetch","edit","write","read","grep","Bash",{"skillNo":121,"name":122,"nameZh":6,"description":123,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":124,"sourceHtmlUrl":125,"categories":126,"keywords":138,"publishedAt":38},"01M6M43AKE95Y64E1JRXEF5QQ3","writing-dna-skill","从至少 20 篇完整文章中蒸馏可复用的写作 DNA，分析语言、文章结构、选题逻辑、素材策略、认知框架和视觉风格，并生成 Writing-DNA.md；按该 DNA 写作时会读取全部蒸馏产物和 5 篇相关原文。用于中英文作者、账号、品牌或出版物的风格分析与一致性写作。 Distill reusable Writing DNA from at least 20 complete articles for Chinese or English authors, publications, brands, and accounts; use for language, structure, topic logic, source strategy, cognitive-frame, and visual-style analysis.","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fwriting-dna-skill\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fwriting-dna-skill\u002FSKILL.md",[127,132,135],{"categoryNo":128,"categoryName":129,"taxonomyType":130,"taxonomyTypeDesc":131},"01JBSEEDCAT000000000000018","写作与文案","SCENARIO","使用场景",{"categoryNo":133,"categoryName":134,"taxonomyType":130,"taxonomyTypeDesc":131},"01JBSEEDCAT000000000000023","Skill与智能体开发",{"categoryNo":136,"categoryName":137,"taxonomyType":106,"taxonomyTypeDesc":107},"01JBSEEDCAT000000000000036","内容创作者",[139,140,141,142,143,144,145,146,147,148,25,149,150,112,151,152,153,22,154,155],"writing dna skill","DNA","writing","strategy","english","logic","visual","structure","chinese","reusable","style","distill","analysis","frame","source","at","complete",{"skillNo":157,"name":158,"nameZh":6,"description":159,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":160,"sourceHtmlUrl":161,"categories":162,"keywords":165,"publishedAt":38},"01M6M428TTCGQ4ME3KY99VN037","stop-slop","Remove AI writing patterns from prose. Use when drafting, editing, or reviewing text to eliminate predictable AI tells.","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fstop-slop\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fstop-slop\u002FSKILL.md",[163,164],{"categoryNo":128,"categoryName":129,"taxonomyType":130,"taxonomyTypeDesc":131},{"categoryNo":136,"categoryName":137,"taxonomyType":106,"taxonomyTypeDesc":107},[166,167,168,169,170,171,172,173,174,25,175,176,177,22,178,141],"slop","stop slop","stop","drafting","when","drafts","https","patterns","prose","remove","editing","text","content",{"skillNo":180,"name":181,"nameZh":6,"description":182,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":183,"sourceHtmlUrl":184,"categories":185,"keywords":186,"publishedAt":38},"01M6M41X0B7EBX42TK435DMGGR","shuorenhua","检查和清理中英文文本里的 AI 套路，适用于“去 AI 味”“说人话”“自然一点”“别像模板”“先标问题”这类改写和审稿需求；按场景控制力度，同时保留事实、术语、语域和责任主体。","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fshuorenhua\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fshuorenhua\u002FSKILL.md",[],[181,187],"文本",{"skillNo":189,"name":190,"nameZh":6,"description":191,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":192,"sourceHtmlUrl":193,"categories":194,"keywords":195,"publishedAt":38},"01M6M3ZD12WBEW5KXKDM4N6DFG","naming","Name products, SaaS, brands, open source projects, bots, and apps. Use when the user needs to name something, find a brand name, or pick a product name. Metaphor-driven process that produces memorable, meaningful names and avoids AI slop.","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fnaming-main\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fnaming-main\u002FSKILL.md",[],[190,196,197,198,199,200,25,201,117,153,202,203,204,205,119,206,207,208,209,170,210,211,166],"repo","websearch","product name","apps","bots","view","brand","pick","gh","projects","open","saas","open source","product","describe","ai slop",{"skillNo":213,"name":214,"nameZh":6,"description":215,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":216,"sourceHtmlUrl":217,"categories":218,"keywords":220,"publishedAt":38},"01M6M3YVXAFAAWJ7F6JMPFREYE","humanizer-zh","去除文本中的 AI 生成痕迹。适用于编辑或审阅文本，使其听起来更自然、更像人类书写。\n基于维基百科的\"AI 写作特征\"综合指南。检测并修复以下模式：夸大的象征意义、\n宣传性语言、以 -ing 结尾的肤浅分析、模糊的归因、破折号过度使用、三段式法则、\nAI 词汇、否定式排比、过多的连接性短语。","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fhumanizer-zh\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fhumanizer-zh\u002FSKILL.md",[219],{"categoryNo":128,"categoryName":129,"taxonomyType":130,"taxonomyTypeDesc":131},[221,222,223,117,116,115,141,112,187,224,166,168],"humanizer","zh","humanizer zh","翻译"]