[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6KQMAMXND3YWMDB0ERJM0T2":3,"skill-same:01M6KQMAMXND3YWMDB0ERJM0T2":90},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":56,"commitSha":57,"skillDirectory":58,"forksCount":13,"repositoryHtmlUrl":59,"repositoryHomepageUrl":6,"ownerHtmlUrl":60,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":61,"install":62,"resources":75},"01M6KQMAMXND3YWMDB0ERJM0T2","moe-training",null,"Train Mixture of Experts (MoE) models using DeepSpeed or HuggingFace. Use when training large-scale models with limited compute (5× cost reduction vs dense models), implementing sparse architectures like Mixtral 8x7B or DeepSeek-V3, or scaling model capacity without proportional compute increase. Covers MoE architectures, routing mechanisms, load balancing, expert parallelism, and inference optimization.","endman100\u002Fskill-router-nexus","endman100","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F31997019?v=4","01M6KPDX17BYDWW4RRB0TNGGMX","01M6KPDX64Y2QBEWVFG5W5PWPF",0,"MIT","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Training\u002Fmoe-training\u002FSKILL.md",[],[18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"moe training","moe","training","Emerging Techniques","when","implementing","orchestra","experts","Efficient Training","Mixtral","Expert Parallelism","Sparse Models","Mixture Of Experts","Accelerate","architectures","scaling","expert","model","Transformers","Optimization","DeepSpeed","2026-11-10 10:12:30",false,"---\nname: moe-training\ndescription: Train Mixture of Experts (MoE) models using DeepSpeed or HuggingFace. Use when training large-scale models with limited compute (5× cost reduction vs dense models), implementing sparse architectures like Mixtral 8x7B or DeepSeek-V3, or scaling model capacity without proportional compute increase. Covers MoE architectures, routing mechanisms, load balancing, expert parallelism, and inference optimization.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Emerging Techniques, MoE, Mixture Of Experts, Sparse Models, DeepSpeed, Expert Parallelism, Mixtral, DeepSeek, Routing, Load Balancing, Efficient Training]\ndependencies: [deepspeed, transformers, torch, accelerate]\n---\n\n# MoE Training: Mixture of Experts\n\n## When to Use This Skill\n\nUse MoE Training when you need to:\n- **Train larger models** with limited compute (5× cost reduction vs dense models)\n- **Scale model capacity** without proportional compute increase\n- **Achieve better performance** per compute budget than dense models\n- **Specialize experts** for different domains\u002Ftasks\u002Flanguages\n- **Reduce inference latency** with sparse activation (only 13B\u002F47B params active in Mixtral)\n- **Implement SOTA models** like Mixtral 8x7B, DeepSeek-V3, Switch Transformers\n\n**Notable MoE Models**: Mixtral 8x7B (Mistral AI), DeepSeek-V3, Switch Transformers (Google), GLaM (Google), NLLB-MoE (Meta)\n\n## Installation\n\n```bash\n# DeepSpeed with MoE support\npip install deepspeed>=0.6.0\n\n# Megatron-DeepSpeed for large-scale training\ngit clone https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FMegatron-DeepSpeed\ncd Megatron-DeepSpeed\npip install -r requirements.txt\n\n# Alternative: HuggingFace Transformers\npip install transformers accelerate\n```\n\n## Quick Start\n\n### Basic MoE Architecture\n\n```python\nimport torch\nimport torch.nn as nn\n\nclass MoELayer(nn.Module):\n    \"\"\"Sparse Mixture of Experts layer.\"\"\"\n\n    def __init__(self, hidden_size, num_experts=8, top_k=2):\n        super().__init__()\n        self.num_experts = num_experts\n        self.top_k = top_k\n\n        # Expert networks (FFN)\n        self.experts = nn.ModuleList([\n            nn.Sequential(\n                nn.Linear(hidden_size, 4 * hidden_size),\n                nn.GELU(),\n                nn.Linear(4 * hidden_size, hidden_size)\n            )\n            for _ in range(num_experts)\n        ])\n\n        # Gating network (router)\n        self.gate = nn.Linear(hidden_size, num_experts)\n\n    def forward(self, x):\n        # x shape: (batch_size, seq_len, hidden_size)\n        batch_size, seq_len, hidden_size = x.shape\n\n        # Flatten for routing\n        x_flat = x.view(-1, hidden_size)  # (batch_size * seq_len, hidden_size)\n\n        # Compute gate scores\n        gate_logits = self.gate(x_flat)  # (batch_size * seq_len, num_experts)\n\n        # Top-k routing\n        gate_scores = torch.softmax(gate_logits, dim=-1)\n        topk_scores, topk_indices = torch.topk(gate_scores, self.top_k, dim=-1)\n\n        # Normalize top-k scores\n        topk_scores = topk_scores \u002F topk_scores.sum(dim=-1, keepdim=True)\n\n        # Dispatch and combine expert outputs\n        output = torch.zeros_like(x_flat)\n\n        for i in range(self.top_k):\n            expert_idx = topk_indices[:, i]\n            expert_scores = topk_scores[:, i].unsqueeze(-1)\n\n            # Route tokens to experts\n            for expert_id in range(self.num_experts):\n                mask = (expert_idx == expert_id)\n                if mask.any():\n                    expert_input = x_flat[mask]\n                    expert_output = self.experts[expert_id](expert_input)\n                    output[mask] += expert_scores[mask] * expert_output\n\n        # Reshape back\n        return output.view(batch_size, seq_len, hidden_size)\n```\n\n### DeepSpeed MoE Training\n\n```bash\n# Training script with MoE\ndeepspeed pretrain_gpt_moe.py \\\n  --num-layers 24 \\\n  --hidden-size 1024 \\\n  --num-attention-heads 16 \\\n  --seq-length 2048 \\\n  --max-position-embeddings 2048 \\\n  --micro-batch-size 4 \\\n  --global-batch-size 256 \\\n  --train-iters 500000 \\\n  --lr 0.0001 \\\n  --min-lr 0.00001 \\\n  --lr-decay-style cosine \\\n  --num-experts 128 \\\n  --moe-expert-parallel-size 4 \\\n  --moe-loss-coeff 0.01 \\\n  --moe-train-capacity-factor 1.25 \\\n  --moe-eval-capacity-factor 2.0 \\\n  --fp16 \\\n  --deepspeed_config ds_config.json\n```\n\n## Core Concepts\n\n### 1. MoE Architecture\n\n**Key Components:**\n- **Experts**: Multiple specialized FFN networks (typically 8-128)\n- **Router\u002FGate**: Learned network that selects which experts to use\n- **Top-k Routing**: Activate only k experts per token (k=1 or k=2)\n- **Load Balancing**: Ensure even expert utilization\n\n```\nInput Token\n    ↓\nRouter (Gate Network)\n    ↓\nTop-k Expert Selection (e.g., 2 out of 8)\n    ↓\nExpert 1 (weight: 0.6) + Expert 5 (weight: 0.4)\n    ↓\nWeighted Combination\n    ↓\nOutput\n```\n\n### 2. Routing Mechanisms\n\n**Top-1 Routing (Switch Transformer):**\n```python\n# Simplest routing: one expert per token\ngate_logits = router(x)  # (batch, seq_len, num_experts)\nexpert_idx = torch.argmax(gate_logits, dim=-1)  # Hard routing\n```\n\n**Top-2 Routing (Mixtral):**\n```python\n# Top-2: two experts per token\ngate_scores = torch.softmax(router(x), dim=-1)\ntop2_scores, top2_indices = torch.topk(gate_scores, k=2, dim=-1)\n\n# Normalize scores\ntop2_scores = top2_scores \u002F top2_scores.sum(dim=-1, keepdim=True)\n\n# Combine expert outputs\noutput = (top2_scores[:, :, 0:1] * expert_outputs[top2_indices[:, :, 0]] +\n          top2_scores[:, :, 1:2] * expert_outputs[top2_indices[:, :, 1]])\n```\n\n**Expert Choice Routing:**\n```python\n# Experts choose top-k tokens (instead of tokens choosing experts)\n# Guarantees perfect load balancing\nexpert_scores = router(x).transpose(-1, -2)  # (batch, num_experts, seq_len)\ntopk_tokens = torch.topk(expert_scores, k=capacity_per_expert, dim=-1)\n```\n\n### 3. Load Balancing\n\n**Auxiliary Loss:**\n```python\ndef load_balancing_loss(gate_logits, expert_indices, num_experts):\n    \"\"\"Encourage uniform expert usage.\"\"\"\n    # Fraction of tokens routed to each expert\n    expert_counts = torch.bincount(expert_indices.flatten(), minlength=num_experts)\n    expert_fraction = expert_counts.float() \u002F expert_indices.numel()\n\n    # Gate probability for each expert (average across tokens)\n    gate_probs = torch.softmax(gate_logits, dim=-1).mean(dim=0)\n\n    # Auxiliary loss: encourage alignment\n    aux_loss = num_experts * (expert_fraction * gate_probs).sum()\n\n    return aux_loss\n\n# Add to main loss\ntotal_loss = language_model_loss + 0.01 * load_balancing_loss(...)\n```\n\n**Router Z-Loss (Stability):**\n```python\ndef router_z_loss(logits):\n    \"\"\"Encourage router to have lower entropy (more decisive).\"\"\"\n    z_loss = torch.logsumexp(logits, dim=-1).pow(2).mean()\n    return z_loss\n\ntotal_loss = lm_loss + 0.01 * aux_loss + 0.001 * router_z_loss(gate_logits)\n```\n\n### 4. Expert Parallelism\n\n```python\n# DeepSpeed configuration\n{\n  \"train_batch_size\": 256,\n  \"fp16\": {\"enabled\": true},\n  \"moe\": {\n    \"enabled\": true,\n    \"num_experts\": 128,\n    \"expert_parallel_size\": 8,  # Distribute 128 experts across 8 GPUs\n    \"capacity_factor\": 1.25,    # Expert capacity = tokens_per_batch * capacity_factor \u002F num_experts\n    \"drop_tokens\": true,        # Drop tokens exceeding capacity\n    \"use_residual\": false\n  }\n}\n```\n\n## Training Configuration\n\n### DeepSpeed MoE Config\n\n```json\n{\n  \"train_batch_size\": 256,\n  \"gradient_accumulation_steps\": 1,\n  \"optimizer\": {\n    \"type\": \"Adam\",\n    \"params\": {\n      \"lr\": 0.0001,\n      \"betas\": [0.9, 0.999],\n      \"eps\": 1e-8\n    }\n  },\n  \"fp16\": {\n    \"enabled\": true,\n    \"loss_scale\": 0,\n    \"initial_scale_power\": 16\n  },\n  \"moe\": {\n    \"enabled\": true,\n    \"num_experts\": 128,\n    \"expert_parallel_size\": 8,\n    \"moe_loss_coeff\": 0.01,\n    \"train_capacity_factor\": 1.25,\n    \"eval_capacity_factor\": 2.0,\n    \"min_capacity\": 4,\n    \"drop_tokens\": true,\n    \"use_residual\": false,\n    \"use_tutel\": false\n  },\n  \"zero_optimization\": {\n    \"stage\": 1\n  }\n}\n```\n\n### Training Script\n\n```bash\n#!\u002Fbin\u002Fbash\n\n# Mixtral-style MoE training\ndeepspeed --num_gpus 8 pretrain_moe.py \\\n  --model-parallel-size 1 \\\n  --num-layers 32 \\\n  --hidden-size 4096 \\\n  --num-attention-heads 32 \\\n  --seq-length 2048 \\\n  --max-position-embeddings 4096 \\\n  --micro-batch-size 2 \\\n  --global-batch-size 256 \\\n  --train-iters 500000 \\\n  --save-interval 5000 \\\n  --eval-interval 1000 \\\n  --eval-iters 100 \\\n  --lr 0.0001 \\\n  --min-lr 0.00001 \\\n  --lr-decay-style cosine \\\n  --lr-warmup-iters 2000 \\\n  --clip-grad 1.0 \\\n  --weight-decay 0.1 \\\n  --num-experts 8 \\\n  --moe-expert-parallel-size 4 \\\n  --moe-loss-coeff 0.01 \\\n  --moe-train-capacity-factor 1.25 \\\n  --moe-eval-capacity-factor 2.0 \\\n  --disable-moe-token-dropping \\\n  --fp16 \\\n  --deepspeed \\\n  --deepspeed_config ds_config_moe.json \\\n  --data-path \u002Fpath\u002Fto\u002Fdata \\\n  --vocab-file \u002Fpath\u002Fto\u002Fvocab.json \\\n  --merge-file \u002Fpath\u002Fto\u002Fmerges.txt\n```\n\n## Advanced Patterns\n\n### Mixtral 8x7B Architecture\n\n```python\nclass MixtralMoEBlock(nn.Module):\n    \"\"\"Mixtral-style MoE block with 8 experts, top-2 routing.\"\"\"\n\n    def __init__(self, config):\n        super().__init__()\n        self.hidden_dim = config.hidden_size\n        self.ffn_dim = config.intermediate_size\n        self.num_experts = config.num_local_experts  # 8\n        self.top_k = config.num_experts_per_tok       # 2\n\n        # 8 expert FFNs\n        self.experts = nn.ModuleList([\n            nn.Sequential(\n                nn.Linear(self.hidden_dim, self.ffn_dim, bias=False),\n                nn.SiLU(),\n                nn.Linear(self.ffn_dim, self.hidden_dim, bias=False)\n            )\n            for _ in range(self.num_experts)\n        ])\n\n        # Router\n        self.gate = nn.Linear(self.hidden_dim, self.num_experts, bias=False)\n\n    def forward(self, hidden_states):\n        batch_size, sequence_length, hidden_dim = hidden_states.shape\n\n        # Flatten\n        hidden_states = hidden_states.view(-1, hidden_dim)\n\n        # Router logits\n        router_logits = self.gate(hidden_states)  # (batch * seq_len, num_experts)\n\n        # Softmax and top-2\n        routing_weights = torch.softmax(router_logits, dim=1)\n        routing_weights, selected_experts = torch.topk(routing_weights, self.top_k, dim=-1)\n\n        # Normalize routing weights\n        routing_weights \u002F= routing_weights.sum(dim=-1, keepdim=True)\n\n        # Initialize output\n        final_hidden_states = torch.zeros_like(hidden_states)\n\n        # Route to experts\n        for expert_idx in range(self.num_experts):\n            expert_layer = self.experts[expert_idx]\n            idx, top_x = torch.where(selected_experts == expert_idx)\n\n            if idx.shape[0] == 0:\n                continue\n\n            # Current expert tokens\n            current_hidden_states = hidden_states[idx]\n\n            # Expert forward\n            current_hidden_states = expert_layer(current_hidden_states)\n\n            # Weighted by routing scores\n            current_hidden_states *= routing_weights[idx, top_x, None]\n\n            # Accumulate\n            final_hidden_states.index_add_(0, idx, current_hidden_states)\n\n        # Reshape\n        return final_hidden_states.view(batch_size, sequence_length, hidden_dim)\n```\n\n### PR-MoE (Pyramid-Residual-MoE)\n\n```bash\n# DeepSpeed PR-MoE: 3x better parameter efficiency\ndeepspeed pretrain_gpt_moe.py \\\n  --num-layers 24 \\\n  --hidden-size 1024 \\\n  --num-attention-heads 16 \\\n  --num-experts \"[128, 64, 32, 16]\" \\\n  --mlp-type residual \\\n  --moe-expert-parallel-size 4 \\\n  --moe-loss-coeff 0.01 \\\n  --fp16\n```\n\n## Best Practices\n\n### 1. Expert Count Selection\n\n```python\n# Rule of thumb: More experts = more capacity, but diminishing returns\n# Typical configurations:\n# - Small models (1B-7B): 8-16 experts\n# - Medium models (7B-30B): 8-64 experts\n# - Large models (30B+): 64-256 experts\n\n# Example: Mixtral 8x7B\n# Total params: 47B (8 experts × 7B each)\n# Active params: 13B (2 experts × 7B, top-2 routing)\n# Efficiency: 47B capacity with 13B compute\n```\n\n### 2. Capacity Factor Tuning\n\n```python\n# Capacity = (tokens_per_batch \u002F num_experts) * capacity_factor\n\n# Training: Lower capacity (faster, drops some tokens)\ntrain_capacity_factor = 1.25  # 25% buffer\n\n# Evaluation: Higher capacity (no dropping)\neval_capacity_factor = 2.0    # 100% buffer\n\n# Formula:\nexpert_capacity = int((seq_len * batch_size \u002F num_experts) * capacity_factor)\n```\n\n### 3. Learning Rate Guidelines\n\n```python\n# MoE models need lower LR than dense models\n# - Dense model: lr = 6e-4\n# - MoE model: lr = 1e-4 (3-6× lower)\n\n# Also extend decay schedule\ndense_lr_decay_iters = 300000\nmoe_lr_decay_iters = 500000  # 1.5-2× longer\n```\n\n### 4. Loss Coefficient Tuning\n\n```python\n# Start with standard values\nmoe_loss_coeff = 0.01    # Auxiliary loss (load balancing)\nrouter_z_loss_coeff = 0.001  # Router entropy (stability)\n\n# If load imbalance persists, increase aux loss\nif max_expert_usage \u002F min_expert_usage > 2.0:\n    moe_loss_coeff = 0.1  # Stronger load balancing\n\n# If training unstable, increase z-loss\nif grad_norm > 10.0:\n    router_z_loss_coeff = 0.01\n```\n\n### 5. Avoid Common Pitfalls\n\n```python\n# ❌ Bad: Using same LR as dense model\noptimizer = Adam(model.parameters(), lr=6e-4)\n\n# ✅ Good: Lower LR for MoE\noptimizer = Adam([\n    {'params': model.non_moe_params, 'lr': 6e-4},\n    {'params': model.moe_params, 'lr': 1e-4}\n])\n\n# ❌ Bad: No load balancing\nloss = lm_loss\n\n# ✅ Good: Add auxiliary loss\nloss = lm_loss + 0.01 * aux_loss + 0.001 * z_loss\n\n# ❌ Bad: Too many experts for small dataset\nnum_experts = 128  # Overfitting risk\n\n# ✅ Good: Match experts to data diversity\nnum_experts = 8  # Better for small datasets\n```\n\n## Inference Optimization\n\n### Sparse Inference\n\n```python\n# Only activate top-k experts (huge memory savings)\n@torch.no_grad()\ndef moe_inference(x, model, top_k=2):\n    \"\"\"Sparse MoE inference: only load k experts.\"\"\"\n    # Router\n    gate_logits = model.gate(x)\n    topk_scores, topk_indices = torch.topk(\n        torch.softmax(gate_logits, dim=-1),\n        k=top_k,\n        dim=-1\n    )\n\n    # Load and run only top-k experts\n    output = torch.zeros_like(x)\n    for i in range(top_k):\n        expert_idx = topk_indices[:, i]\n        # Load expert from disk\u002Foffload if needed\n        expert = model.load_expert(expert_idx)\n        output += topk_scores[:, i:i+1] * expert(x)\n\n    return output\n```\n\n## Resources\n\n- **DeepSpeed MoE Tutorial**: https:\u002F\u002Fwww.deepspeed.ai\u002Ftutorials\u002Fmixture-of-experts-nlg\u002F\n- **Mixtral Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2401.04088\n- **Switch Transformers**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2101.03961\n- **HuggingFace MoE Guide**: https:\u002F\u002Fhuggingface.co\u002Fblog\u002Fmoe\n- **NVIDIA MoE Blog**: https:\u002F\u002Fdeveloper.nvidia.com\u002Fblog\u002Fapplying-mixture-of-experts-in-llm-architectures\u002F\n\n## See Also\n\n- `references\u002Farchitectures.md` - MoE model architectures (Mixtral, Switch, DeepSeek-V3)\n- `references\u002Ftraining.md` - Advanced training techniques and optimization\n- `references\u002Finference.md` - Production deployment and serving patterns\n\n\n","# MoE Training: Mixture of Experts\n\n## When to Use This Skill\n\nUse MoE Training when you need to:\n- **Train larger models** with limited compute (5× cost reduction vs dense models)\n- **Scale model capacity** without proportional compute increase\n- **Achieve better performance** per compute budget than dense models\n- **Specialize experts** for different domains\u002Ftasks\u002Flanguages\n- **Reduce inference latency** with sparse activation (only 13B\u002F47B params active in Mixtral)\n- **Implement SOTA models** like Mixtral 8x7B, DeepSeek-V3, Switch Transformers\n\n**Notable MoE Models**: Mixtral 8x7B (Mistral AI), DeepSeek-V3, Switch Transformers (Google), GLaM (Google), NLLB-MoE (Meta)\n\n## Installation\n\n```bash\n# DeepSpeed with MoE support\npip install deepspeed>=0.6.0\n\n# Megatron-DeepSpeed for large-scale training\ngit clone https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FMegatron-DeepSpeed\ncd Megatron-DeepSpeed\npip install -r requirements.txt\n\n# Alternative: HuggingFace Transformers\npip install transformers accelerate\n```\n\n## Quick Start\n\n### Basic MoE Architecture\n\n```python\nimport torch\nimport torch.nn as nn\n\nclass MoELayer(nn.Module):\n    \"\"\"Sparse Mixture of Experts layer.\"\"\"\n\n    def __init__(self, hidden_size, num_experts=8, top_k=2):\n        super().__init__()\n        self.num_experts = num_experts\n        self.top_k = top_k\n\n        # Expert networks (FFN)\n        self.experts = nn.ModuleList([\n            nn.Sequential(\n                nn.Linear(hidden_size, 4 * hidden_size),\n                nn.GELU(),\n                nn.Linear(4 * hidden_size, hidden_size)\n            )\n            for _ in range(num_experts)\n        ])\n\n        # Gating network (router)\n        self.gate = nn.Linear(hidden_size, num_experts)\n\n    def forward(self, x):\n        # x shape: (batch_size, seq_len, hidden_size)\n        batch_size, seq_len, hidden_size = x.shape\n\n        # Flatten for routing\n        x_flat = x.view(-1, hidden_size)  # (batch_size * seq_len, hidden_size)\n\n        # Compute gate scores\n        gate_logits = self.gate(x_flat)  # (batch_size * seq_len, num_experts)\n\n        # Top-k routing\n        gate_scores = torch.softmax(gate_logits, dim=-1)\n        topk_scores, topk_indices = torch.topk(gate_scores, self.top_k, dim=-1)\n\n        # Normalize top-k scores\n        topk_scores = topk_scores \u002F topk_scores.sum(dim=-1, keepdim=True)\n\n        # Dispatch and combine expert outputs\n        output = torch.zeros_like(x_flat)\n\n        for i in range(self.top_k):\n            expert_idx = topk_indices[:, i]\n            expert_scores = topk_scores[:, i].unsqueeze(-1)\n\n            # Route tokens to experts\n            for expert_id in range(self.num_experts):\n                mask = (expert_idx == expert_id)\n                if mask.any():\n                    expert_input = x_flat[mask]\n                    expert_output = self.experts[expert_id](expert_input)\n                    output[mask] += expert_scores[mask] * expert_output\n\n        # Reshape back\n        return output.view(batch_size, seq_len, hidden_size)\n```\n\n### DeepSpeed MoE Training\n\n```bash\n# Training script with MoE\ndeepspeed pretrain_gpt_moe.py \\\n  --num-layers 24 \\\n  --hidden-size 1024 \\\n  --num-attention-heads 16 \\\n  --seq-length 2048 \\\n  --max-position-embeddings 2048 \\\n  --micro-batch-size 4 \\\n  --global-batch-size 256 \\\n  --train-iters 500000 \\\n  --lr 0.0001 \\\n  --min-lr 0.00001 \\\n  --lr-decay-style cosine \\\n  --num-experts 128 \\\n  --moe-expert-parallel-size 4 \\\n  --moe-loss-coeff 0.01 \\\n  --moe-train-capacity-factor 1.25 \\\n  --moe-eval-capacity-factor 2.0 \\\n  --fp16 \\\n  --deepspeed_config ds_config.json\n```\n\n## Core Concepts\n\n### 1. MoE Architecture\n\n**Key Components:**\n- **Experts**: Multiple specialized FFN networks (typically 8-128)\n- **Router\u002FGate**: Learned network that selects which experts to use\n- **Top-k Routing**: Activate only k experts per token (k=1 or k=2)\n- **Load Balancing**: Ensure even expert utilization\n\n```\nInput Token\n    ↓\nRouter (Gate Network)\n    ↓\nTop-k Expert Selection (e.g., 2 out of 8)\n    ↓\nExpert 1 (weight: 0.6) + Expert 5 (weight: 0.4)\n    ↓\nWeighted Combination\n    ↓\nOutput\n```\n\n### 2. Routing Mechanisms\n\n**Top-1 Routing (Switch Transformer):**\n```python\n# Simplest routing: one expert per token\ngate_logits = router(x)  # (batch, seq_len, num_experts)\nexpert_idx = torch.argmax(gate_logits, dim=-1)  # Hard routing\n```\n\n**Top-2 Routing (Mixtral):**\n```python\n# Top-2: two experts per token\ngate_scores = torch.softmax(router(x), dim=-1)\ntop2_scores, top2_indices = torch.topk(gate_scores, k=2, dim=-1)\n\n# Normalize scores\ntop2_scores = top2_scores \u002F top2_scores.sum(dim=-1, keepdim=True)\n\n# Combine expert outputs\noutput = (top2_scores[:, :, 0:1] * expert_outputs[top2_indices[:, :, 0]] +\n          top2_scores[:, :, 1:2] * expert_outputs[top2_indices[:, :, 1]])\n```\n\n**Expert Choice Routing:**\n```python\n# Experts choose top-k tokens (instead of tokens choosing experts)\n# Guarantees perfect load balancing\nexpert_scores = router(x).transpose(-1, -2)  # (batch, num_experts, seq_len)\ntopk_tokens = torch.topk(expert_scores, k=capacity_per_expert, dim=-1)\n```\n\n### 3. Load Balancing\n\n**Auxiliary Loss:**\n```python\ndef load_balancing_loss(gate_logits, expert_indices, num_experts):\n    \"\"\"Encourage uniform expert usage.\"\"\"\n    # Fraction of tokens routed to each expert\n    expert_counts = torch.bincount(expert_indices.flatten(), minlength=num_experts)\n    expert_fraction = expert_counts.float() \u002F expert_indices.numel()\n\n    # Gate probability for each expert (average across tokens)\n    gate_probs = torch.softmax(gate_logits, dim=-1).mean(dim=0)\n\n    # Auxiliary loss: encourage alignment\n    aux_loss = num_experts * (expert_fraction * gate_probs).sum()\n\n    return aux_loss\n\n# Add to main loss\ntotal_loss = language_model_loss + 0.01 * load_balancing_loss(...)\n```\n\n**Router Z-Loss (Stability):**\n```python\ndef router_z_loss(logits):\n    \"\"\"Encourage router to have lower entropy (more decisive).\"\"\"\n    z_loss = torch.logsumexp(logits, dim=-1).pow(2).mean()\n    return z_loss\n\ntotal_loss = lm_loss + 0.01 * aux_loss + 0.001 * router_z_loss(gate_logits)\n```\n\n### 4. Expert Parallelism\n\n```python\n# DeepSpeed configuration\n{\n  \"train_batch_size\": 256,\n  \"fp16\": {\"enabled\": true},\n  \"moe\": {\n    \"enabled\": true,\n    \"num_experts\": 128,\n    \"expert_parallel_size\": 8,  # Distribute 128 experts across 8 GPUs\n    \"capacity_factor\": 1.25,    # Expert capacity = tokens_per_batch * capacity_factor \u002F num_experts\n    \"drop_tokens\": true,        # Drop tokens exceeding capacity\n    \"use_residual\": false\n  }\n}\n```\n\n## Training Configuration\n\n### DeepSpeed MoE Config\n\n```json\n{\n  \"train_batch_size\": 256,\n  \"gradient_accumulation_steps\": 1,\n  \"optimizer\": {\n    \"type\": \"Adam\",\n    \"params\": {\n      \"lr\": 0.0001,\n      \"betas\": [0.9, 0.999],\n      \"eps\": 1e-8\n    }\n  },\n  \"fp16\": {\n    \"enabled\": true,\n    \"loss_scale\": 0,\n    \"initial_scale_power\": 16\n  },\n  \"moe\": {\n    \"enabled\": true,\n    \"num_experts\": 128,\n    \"expert_parallel_size\": 8,\n    \"moe_loss_coeff\": 0.01,\n    \"train_capacity_factor\": 1.25,\n    \"eval_capacity_factor\": 2.0,\n    \"min_capacity\": 4,\n    \"drop_tokens\": true,\n    \"use_residual\": false,\n    \"use_tutel\": false\n  },\n  \"zero_optimization\": {\n    \"stage\": 1\n  }\n}\n```\n\n### Training Script\n\n```bash\n#!\u002Fbin\u002Fbash\n\n# Mixtral-style MoE training\ndeepspeed --num_gpus 8 pretrain_moe.py \\\n  --model-parallel-size 1 \\\n  --num-layers 32 \\\n  --hidden-size 4096 \\\n  --num-attention-heads 32 \\\n  --seq-length 2048 \\\n  --max-position-embeddings 4096 \\\n  --micro-batch-size 2 \\\n  --global-batch-size 256 \\\n  --train-iters 500000 \\\n  --save-interval 5000 \\\n  --eval-interval 1000 \\\n  --eval-iters 100 \\\n  --lr 0.0001 \\\n  --min-lr 0.00001 \\\n  --lr-decay-style cosine \\\n  --lr-warmup-iters 2000 \\\n  --clip-grad 1.0 \\\n  --weight-decay 0.1 \\\n  --num-experts 8 \\\n  --moe-expert-parallel-size 4 \\\n  --moe-loss-coeff 0.01 \\\n  --moe-train-capacity-factor 1.25 \\\n  --moe-eval-capacity-factor 2.0 \\\n  --disable-moe-token-dropping \\\n  --fp16 \\\n  --deepspeed \\\n  --deepspeed_config ds_config_moe.json \\\n  --data-path \u002Fpath\u002Fto\u002Fdata \\\n  --vocab-file \u002Fpath\u002Fto\u002Fvocab.json \\\n  --merge-file \u002Fpath\u002Fto\u002Fmerges.txt\n```\n\n## Advanced Patterns\n\n### Mixtral 8x7B Architecture\n\n```python\nclass MixtralMoEBlock(nn.Module):\n    \"\"\"Mixtral-style MoE block with 8 experts, top-2 routing.\"\"\"\n\n    def __init__(self, config):\n        super().__init__()\n        self.hidden_dim = config.hidden_size\n        self.ffn_dim = config.intermediate_size\n        self.num_experts = config.num_local_experts  # 8\n        self.top_k = config.num_experts_per_tok       # 2\n\n        # 8 expert FFNs\n        self.experts = nn.ModuleList([\n            nn.Sequential(\n                nn.Linear(self.hidden_dim, self.ffn_dim, bias=False),\n                nn.SiLU(),\n                nn.Linear(self.ffn_dim, self.hidden_dim, bias=False)\n            )\n            for _ in range(self.num_experts)\n        ])\n\n        # Router\n        self.gate = nn.Linear(self.hidden_dim, self.num_experts, bias=False)\n\n    def forward(self, hidden_states):\n        batch_size, sequence_length, hidden_dim = hidden_states.shape\n\n        # Flatten\n        hidden_states = hidden_states.view(-1, hidden_dim)\n\n        # Router logits\n        router_logits = self.gate(hidden_states)  # (batch * seq_len, num_experts)\n\n        # Softmax and top-2\n        routing_weights = torch.softmax(router_logits, dim=1)\n        routing_weights, selected_experts = torch.topk(routing_weights, self.top_k, dim=-1)\n\n        # Normalize routing weights\n        routing_weights \u002F= routing_weights.sum(dim=-1, keepdim=True)\n\n        # Initialize output\n        final_hidden_states = torch.zeros_like(hidden_states)\n\n        # Route to experts\n        for expert_idx in range(self.num_experts):\n            expert_layer = self.experts[expert_idx]\n            idx, top_x = torch.where(selected_experts == expert_idx)\n\n            if idx.shape[0] == 0:\n                continue\n\n            # Current expert tokens\n            current_hidden_states = hidden_states[idx]\n\n            # Expert forward\n            current_hidden_states = expert_layer(current_hidden_states)\n\n            # Weighted by routing scores\n            current_hidden_states *= routing_weights[idx, top_x, None]\n\n            # Accumulate\n            final_hidden_states.index_add_(0, idx, current_hidden_states)\n\n        # Reshape\n        return final_hidden_states.view(batch_size, sequence_length, hidden_dim)\n```\n\n### PR-MoE (Pyramid-Residual-MoE)\n\n```bash\n# DeepSpeed PR-MoE: 3x better parameter efficiency\ndeepspeed pretrain_gpt_moe.py \\\n  --num-layers 24 \\\n  --hidden-size 1024 \\\n  --num-attention-heads 16 \\\n  --num-experts \"[128, 64, 32, 16]\" \\\n  --mlp-type residual \\\n  --moe-expert-parallel-size 4 \\\n  --moe-loss-coeff 0.01 \\\n  --fp16\n```\n\n## Best Practices\n\n### 1. Expert Count Selection\n\n```python\n# Rule of thumb: More experts = more capacity, but diminishing returns\n# Typical configurations:\n# - Small models (1B-7B): 8-16 experts\n# - Medium models (7B-30B): 8-64 experts\n# - Large models (30B+): 64-256 experts\n\n# Example: Mixtral 8x7B\n# Total params: 47B (8 experts × 7B each)\n# Active params: 13B (2 experts × 7B, top-2 routing)\n# Efficiency: 47B capacity with 13B compute\n```\n\n### 2. Capacity Factor Tuning\n\n```python\n# Capacity = (tokens_per_batch \u002F num_experts) * capacity_factor\n\n# Training: Lower capacity (faster, drops some tokens)\ntrain_capacity_factor = 1.25  # 25% buffer\n\n# Evaluation: Higher capacity (no dropping)\neval_capacity_factor = 2.0    # 100% buffer\n\n# Formula:\nexpert_capacity = int((seq_len * batch_size \u002F num_experts) * capacity_factor)\n```\n\n### 3. Learning Rate Guidelines\n\n```python\n# MoE models need lower LR than dense models\n# - Dense model: lr = 6e-4\n# - MoE model: lr = 1e-4 (3-6× lower)\n\n# Also extend decay schedule\ndense_lr_decay_iters = 300000\nmoe_lr_decay_iters = 500000  # 1.5-2× longer\n```\n\n### 4. Loss Coefficient Tuning\n\n```python\n# Start with standard values\nmoe_loss_coeff = 0.01    # Auxiliary loss (load balancing)\nrouter_z_loss_coeff = 0.001  # Router entropy (stability)\n\n# If load imbalance persists, increase aux loss\nif max_expert_usage \u002F min_expert_usage > 2.0:\n    moe_loss_coeff = 0.1  # Stronger load balancing\n\n# If training unstable, increase z-loss\nif grad_norm > 10.0:\n    router_z_loss_coeff = 0.01\n```\n\n### 5. Avoid Common Pitfalls\n\n```python\n# ❌ Bad: Using same LR as dense model\noptimizer = Adam(model.parameters(), lr=6e-4)\n\n# ✅ Good: Lower LR for MoE\noptimizer = Adam([\n    {'params': model.non_moe_params, 'lr': 6e-4},\n    {'params': model.moe_params, 'lr': 1e-4}\n])\n\n# ❌ Bad: No load balancing\nloss = lm_loss\n\n# ✅ Good: Add auxiliary loss\nloss = lm_loss + 0.01 * aux_loss + 0.001 * z_loss\n\n# ❌ Bad: Too many experts for small dataset\nnum_experts = 128  # Overfitting risk\n\n# ✅ Good: Match experts to data diversity\nnum_experts = 8  # Better for small datasets\n```\n\n## Inference Optimization\n\n### Sparse Inference\n\n```python\n# Only activate top-k experts (huge memory savings)\n@torch.no_grad()\ndef moe_inference(x, model, top_k=2):\n    \"\"\"Sparse MoE inference: only load k experts.\"\"\"\n    # Router\n    gate_logits = model.gate(x)\n    topk_scores, topk_indices = torch.topk(\n        torch.softmax(gate_logits, dim=-1),\n        k=top_k,\n        dim=-1\n    )\n\n    # Load and run only top-k experts\n    output = torch.zeros_like(x)\n    for i in range(top_k):\n        expert_idx = topk_indices[:, i]\n        # Load expert from disk\u002Foffload if needed\n        expert = model.load_expert(expert_idx)\n        output += topk_scores[:, i:i+1] * expert(x)\n\n    return output\n```\n\n## Resources\n\n- **DeepSpeed MoE Tutorial**: https:\u002F\u002Fwww.deepspeed.ai\u002Ftutorials\u002Fmixture-of-experts-nlg\u002F\n- **Mixtral Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2401.04088\n- **Switch Transformers**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2101.03961\n- **HuggingFace MoE Guide**: https:\u002F\u002Fhuggingface.co\u002Fblog\u002Fmoe\n- **NVIDIA MoE Blog**: https:\u002F\u002Fdeveloper.nvidia.com\u002Fblog\u002Fapplying-mixture-of-experts-in-llm-architectures\u002F\n\n## See Also\n\n- `references\u002Farchitectures.md` - MoE model architectures (Mixtral, Switch, DeepSeek-V3)\n- `references\u002Ftraining.md` - Advanced training techniques and optimization\n- `references\u002Finference.md` - Production deployment and serving patterns\n\n\n",{"name":5,"tags":44,"author":49,"license":14,"version":50,"description":7,"dependencies":51},[21,45,30,29,38,28,27,46,47,48,26],"MoE","DeepSeek","Routing","Load Balancing","Orchestra Research","1.0.0",[52,53,54,55],"deepspeed","transformers","torch","accelerate","main","f19b254e8975d0bfddb394686706106930d7964d","AI-Training\u002Fmoe-training","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus","https:\u002F\u002Fgithub.com\u002Fendman100","2026-11-11 18:19:10",{"source":8,"skillName":5,"commands":63,"prompt":73,"zipAvailable":74},[64,67,70],{"tool":65,"command":66},"npx","npx skills add endman100\u002Fskill-router-nexus --skill moe-training",{"tool":68,"command":69},"bunx","bunx skills add endman100\u002Fskill-router-nexus --skill moe-training",{"tool":71,"command":72},"pnpm","pnpm dlx skills add endman100\u002Fskill-router-nexus --skill moe-training","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Training\u002Fmoe-training\u002FSKILL.md",true,[76,82,86],{"path":77,"resourceType":78,"resourceTypeDesc":79,"sizeBytes":80,"sourceHtmlUrl":81},"references\u002Farchitectures.md","REFERENCE","参考文档",14044,"https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Training\u002Fmoe-training\u002Freferences\u002Farchitectures.md",{"path":83,"resourceType":78,"resourceTypeDesc":79,"sizeBytes":84,"sourceHtmlUrl":85},"references\u002Finference.md",8239,"https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Training\u002Fmoe-training\u002Freferences\u002Finference.md",{"path":87,"resourceType":78,"resourceTypeDesc":79,"sizeBytes":88,"sourceHtmlUrl":89},"references\u002Ftraining.md",9707,"https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Training\u002Fmoe-training\u002Freferences\u002Ftraining.md",[91,115,153,175,184,208],{"skillNo":92,"name":93,"nameZh":6,"description":94,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":95,"sourceHtmlUrl":96,"categories":97,"keywords":103,"publishedAt":39},"01M6MPCT5BXVER8HQZTW3QE9AS","create-master","基于佛教经典文献，生成特定高僧大德的 AI 教学角色","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FMetaphysics-and-Spirituality\u002FMaster-skill\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FMetaphysics-and-Spirituality\u002FMaster-skill\u002FSKILL.md",[98],{"categoryNo":99,"categoryName":100,"taxonomyType":101,"taxonomyTypeDesc":102},"01JBSEEDCAT000000000000039","教师与教育","PROFESSION","适用职业",[104,105,106,107,108,109,110,111,112,113,114],"create master","master","create","生成","true","webfetch","edit","write","read","grep","Bash",{"skillNo":116,"name":117,"nameZh":6,"description":118,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":119,"sourceHtmlUrl":120,"categories":121,"keywords":133,"publishedAt":39},"01M6M43AKE95Y64E1JRXEF5QQ3","writing-dna-skill","从至少 20 篇完整文章中蒸馏可复用的写作 DNA，分析语言、文章结构、选题逻辑、素材策略、认知框架和视觉风格，并生成 Writing-DNA.md；按该 DNA 写作时会读取全部蒸馏产物和 5 篇相关原文。用于中英文作者、账号、品牌或出版物的风格分析与一致性写作。 Distill reusable Writing DNA from at least 20 complete articles for Chinese or English authors, publications, brands, and accounts; use for language, structure, topic logic, source strategy, cognitive-frame, and visual-style analysis.","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fwriting-dna-skill\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fwriting-dna-skill\u002FSKILL.md",[122,127,130],{"categoryNo":123,"categoryName":124,"taxonomyType":125,"taxonomyTypeDesc":126},"01JBSEEDCAT000000000000018","写作与文案","SCENARIO","使用场景",{"categoryNo":128,"categoryName":129,"taxonomyType":125,"taxonomyTypeDesc":126},"01JBSEEDCAT000000000000023","Skill与智能体开发",{"categoryNo":131,"categoryName":132,"taxonomyType":101,"taxonomyTypeDesc":102},"01JBSEEDCAT000000000000036","内容创作者",[134,135,136,137,138,139,140,141,142,143,144,145,146,107,147,148,149,150,151,152],"writing dna skill","DNA","writing","strategy","english","logic","visual","structure","chinese","reusable","use","style","distill","analysis","frame","source","for","at","complete",{"skillNo":154,"name":155,"nameZh":6,"description":156,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":157,"sourceHtmlUrl":158,"categories":159,"keywords":162,"publishedAt":39},"01M6M428TTCGQ4ME3KY99VN037","stop-slop","Remove AI writing patterns from prose. Use when drafting, editing, or reviewing text to eliminate predictable AI tells.","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fstop-slop\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fstop-slop\u002FSKILL.md",[160,161],{"categoryNo":123,"categoryName":124,"taxonomyType":125,"taxonomyTypeDesc":126},{"categoryNo":131,"categoryName":132,"taxonomyType":101,"taxonomyTypeDesc":102},[163,164,165,166,22,167,168,169,170,144,171,172,173,150,174,136],"slop","stop slop","stop","drafting","drafts","https","patterns","prose","remove","editing","text","content",{"skillNo":176,"name":177,"nameZh":6,"description":178,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":179,"sourceHtmlUrl":180,"categories":181,"keywords":182,"publishedAt":39},"01M6M41X0B7EBX42TK435DMGGR","shuorenhua","检查和清理中英文文本里的 AI 套路，适用于“去 AI 味”“说人话”“自然一点”“别像模板”“先标问题”这类改写和审稿需求；按场景控制力度，同时保留事实、术语、语域和责任主体。","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fshuorenhua\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fshuorenhua\u002FSKILL.md",[],[177,183],"文本",{"skillNo":185,"name":186,"nameZh":6,"description":187,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":188,"sourceHtmlUrl":189,"categories":190,"keywords":191,"publishedAt":39},"01M6M3ZD12WBEW5KXKDM4N6DFG","naming","Name products, SaaS, brands, open source projects, bots, and apps. Use when the user needs to name something, find a brand name, or pick a product name. Metaphor-driven process that produces memorable, meaningful names and avoids AI slop.","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fnaming-main\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fnaming-main\u002FSKILL.md",[],[186,192,193,194,195,196,144,197,112,149,198,199,200,201,114,202,203,204,205,22,206,207,163],"repo","websearch","product name","apps","bots","view","brand","pick","gh","projects","open","saas","open source","product","describe","ai slop",{"skillNo":209,"name":210,"nameZh":6,"description":211,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":212,"sourceHtmlUrl":213,"categories":214,"keywords":216,"publishedAt":39},"01M6M3YVXAFAAWJ7F6JMPFREYE","humanizer-zh","去除文本中的 AI 生成痕迹。适用于编辑或审阅文本，使其听起来更自然、更像人类书写。\n基于维基百科的\"AI 写作特征\"综合指南。检测并修复以下模式：夸大的象征意义、\n宣传性语言、以 -ing 结尾的肤浅分析、模糊的归因、破折号过度使用、三段式法则、\nAI 词汇、否定式排比、过多的连接性短语。","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fhumanizer-zh\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fhumanizer-zh\u002FSKILL.md",[215],{"categoryNo":123,"categoryName":124,"taxonomyType":125,"taxonomyTypeDesc":126},[217,218,219,112,111,110,136,107,183,220,163,165],"humanizer","zh","humanizer zh","翻译"]