[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6KQJXY9HPMWRB7PXBEF29BC":3,"skill-same:01M6KQJXY9HPMWRB7PXBEF29BC":95},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":57,"commitSha":58,"skillDirectory":59,"forksCount":13,"repositoryHtmlUrl":60,"repositoryHomepageUrl":6,"ownerHtmlUrl":61,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":62,"install":63,"resources":76},"01M6KQJXY9HPMWRB7PXBEF29BC","training-llms-megatron",null,"Trains large language models (2B-462B parameters) using NVIDIA Megatron-Core with advanced parallelism strategies. Use when training models >1B parameters, need maximum GPU efficiency (47% MFU on H100), or require tensor\u002Fpipeline\u002Fsequence\u002Fcontext\u002Fexpert parallelism. Production-ready framework used for Nemotron, LLaMA, DeepSeek.","endman100\u002Fskill-router-nexus","endman100","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F31997019?v=4","01M6KPDX17BYDWW4RRB0TNGGMX","01M6KPDX64Y2QBEWVFG5W5PWPF",0,"MIT","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Training\u002Fmegatron-core\u002Fmegatron-core\u002FSKILL.md",[],[18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"training","llms","training llms megatron","megatron","when","sequence","production-ready","advanced","Expert Parallelism","Model Parallelism","Megatron-Core","H100","apex","strategies","efficiency","expert","ready","context","model","pipeline","framework","2026-11-10 10:12:30",false,"---\nname: training-llms-megatron\ndescription: Trains large language models (2B-462B parameters) using NVIDIA Megatron-Core with advanced parallelism strategies. Use when training models >1B parameters, need maximum GPU efficiency (47% MFU on H100), or require tensor\u002Fpipeline\u002Fsequence\u002Fcontext\u002Fexpert parallelism. Production-ready framework used for Nemotron, LLaMA, DeepSeek.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Megatron-Core, Large-Scale Training, NVIDIA, Tensor Parallelism, Pipeline Parallelism, Model Parallelism, H100, Distributed Training, Production]\ndependencies: [megatron-core, torch, apex, transformer-engine]\n---\n\n# Megatron-Core - Large-Scale LLM Training\n\n## Quick start\n\nMegatron-Core trains LLMs from 2B to 462B parameters with up to 47% Model FLOP Utilization on H100 GPUs through advanced parallelism strategies.\n\n**Installation**:\n```bash\n# Docker (recommended)\ndocker run --gpus all -it --rm nvcr.io\u002Fnvidia\u002Fpytorch:25.04-py3\n\n# Or pip\npip install megatron-core\n```\n\n**Simple distributed training**:\n```bash\n# Train with 2 GPUs using data parallelism\ntorchrun --nproc_per_node=2 examples\u002Frun_simple_mcore_train_loop.py\n\n# Or LLaMA-3 8B training\n.\u002Fexamples\u002Fllama\u002Ftrain_llama3_8b_fp8.sh\n```\n\n## Common workflows\n\n### Workflow 1: Train LLaMA-style model with 3D parallelism\n\nCopy this checklist:\n\n```\nLLaMA Training Setup:\n- [ ] Step 1: Choose parallelism configuration\n- [ ] Step 2: Configure training hyperparameters\n- [ ] Step 3: Launch distributed training\n- [ ] Step 4: Monitor performance metrics\n```\n\n**Step 1: Choose parallelism configuration**\n\nModel size determines parallelism strategy:\n\n| Model Size | GPUs | Tensor Parallel | Pipeline Parallel | Data Parallel | Context Parallel |\n|------------|------|-----------------|-------------------|---------------|------------------|\n| 7B | 8 | 1 | 1 | 8 | 1 |\n| 13B | 8 | 2 | 1 | 4 | 1 |\n| 70B | 64 | 4 | 4 | 4 | 1 |\n| 405B | 128 | 8 | 8 | 2 | 2 |\n\n**Step 2: Configure training hyperparameters**\n\n```bash\n#!\u002Fbin\u002Fbash\n# train_llama_70b.sh\n\nGPUS_PER_NODE=8\nNNODES=8  # 64 GPUs total\nTP=4      # Tensor parallel\nPP=4      # Pipeline parallel\nCP=1      # Context parallel\n\n# LLaMA 70B configuration\nMODEL_SIZE=70  # Billion parameters\nHIDDEN_SIZE=8192\nNUM_LAYERS=80\nNUM_HEADS=64\nSEQ_LENGTH=4096\n\n# Training hyperparameters\nMICRO_BATCH=1\nGLOBAL_BATCH=1024\nLR=3e-4\n\ntorchrun \\\n  --nproc_per_node=$GPUS_PER_NODE \\\n  --nnodes=$NNODES \\\n  pretrain_gpt.py \\\n  --tensor-model-parallel-size $TP \\\n  --pipeline-model-parallel-size $PP \\\n  --context-parallel-size $CP \\\n  --sequence-parallel \\\n  --num-layers $NUM_LAYERS \\\n  --hidden-size $HIDDEN_SIZE \\\n  --num-attention-heads $NUM_HEADS \\\n  --seq-length $SEQ_LENGTH \\\n  --max-position-embeddings $SEQ_LENGTH \\\n  --micro-batch-size $MICRO_BATCH \\\n  --global-batch-size $GLOBAL_BATCH \\\n  --lr $LR \\\n  --train-iters 100000 \\\n  --lr-decay-style cosine \\\n  --lr-warmup-iters 2000 \\\n  --weight-decay 0.1 \\\n  --clip-grad 1.0 \\\n  --bf16 \\\n  --use-mcore-models \\\n  --transformer-impl transformer_engine \\\n  --data-path \u002Fpath\u002Fto\u002Fdata \\\n  --vocab-file \u002Fpath\u002Fto\u002Fvocab.json \\\n  --merge-file \u002Fpath\u002Fto\u002Fmerges.txt\n```\n\n**Step 3: Launch distributed training**\n\n```bash\n# Single node (8 GPUs)\nbash train_llama_70b.sh\n\n# Multi-node with SLURM\nsbatch --nodes=8 --gpus-per-node=8 train_llama_70b.sh\n```\n\n**Step 4: Monitor performance metrics**\n\nKey metrics to track:\n```\nModel FLOP Utilization (MFU): Target >40% on H100\nThroughput: Tokens\u002Fsec\u002FGPU\nMemory usage: \u003C80GB per GPU for 70B model\nLoss: Should decrease steadily\n```\n\n### Workflow 2: Configure Mixture of Experts (MoE) training\n\nFor sparse MoE models like Mixtral.\n\n```\nMoE Training:\n- [ ] Step 1: Configure expert parallelism\n- [ ] Step 2: Set MoE hyperparameters\n- [ ] Step 3: Launch training with EP\n```\n\n**Step 1: Configure expert parallelism**\n\n```bash\n# Mixtral 8x7B example\nTENSOR_PARALLEL=2\nPIPELINE_PARALLEL=1\nEXPERT_PARALLEL=4  # Split 8 experts across 4 GPUs\nDATA_PARALLEL=4\n\nTOTAL_GPUS=$((TENSOR_PARALLEL * PIPELINE_PARALLEL * EXPERT_PARALLEL * DATA_PARALLEL))\n# = 2 * 1 * 4 * 4 = 32 GPUs\n```\n\n**Step 2: Set MoE hyperparameters**\n\n```bash\ntorchrun \\\n  --nproc_per_node=8 \\\n  pretrain_gpt.py \\\n  --tensor-model-parallel-size 2 \\\n  --pipeline-model-parallel-size 1 \\\n  --expert-model-parallel-size 4 \\\n  --num-experts 8 \\\n  --moe-router-topk 2 \\\n  --moe-router-load-balancing-type aux_loss \\\n  --moe-aux-loss-coeff 0.01 \\\n  --hidden-size 4096 \\\n  --num-layers 32 \\\n  --num-attention-heads 32 \\\n  --seq-length 4096 \\\n  --max-position-embeddings 4096 \\\n  --bf16 \\\n  --use-mcore-models \\\n  --transformer-impl transformer_engine \\\n  --data-path \u002Fpath\u002Fto\u002Fdata \\\n  --vocab-file \u002Fpath\u002Fto\u002Fvocab.json \\\n  --merge-file \u002Fpath\u002Fto\u002Fmerges.txt\n```\n\n**Step 3: Launch training with EP**\n\nExpert parallelism distributes different experts across GPUs, reducing memory while maintaining capacity.\n\n```\nMemory without EP: 8 experts × 7B = 56GB per GPU\nMemory with EP=4: 2 experts × 7B = 14GB per GPU\nSavings: 75% memory reduction\n```\n\n### Workflow 3: Optimize for maximum throughput\n\nAchieve 47% MFU on H100.\n\n```\nPerformance Optimization:\n- [ ] Step 1: Enable Flash Attention\n- [ ] Step 2: Use FP8 precision (H100)\n- [ ] Step 3: Optimize micro-batch size\n- [ ] Step 4: Tune parallelism degrees\n```\n\n**Step 1: Enable optimizations**\n\n```bash\n--use-mcore-models  # Use Megatron Core models\n--transformer-impl transformer_engine  # Use Transformer Engine\n--sequence-parallel  # Reduce activation memory (use with TP)\n```\n\n**Step 2: Use FP8 precision (H100 only)**\n\n```bash\n--fp8-hybrid  # FP8 mixed precision training\n# Transformer Engine handles FP8 automatically\n```\n\nResult: 1.5-2x speedup on H100 vs BF16.\n\n**Step 3: Optimize micro-batch size**\n\nFind largest micro-batch that fits in memory:\n\n```bash\n# Start with 1, increase until OOM\nfor MBS in 1 2 4 8; do\n  echo \"Testing micro-batch-size=$MBS\"\n  torchrun ... --micro-batch-size $MBS\ndone\n```\n\nTypical values:\n- 7B model: 4-8\n- 70B model: 1-2\n- 405B model: 1\n\n**Step 4: Tune parallelism degrees**\n\nRules of thumb:\n```\nTensor Parallel: Use ≤8 (limited by NVLink within node)\nPipeline Parallel: Use for >70B models\nContext Parallel: Use for sequences >8K tokens\nData Parallel: Fill remaining GPUs\n```\n\nExample 405B on 128 H100s:\n```\nTP=8 (1 node)\nPP=8 (across nodes)\nCP=2 (long sequences)\nDP=1\nTotal = 8 × 8 × 2 × 1 = 128 GPUs\n```\n\n## When to use vs alternatives\n\n**Use Megatron-Core when:**\n- Training models >10B parameters\n- Need maximum efficiency (target >40% MFU)\n- Using NVIDIA GPUs (A100, H100)\n- Production training at scale\n- Want fine-grained parallelism control\n\n**Use alternatives instead:**\n- **PyTorch FSDP**: Models \u003C70B, simpler API, PyTorch native\n- **DeepSpeed**: Easier setup, good for \u003C100B models\n- **HuggingFace Accelerate**: Prototyping, simpler workflows\n- **LitGPT**: Educational, single-file implementations\n\n## Common issues\n\n**Issue: Low GPU utilization (\u003C30% MFU)**\n\nCauses:\n1. Micro-batch too small\n2. Too much parallelism overhead\n3. Not using Flash Attention\n\nFixes:\n```bash\n# Increase micro-batch\n--micro-batch-size 4  # Was 1\n\n# Enable optimizations\n--use-flash-attn\n--sequence-parallel\n\n# Reduce TP if >8\n--tensor-model-parallel-size 4  # Was 16\n```\n\n**Issue: Out of memory**\n\nReduce memory with:\n```bash\n--tensor-model-parallel-size 2  # Split model across GPUs\n--recompute-granularity full  # Gradient checkpointing\n--recompute-method block  # Checkpoint transformer blocks\n--recompute-num-layers 1  # Checkpoint every layer\n```\n\nOr use CPU\u002FNVMe offloading:\n```bash\n--cpu-optimizer  # Offload optimizer to CPU\n--cpu-optimizer-type ADAM  # CPU Adam variant\n```\n\n**Issue: Training slower than expected**\n\nCheck:\n1. **Network bottleneck**: Ensure InfiniBand\u002FNVLink enabled\n2. **Pipeline bubbles**: Use interleaved pipeline schedule\n   ```bash\n   --num-layers-per-virtual-pipeline-stage 2\n   ```\n3. **Data loading**: Use fast data loader\n   ```bash\n   --dataloader-type cyclic\n   ```\n\n**Issue: Diverging loss**\n\nStabilize training:\n```bash\n--lr-warmup-iters 2000  # Longer warmup\n--clip-grad 1.0  # Gradient clipping\n--init-method-std 0.006  # Smaller init\n--attention-dropout 0.0  # No dropout in attention\n--hidden-dropout 0.0  # No dropout in FFN\n```\n\n## Advanced topics\n\n**Parallelism strategies**: See [references\u002Fparallelism-guide.md](references\u002Fparallelism-guide.md) for detailed comparison of TP\u002FPP\u002FDP\u002FCP\u002FEP with performance analysis and when to use each.\n\n**Performance benchmarks**: See [references\u002Fbenchmarks.md](references\u002Fbenchmarks.md) for MFU numbers across different model sizes and GPU configurations.\n\n**Production configurations**: See [references\u002Fproduction-examples.md](references\u002Fproduction-examples.md) for real-world setups from LLaMA 3 405B, Nemotron-4 340B, and DeepSeek-V3 671B.\n\n**Training recipes**: See [references\u002Ftraining-recipes.md](references\u002Ftraining-recipes.md) for complete hyperparameter configurations for GPT\u002FLLaMA\u002FMixtral architectures.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA Ampere+ (A100, H100, B200)\n  - Turing works but slower\n  - FP8 requires Hopper\u002FAda\u002FBlackwell\n- **Network**: InfiniBand or 400Gb+ Ethernet for multi-node\n- **Memory per GPU**:\n  - 7B model: 40GB+\n  - 70B model: 80GB (with TP=4)\n  - 405B model: 80GB (with TP=8, PP=8)\n- **Storage**: Fast NVMe for checkpoints (1TB+ for 70B+ models)\n\n## Resources\n\n- Docs: https:\u002F\u002Fdocs.nvidia.com\u002Fmegatron-core\u002F\n- GitHub: https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM\n- Papers:\n  - \"Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism\" (2019)\n  - \"Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM\" (2021)\n- NeMo Framework: https:\u002F\u002Fdocs.nvidia.com\u002Fnemo-framework\u002F (built on Megatron-Core)\n\n\n","# Megatron-Core - Large-Scale LLM Training\n\n## Quick start\n\nMegatron-Core trains LLMs from 2B to 462B parameters with up to 47% Model FLOP Utilization on H100 GPUs through advanced parallelism strategies.\n\n**Installation**:\n```bash\n# Docker (recommended)\ndocker run --gpus all -it --rm nvcr.io\u002Fnvidia\u002Fpytorch:25.04-py3\n\n# Or pip\npip install megatron-core\n```\n\n**Simple distributed training**:\n```bash\n# Train with 2 GPUs using data parallelism\ntorchrun --nproc_per_node=2 examples\u002Frun_simple_mcore_train_loop.py\n\n# Or LLaMA-3 8B training\n.\u002Fexamples\u002Fllama\u002Ftrain_llama3_8b_fp8.sh\n```\n\n## Common workflows\n\n### Workflow 1: Train LLaMA-style model with 3D parallelism\n\nCopy this checklist:\n\n```\nLLaMA Training Setup:\n- [ ] Step 1: Choose parallelism configuration\n- [ ] Step 2: Configure training hyperparameters\n- [ ] Step 3: Launch distributed training\n- [ ] Step 4: Monitor performance metrics\n```\n\n**Step 1: Choose parallelism configuration**\n\nModel size determines parallelism strategy:\n\n| Model Size | GPUs | Tensor Parallel | Pipeline Parallel | Data Parallel | Context Parallel |\n|------------|------|-----------------|-------------------|---------------|------------------|\n| 7B | 8 | 1 | 1 | 8 | 1 |\n| 13B | 8 | 2 | 1 | 4 | 1 |\n| 70B | 64 | 4 | 4 | 4 | 1 |\n| 405B | 128 | 8 | 8 | 2 | 2 |\n\n**Step 2: Configure training hyperparameters**\n\n```bash\n#!\u002Fbin\u002Fbash\n# train_llama_70b.sh\n\nGPUS_PER_NODE=8\nNNODES=8  # 64 GPUs total\nTP=4      # Tensor parallel\nPP=4      # Pipeline parallel\nCP=1      # Context parallel\n\n# LLaMA 70B configuration\nMODEL_SIZE=70  # Billion parameters\nHIDDEN_SIZE=8192\nNUM_LAYERS=80\nNUM_HEADS=64\nSEQ_LENGTH=4096\n\n# Training hyperparameters\nMICRO_BATCH=1\nGLOBAL_BATCH=1024\nLR=3e-4\n\ntorchrun \\\n  --nproc_per_node=$GPUS_PER_NODE \\\n  --nnodes=$NNODES \\\n  pretrain_gpt.py \\\n  --tensor-model-parallel-size $TP \\\n  --pipeline-model-parallel-size $PP \\\n  --context-parallel-size $CP \\\n  --sequence-parallel \\\n  --num-layers $NUM_LAYERS \\\n  --hidden-size $HIDDEN_SIZE \\\n  --num-attention-heads $NUM_HEADS \\\n  --seq-length $SEQ_LENGTH \\\n  --max-position-embeddings $SEQ_LENGTH \\\n  --micro-batch-size $MICRO_BATCH \\\n  --global-batch-size $GLOBAL_BATCH \\\n  --lr $LR \\\n  --train-iters 100000 \\\n  --lr-decay-style cosine \\\n  --lr-warmup-iters 2000 \\\n  --weight-decay 0.1 \\\n  --clip-grad 1.0 \\\n  --bf16 \\\n  --use-mcore-models \\\n  --transformer-impl transformer_engine \\\n  --data-path \u002Fpath\u002Fto\u002Fdata \\\n  --vocab-file \u002Fpath\u002Fto\u002Fvocab.json \\\n  --merge-file \u002Fpath\u002Fto\u002Fmerges.txt\n```\n\n**Step 3: Launch distributed training**\n\n```bash\n# Single node (8 GPUs)\nbash train_llama_70b.sh\n\n# Multi-node with SLURM\nsbatch --nodes=8 --gpus-per-node=8 train_llama_70b.sh\n```\n\n**Step 4: Monitor performance metrics**\n\nKey metrics to track:\n```\nModel FLOP Utilization (MFU): Target >40% on H100\nThroughput: Tokens\u002Fsec\u002FGPU\nMemory usage: \u003C80GB per GPU for 70B model\nLoss: Should decrease steadily\n```\n\n### Workflow 2: Configure Mixture of Experts (MoE) training\n\nFor sparse MoE models like Mixtral.\n\n```\nMoE Training:\n- [ ] Step 1: Configure expert parallelism\n- [ ] Step 2: Set MoE hyperparameters\n- [ ] Step 3: Launch training with EP\n```\n\n**Step 1: Configure expert parallelism**\n\n```bash\n# Mixtral 8x7B example\nTENSOR_PARALLEL=2\nPIPELINE_PARALLEL=1\nEXPERT_PARALLEL=4  # Split 8 experts across 4 GPUs\nDATA_PARALLEL=4\n\nTOTAL_GPUS=$((TENSOR_PARALLEL * PIPELINE_PARALLEL * EXPERT_PARALLEL * DATA_PARALLEL))\n# = 2 * 1 * 4 * 4 = 32 GPUs\n```\n\n**Step 2: Set MoE hyperparameters**\n\n```bash\ntorchrun \\\n  --nproc_per_node=8 \\\n  pretrain_gpt.py \\\n  --tensor-model-parallel-size 2 \\\n  --pipeline-model-parallel-size 1 \\\n  --expert-model-parallel-size 4 \\\n  --num-experts 8 \\\n  --moe-router-topk 2 \\\n  --moe-router-load-balancing-type aux_loss \\\n  --moe-aux-loss-coeff 0.01 \\\n  --hidden-size 4096 \\\n  --num-layers 32 \\\n  --num-attention-heads 32 \\\n  --seq-length 4096 \\\n  --max-position-embeddings 4096 \\\n  --bf16 \\\n  --use-mcore-models \\\n  --transformer-impl transformer_engine \\\n  --data-path \u002Fpath\u002Fto\u002Fdata \\\n  --vocab-file \u002Fpath\u002Fto\u002Fvocab.json \\\n  --merge-file \u002Fpath\u002Fto\u002Fmerges.txt\n```\n\n**Step 3: Launch training with EP**\n\nExpert parallelism distributes different experts across GPUs, reducing memory while maintaining capacity.\n\n```\nMemory without EP: 8 experts × 7B = 56GB per GPU\nMemory with EP=4: 2 experts × 7B = 14GB per GPU\nSavings: 75% memory reduction\n```\n\n### Workflow 3: Optimize for maximum throughput\n\nAchieve 47% MFU on H100.\n\n```\nPerformance Optimization:\n- [ ] Step 1: Enable Flash Attention\n- [ ] Step 2: Use FP8 precision (H100)\n- [ ] Step 3: Optimize micro-batch size\n- [ ] Step 4: Tune parallelism degrees\n```\n\n**Step 1: Enable optimizations**\n\n```bash\n--use-mcore-models  # Use Megatron Core models\n--transformer-impl transformer_engine  # Use Transformer Engine\n--sequence-parallel  # Reduce activation memory (use with TP)\n```\n\n**Step 2: Use FP8 precision (H100 only)**\n\n```bash\n--fp8-hybrid  # FP8 mixed precision training\n# Transformer Engine handles FP8 automatically\n```\n\nResult: 1.5-2x speedup on H100 vs BF16.\n\n**Step 3: Optimize micro-batch size**\n\nFind largest micro-batch that fits in memory:\n\n```bash\n# Start with 1, increase until OOM\nfor MBS in 1 2 4 8; do\n  echo \"Testing micro-batch-size=$MBS\"\n  torchrun ... --micro-batch-size $MBS\ndone\n```\n\nTypical values:\n- 7B model: 4-8\n- 70B model: 1-2\n- 405B model: 1\n\n**Step 4: Tune parallelism degrees**\n\nRules of thumb:\n```\nTensor Parallel: Use ≤8 (limited by NVLink within node)\nPipeline Parallel: Use for >70B models\nContext Parallel: Use for sequences >8K tokens\nData Parallel: Fill remaining GPUs\n```\n\nExample 405B on 128 H100s:\n```\nTP=8 (1 node)\nPP=8 (across nodes)\nCP=2 (long sequences)\nDP=1\nTotal = 8 × 8 × 2 × 1 = 128 GPUs\n```\n\n## When to use vs alternatives\n\n**Use Megatron-Core when:**\n- Training models >10B parameters\n- Need maximum efficiency (target >40% MFU)\n- Using NVIDIA GPUs (A100, H100)\n- Production training at scale\n- Want fine-grained parallelism control\n\n**Use alternatives instead:**\n- **PyTorch FSDP**: Models \u003C70B, simpler API, PyTorch native\n- **DeepSpeed**: Easier setup, good for \u003C100B models\n- **HuggingFace Accelerate**: Prototyping, simpler workflows\n- **LitGPT**: Educational, single-file implementations\n\n## Common issues\n\n**Issue: Low GPU utilization (\u003C30% MFU)**\n\nCauses:\n1. Micro-batch too small\n2. Too much parallelism overhead\n3. Not using Flash Attention\n\nFixes:\n```bash\n# Increase micro-batch\n--micro-batch-size 4  # Was 1\n\n# Enable optimizations\n--use-flash-attn\n--sequence-parallel\n\n# Reduce TP if >8\n--tensor-model-parallel-size 4  # Was 16\n```\n\n**Issue: Out of memory**\n\nReduce memory with:\n```bash\n--tensor-model-parallel-size 2  # Split model across GPUs\n--recompute-granularity full  # Gradient checkpointing\n--recompute-method block  # Checkpoint transformer blocks\n--recompute-num-layers 1  # Checkpoint every layer\n```\n\nOr use CPU\u002FNVMe offloading:\n```bash\n--cpu-optimizer  # Offload optimizer to CPU\n--cpu-optimizer-type ADAM  # CPU Adam variant\n```\n\n**Issue: Training slower than expected**\n\nCheck:\n1. **Network bottleneck**: Ensure InfiniBand\u002FNVLink enabled\n2. **Pipeline bubbles**: Use interleaved pipeline schedule\n   ```bash\n   --num-layers-per-virtual-pipeline-stage 2\n   ```\n3. **Data loading**: Use fast data loader\n   ```bash\n   --dataloader-type cyclic\n   ```\n\n**Issue: Diverging loss**\n\nStabilize training:\n```bash\n--lr-warmup-iters 2000  # Longer warmup\n--clip-grad 1.0  # Gradient clipping\n--init-method-std 0.006  # Smaller init\n--attention-dropout 0.0  # No dropout in attention\n--hidden-dropout 0.0  # No dropout in FFN\n```\n\n## Advanced topics\n\n**Parallelism strategies**: See [references\u002Fparallelism-guide.md](references\u002Fparallelism-guide.md) for detailed comparison of TP\u002FPP\u002FDP\u002FCP\u002FEP with performance analysis and when to use each.\n\n**Performance benchmarks**: See [references\u002Fbenchmarks.md](references\u002Fbenchmarks.md) for MFU numbers across different model sizes and GPU configurations.\n\n**Production configurations**: See [references\u002Fproduction-examples.md](references\u002Fproduction-examples.md) for real-world setups from LLaMA 3 405B, Nemotron-4 340B, and DeepSeek-V3 671B.\n\n**Training recipes**: See [references\u002Ftraining-recipes.md](references\u002Ftraining-recipes.md) for complete hyperparameter configurations for GPT\u002FLLaMA\u002FMixtral architectures.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA Ampere+ (A100, H100, B200)\n  - Turing works but slower\n  - FP8 requires Hopper\u002FAda\u002FBlackwell\n- **Network**: InfiniBand or 400Gb+ Ethernet for multi-node\n- **Memory per GPU**:\n  - 7B model: 40GB+\n  - 70B model: 80GB (with TP=4)\n  - 405B model: 80GB (with TP=8, PP=8)\n- **Storage**: Fast NVMe for checkpoints (1TB+ for 70B+ models)\n\n## Resources\n\n- Docs: https:\u002F\u002Fdocs.nvidia.com\u002Fmegatron-core\u002F\n- GitHub: https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM\n- Papers:\n  - \"Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism\" (2019)\n  - \"Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM\" (2021)\n- NeMo Framework: https:\u002F\u002Fdocs.nvidia.com\u002Fnemo-framework\u002F (built on Megatron-Core)\n\n\n",{"name":5,"tags":44,"author":51,"license":14,"version":52,"description":7,"dependencies":53},[28,45,46,47,48,27,29,49,50],"Large-Scale Training","NVIDIA","Tensor Parallelism","Pipeline Parallelism","Distributed Training","Production","Orchestra Research","1.0.0",[54,55,30,56],"megatron-core","torch","transformer-engine","main","f19b254e8975d0bfddb394686706106930d7964d","AI-Training\u002Fmegatron-core\u002Fmegatron-core","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus","https:\u002F\u002Fgithub.com\u002Fendman100","2026-11-11 18:19:10",{"source":8,"skillName":5,"commands":64,"prompt":74,"zipAvailable":75},[65,68,71],{"tool":66,"command":67},"npx","npx skills add endman100\u002Fskill-router-nexus --skill training-llms-megatron",{"tool":69,"command":70},"bunx","bunx skills add endman100\u002Fskill-router-nexus --skill training-llms-megatron",{"tool":72,"command":73},"pnpm","pnpm dlx skills add endman100\u002Fskill-router-nexus --skill training-llms-megatron","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Training\u002Fmegatron-core\u002Fmegatron-core\u002FSKILL.md",true,[77,83,87,91],{"path":78,"resourceType":79,"resourceTypeDesc":80,"sizeBytes":81,"sourceHtmlUrl":82},"references\u002Fbenchmarks.md","REFERENCE","参考文档",7363,"https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Training\u002Fmegatron-core\u002Fmegatron-core\u002Freferences\u002Fbenchmarks.md",{"path":84,"resourceType":79,"resourceTypeDesc":80,"sizeBytes":85,"sourceHtmlUrl":86},"references\u002Fparallelism-guide.md",9591,"https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Training\u002Fmegatron-core\u002Fmegatron-core\u002Freferences\u002Fparallelism-guide.md",{"path":88,"resourceType":79,"resourceTypeDesc":80,"sizeBytes":89,"sourceHtmlUrl":90},"references\u002Fproduction-examples.md",10903,"https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Training\u002Fmegatron-core\u002Fmegatron-core\u002Freferences\u002Fproduction-examples.md",{"path":92,"resourceType":79,"resourceTypeDesc":80,"sizeBytes":93,"sourceHtmlUrl":94},"references\u002Ftraining-recipes.md",11588,"https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Training\u002Fmegatron-core\u002Fmegatron-core\u002Freferences\u002Ftraining-recipes.md",[96,120,158,180,189,213],{"skillNo":97,"name":98,"nameZh":6,"description":99,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":100,"sourceHtmlUrl":101,"categories":102,"keywords":108,"publishedAt":39},"01M6MPCT5BXVER8HQZTW3QE9AS","create-master","基于佛教经典文献，生成特定高僧大德的 AI 教学角色","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FMetaphysics-and-Spirituality\u002FMaster-skill\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FMetaphysics-and-Spirituality\u002FMaster-skill\u002FSKILL.md",[103],{"categoryNo":104,"categoryName":105,"taxonomyType":106,"taxonomyTypeDesc":107},"01JBSEEDCAT000000000000039","教师与教育","PROFESSION","适用职业",[109,110,111,112,113,114,115,116,117,118,119],"create master","master","create","生成","true","webfetch","edit","write","read","grep","Bash",{"skillNo":121,"name":122,"nameZh":6,"description":123,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":124,"sourceHtmlUrl":125,"categories":126,"keywords":138,"publishedAt":39},"01M6M43AKE95Y64E1JRXEF5QQ3","writing-dna-skill","从至少 20 篇完整文章中蒸馏可复用的写作 DNA，分析语言、文章结构、选题逻辑、素材策略、认知框架和视觉风格，并生成 Writing-DNA.md；按该 DNA 写作时会读取全部蒸馏产物和 5 篇相关原文。用于中英文作者、账号、品牌或出版物的风格分析与一致性写作。 Distill reusable Writing DNA from at least 20 complete articles for Chinese or English authors, publications, brands, and accounts; use for language, structure, topic logic, source strategy, cognitive-frame, and visual-style analysis.","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fwriting-dna-skill\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fwriting-dna-skill\u002FSKILL.md",[127,132,135],{"categoryNo":128,"categoryName":129,"taxonomyType":130,"taxonomyTypeDesc":131},"01JBSEEDCAT000000000000018","写作与文案","SCENARIO","使用场景",{"categoryNo":133,"categoryName":134,"taxonomyType":130,"taxonomyTypeDesc":131},"01JBSEEDCAT000000000000023","Skill与智能体开发",{"categoryNo":136,"categoryName":137,"taxonomyType":106,"taxonomyTypeDesc":107},"01JBSEEDCAT000000000000036","内容创作者",[139,140,141,142,143,144,145,146,147,148,149,150,151,112,152,153,154,155,156,157],"writing dna skill","DNA","writing","strategy","english","logic","visual","structure","chinese","reusable","use","style","distill","analysis","frame","source","for","at","complete",{"skillNo":159,"name":160,"nameZh":6,"description":161,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":162,"sourceHtmlUrl":163,"categories":164,"keywords":167,"publishedAt":39},"01M6M428TTCGQ4ME3KY99VN037","stop-slop","Remove AI writing patterns from prose. Use when drafting, editing, or reviewing text to eliminate predictable AI tells.","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fstop-slop\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fstop-slop\u002FSKILL.md",[165,166],{"categoryNo":128,"categoryName":129,"taxonomyType":130,"taxonomyTypeDesc":131},{"categoryNo":136,"categoryName":137,"taxonomyType":106,"taxonomyTypeDesc":107},[168,169,170,171,22,172,173,174,175,149,176,177,178,155,179,141],"slop","stop slop","stop","drafting","drafts","https","patterns","prose","remove","editing","text","content",{"skillNo":181,"name":182,"nameZh":6,"description":183,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":184,"sourceHtmlUrl":185,"categories":186,"keywords":187,"publishedAt":39},"01M6M41X0B7EBX42TK435DMGGR","shuorenhua","检查和清理中英文文本里的 AI 套路，适用于“去 AI 味”“说人话”“自然一点”“别像模板”“先标问题”这类改写和审稿需求；按场景控制力度，同时保留事实、术语、语域和责任主体。","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fshuorenhua\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fshuorenhua\u002FSKILL.md",[],[182,188],"文本",{"skillNo":190,"name":191,"nameZh":6,"description":192,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":193,"sourceHtmlUrl":194,"categories":195,"keywords":196,"publishedAt":39},"01M6M3ZD12WBEW5KXKDM4N6DFG","naming","Name products, SaaS, brands, open source projects, bots, and apps. Use when the user needs to name something, find a brand name, or pick a product name. Metaphor-driven process that produces memorable, meaningful names and avoids AI slop.","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fnaming-main\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fnaming-main\u002FSKILL.md",[],[191,197,198,199,200,201,149,202,117,154,203,204,205,206,119,207,208,209,210,22,211,212,168],"repo","websearch","product name","apps","bots","view","brand","pick","gh","projects","open","saas","open source","product","describe","ai slop",{"skillNo":214,"name":215,"nameZh":6,"description":216,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":217,"sourceHtmlUrl":218,"categories":219,"keywords":221,"publishedAt":39},"01M6M3YVXAFAAWJ7F6JMPFREYE","humanizer-zh","去除文本中的 AI 生成痕迹。适用于编辑或审阅文本，使其听起来更自然、更像人类书写。\n基于维基百科的\"AI 写作特征\"综合指南。检测并修复以下模式：夸大的象征意义、\n宣传性语言、以 -ing 结尾的肤浅分析、模糊的归因、破折号过度使用、三段式法则、\nAI 词汇、否定式排比、过多的连接性短语。","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fhumanizer-zh\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fhumanizer-zh\u002FSKILL.md",[220],{"categoryNo":128,"categoryName":129,"taxonomyType":130,"taxonomyTypeDesc":131},[222,223,224,117,116,115,141,112,188,225,168,170],"humanizer","zh","humanizer zh","翻译"]