[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6QN5EJXGMEZQP602V73PAYV":3,"skill-same:01M6QN5EJXGMEZQP602V73PAYV":102},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":61,"commitSha":62,"skillDirectory":63,"forksCount":14,"repositoryHtmlUrl":64,"repositoryHomepageUrl":65,"ownerHtmlUrl":66,"ownerBlogUrl":67,"ownerDescription":68,"updatedAt":69,"install":70,"resources":83},"01M6QN5EJXGMEZQP602V73PAYV","training-llms-megatron",null,"Trains large language models (2B-462B parameters) using NVIDIA Megatron-Core with advanced parallelism strategies. Use when training models >1B parameters, need maximum GPU efficiency (47% MFU on H100), or require tensor\u002Fpipeline\u002Fsequence\u002Fcontext\u002Fexpert parallelism. Production-ready framework used for Nemotron, LLaMA, DeepSeek.","Tashima-Tarsh\u002FDisha6.6","Tashima-Tarsh","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F269906498?v=4","01M6PBH98X03XERH0HFT4PFB1P","NITISH KUMAR","01M6PBH9DA0M6KM3Z12P848F0K",0,"MIT","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fqc-megatron-core\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"llms","training","training llms megatron","megatron","language models","large language models","for","production-ready","trains","scale","strategies","expert","models","ready","use","context","engine","NVIDIA","distributed","research","2026-11-10 10:12:30",false,"---\nname: training-llms-megatron\ndescription: Trains large language models (2B-462B parameters) using NVIDIA Megatron-Core with advanced parallelism strategies. Use when training models >1B parameters, need maximum GPU efficiency (47% MFU on H100), or require tensor\u002Fpipeline\u002Fsequence\u002Fcontext\u002Fexpert parallelism. Production-ready framework used for Nemotron, LLaMA, DeepSeek.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Megatron-Core, Large-Scale Training, NVIDIA, Tensor Parallelism, Pipeline Parallelism, Model Parallelism, H100, Distributed Training, Production]\ndependencies: [megatron-core, torch, apex, transformer-engine]\ncategory: \"ai-research\"\n---\n\n# Megatron-Core - Large-Scale LLM Training\n\n## Quick start\n\nMegatron-Core trains LLMs from 2B to 462B parameters with up to 47% Model FLOP Utilization on H100 GPUs through advanced parallelism strategies.\n\n**Installation**:\n```bash\n# Docker (recommended)\ndocker run --gpus all -it --rm nvcr.io\u002Fnvidia\u002Fpytorch:25.04-py3\n\n# Or pip\npip install megatron-core\n```\n\n**Simple distributed training**:\n```bash\n# Train with 2 GPUs using data parallelism\ntorchrun --nproc_per_node=2 examples\u002Frun_simple_mcore_train_loop.py\n\n# Or LLaMA-3 8B training\n.\u002Fexamples\u002Fllama\u002Ftrain_llama3_8b_fp8.sh\n```\n\n## Common workflows\n\n### Workflow 1: Train LLaMA-style model with 3D parallelism\n\nCopy this checklist:\n\n```\nLLaMA Training Setup:\n- [ ] Step 1: Choose parallelism configuration\n- [ ] Step 2: Configure training hyperparameters\n- [ ] Step 3: Launch distributed training\n- [ ] Step 4: Monitor performance metrics\n```\n\n**Step 1: Choose parallelism configuration**\n\nModel size determines parallelism strategy:\n\n| Model Size | GPUs | Tensor Parallel | Pipeline Parallel | Data Parallel | Context Parallel |\n|------------|------|-----------------|-------------------|---------------|------------------|\n| 7B | 8 | 1 | 1 | 8 | 1 |\n| 13B | 8 | 2 | 1 | 4 | 1 |\n| 70B | 64 | 4 | 4 | 4 | 1 |\n| 405B | 128 | 8 | 8 | 2 | 2 |\n\n**Step 2: Configure training hyperparameters**\n\n```bash\n#!\u002Fbin\u002Fbash\n# train_llama_70b.sh\n\nGPUS_PER_NODE=8\nNNODES=8  # 64 GPUs total\nTP=4      # Tensor parallel\nPP=4      # Pipeline parallel\nCP=1      # Context parallel\n\n# LLaMA 70B configuration\nMODEL_SIZE=70  # Billion parameters\nHIDDEN_SIZE=8192\nNUM_LAYERS=80\nNUM_HEADS=64\nSEQ_LENGTH=4096\n\n# Training hyperparameters\nMICRO_BATCH=1\nGLOBAL_BATCH=1024\nLR=3e-4\n\ntorchrun \\\n  --nproc_per_node=$GPUS_PER_NODE \\\n  --nnodes=$NNODES \\\n  pretrain_gpt.py \\\n  --tensor-model-parallel-size $TP \\\n  --pipeline-model-parallel-size $PP \\\n  --context-parallel-size $CP \\\n  --sequence-parallel \\\n  --num-layers $NUM_LAYERS \\\n  --hidden-size $HIDDEN_SIZE \\\n  --num-attention-heads $NUM_HEADS \\\n  --seq-length $SEQ_LENGTH \\\n  --max-position-embeddings $SEQ_LENGTH \\\n  --micro-batch-size $MICRO_BATCH \\\n  --global-batch-size $GLOBAL_BATCH \\\n  --lr $LR \\\n  --train-iters 100000 \\\n  --lr-decay-style cosine \\\n  --lr-warmup-iters 2000 \\\n  --weight-decay 0.1 \\\n  --clip-grad 1.0 \\\n  --bf16 \\\n  --use-mcore-models \\\n  --transformer-impl transformer_engine \\\n  --data-path \u002Fpath\u002Fto\u002Fdata \\\n  --vocab-file \u002Fpath\u002Fto\u002Fvocab.json \\\n  --merge-file \u002Fpath\u002Fto\u002Fmerges.txt\n```\n\n**Step 3: Launch distributed training**\n\n```bash\n# Single node (8 GPUs)\nbash train_llama_70b.sh\n\n# Multi-node with SLURM\nsbatch --nodes=8 --gpus-per-node=8 train_llama_70b.sh\n```\n\n**Step 4: Monitor performance metrics**\n\nKey metrics to track:\n```\nModel FLOP Utilization (MFU): Target >40% on H100\nThroughput: Tokens\u002Fsec\u002FGPU\nMemory usage: \u003C80GB per GPU for 70B model\nLoss: Should decrease steadily\n```\n\n### Workflow 2: Configure Mixture of Experts (MoE) training\n\nFor sparse MoE models like Mixtral.\n\n```\nMoE Training:\n- [ ] Step 1: Configure expert parallelism\n- [ ] Step 2: Set MoE hyperparameters\n- [ ] Step 3: Launch training with EP\n```\n\n**Step 1: Configure expert parallelism**\n\n```bash\n# Mixtral 8x7B example\nTENSOR_PARALLEL=2\nPIPELINE_PARALLEL=1\nEXPERT_PARALLEL=4  # Split 8 experts across 4 GPUs\nDATA_PARALLEL=4\n\nTOTAL_GPUS=$((TENSOR_PARALLEL * PIPELINE_PARALLEL * EXPERT_PARALLEL * DATA_PARALLEL))\n# = 2 * 1 * 4 * 4 = 32 GPUs\n```\n\n**Step 2: Set MoE hyperparameters**\n\n```bash\ntorchrun \\\n  --nproc_per_node=8 \\\n  pretrain_gpt.py \\\n  --tensor-model-parallel-size 2 \\\n  --pipeline-model-parallel-size 1 \\\n  --expert-model-parallel-size 4 \\\n  --num-experts 8 \\\n  --moe-router-topk 2 \\\n  --moe-router-load-balancing-type aux_loss \\\n  --moe-aux-loss-coeff 0.01 \\\n  --hidden-size 4096 \\\n  --num-layers 32 \\\n  --num-attention-heads 32 \\\n  --seq-length 4096 \\\n  --max-position-embeddings 4096 \\\n  --bf16 \\\n  --use-mcore-models \\\n  --transformer-impl transformer_engine \\\n  --data-path \u002Fpath\u002Fto\u002Fdata \\\n  --vocab-file \u002Fpath\u002Fto\u002Fvocab.json \\\n  --merge-file \u002Fpath\u002Fto\u002Fmerges.txt\n```\n\n**Step 3: Launch training with EP**\n\nExpert parallelism distributes different experts across GPUs, reducing memory while maintaining capacity.\n\n```\nMemory without EP: 8 experts × 7B = 56GB per GPU\nMemory with EP=4: 2 experts × 7B = 14GB per GPU\nSavings: 75% memory reduction\n```\n\n### Workflow 3: Optimize for maximum throughput\n\nAchieve 47% MFU on H100.\n\n```\nPerformance Optimization:\n- [ ] Step 1: Enable Flash Attention\n- [ ] Step 2: Use FP8 precision (H100)\n- [ ] Step 3: Optimize micro-batch size\n- [ ] Step 4: Tune parallelism degrees\n```\n\n**Step 1: Enable optimizations**\n\n```bash\n--use-mcore-models  # Use Megatron Core models\n--transformer-impl transformer_engine  # Use Transformer Engine\n--sequence-parallel  # Reduce activation memory (use with TP)\n```\n\n**Step 2: Use FP8 precision (H100 only)**\n\n```bash\n--fp8-hybrid  # FP8 mixed precision training\n# Transformer Engine handles FP8 automatically\n```\n\nResult: 1.5-2x speedup on H100 vs BF16.\n\n**Step 3: Optimize micro-batch size**\n\nFind largest micro-batch that fits in memory:\n\n```bash\n# Start with 1, increase until OOM\nfor MBS in 1 2 4 8; do\n  echo \"Testing micro-batch-size=$MBS\"\n  torchrun ... --micro-batch-size $MBS\ndone\n```\n\nTypical values:\n- 7B model: 4-8\n- 70B model: 1-2\n- 405B model: 1\n\n**Step 4: Tune parallelism degrees**\n\nRules of thumb:\n```\nTensor Parallel: Use ≤8 (limited by NVLink within node)\nPipeline Parallel: Use for >70B models\nContext Parallel: Use for sequences >8K tokens\nData Parallel: Fill remaining GPUs\n```\n\nExample 405B on 128 H100s:\n```\nTP=8 (1 node)\nPP=8 (across nodes)\nCP=2 (long sequences)\nDP=1\nTotal = 8 × 8 × 2 × 1 = 128 GPUs\n```\n\n## When to use vs alternatives\n\n**Use Megatron-Core when:**\n- Training models >10B parameters\n- Need maximum efficiency (target >40% MFU)\n- Using NVIDIA GPUs (A100, H100)\n- Production training at scale\n- Want fine-grained parallelism control\n\n**Use alternatives instead:**\n- **PyTorch FSDP**: Models \u003C70B, simpler API, PyTorch native\n- **DeepSpeed**: Easier setup, good for \u003C100B models\n- **HuggingFace Accelerate**: Prototyping, simpler workflows\n- **LitGPT**: Educational, single-file implementations\n\n## Common issues\n\n**Issue: Low GPU utilization (\u003C30% MFU)**\n\nCauses:\n1. Micro-batch too small\n2. Too much parallelism overhead\n3. Not using Flash Attention\n\nFixes:\n```bash\n# Increase micro-batch\n--micro-batch-size 4  # Was 1\n\n# Enable optimizations\n--use-flash-attn\n--sequence-parallel\n\n# Reduce TP if >8\n--tensor-model-parallel-size 4  # Was 16\n```\n\n**Issue: Out of memory**\n\nReduce memory with:\n```bash\n--tensor-model-parallel-size 2  # Split model across GPUs\n--recompute-granularity full  # Gradient checkpointing\n--recompute-method block  # Checkpoint transformer blocks\n--recompute-num-layers 1  # Checkpoint every layer\n```\n\nOr use CPU\u002FNVMe offloading:\n```bash\n--cpu-optimizer  # Offload optimizer to CPU\n--cpu-optimizer-type ADAM  # CPU Adam variant\n```\n\n**Issue: Training slower than expected**\n\nCheck:\n1. **Network bottleneck**: Ensure InfiniBand\u002FNVLink enabled\n2. **Pipeline bubbles**: Use interleaved pipeline schedule\n   ```bash\n   --num-layers-per-virtual-pipeline-stage 2\n   ```\n3. **Data loading**: Use fast data loader\n   ```bash\n   --dataloader-type cyclic\n   ```\n\n**Issue: Diverging loss**\n\nStabilize training:\n```bash\n--lr-warmup-iters 2000  # Longer warmup\n--clip-grad 1.0  # Gradient clipping\n--init-method-std 0.006  # Smaller init\n--attention-dropout 0.0  # No dropout in attention\n--hidden-dropout 0.0  # No dropout in FFN\n```\n\n## Advanced topics\n\n**Parallelism strategies**: See [references\u002Fparallelism-guide.md](references\u002Fparallelism-guide.md) for detailed comparison of TP\u002FPP\u002FDP\u002FCP\u002FEP with performance analysis and when to use each.\n\n**Performance benchmarks**: See [references\u002Fbenchmarks.md](references\u002Fbenchmarks.md) for MFU numbers across different model sizes and GPU configurations.\n\n**Production configurations**: See [references\u002Fproduction-examples.md](references\u002Fproduction-examples.md) for real-world setups from LLaMA 3 405B, Nemotron-4 340B, and DeepSeek-V3 671B.\n\n**Training recipes**: See [references\u002Ftraining-recipes.md](references\u002Ftraining-recipes.md) for complete hyperparameter configurations for GPT\u002FLLaMA\u002FMixtral architectures.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA Ampere+ (A100, H100, B200)\n  - Turing works but slower\n  - FP8 requires Hopper\u002FAda\u002FBlackwell\n- **Network**: InfiniBand or 400Gb+ Ethernet for multi-node\n- **Memory per GPU**:\n  - 7B model: 40GB+\n  - 70B model: 80GB (with TP=4)\n  - 405B model: 80GB (with TP=8, PP=8)\n- **Storage**: Fast NVMe for checkpoints (1TB+ for 70B+ models)\n\n## Resources\n\n- Docs: https:\u002F\u002Fdocs.nvidia.com\u002Fmegatron-core\u002F\n- GitHub: https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM\n- Papers:\n  - \"Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism\" (2019)\n  - \"Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM\" (2021)\n- NeMo Framework: https:\u002F\u002Fdocs.nvidia.com\u002Fnemo-framework\u002F (built on Megatron-Core)\n\n\n","# Megatron-Core - Large-Scale LLM Training\n\n## Quick start\n\nMegatron-Core trains LLMs from 2B to 462B parameters with up to 47% Model FLOP Utilization on H100 GPUs through advanced parallelism strategies.\n\n**Installation**:\n```bash\n# Docker (recommended)\ndocker run --gpus all -it --rm nvcr.io\u002Fnvidia\u002Fpytorch:25.04-py3\n\n# Or pip\npip install megatron-core\n```\n\n**Simple distributed training**:\n```bash\n# Train with 2 GPUs using data parallelism\ntorchrun --nproc_per_node=2 examples\u002Frun_simple_mcore_train_loop.py\n\n# Or LLaMA-3 8B training\n.\u002Fexamples\u002Fllama\u002Ftrain_llama3_8b_fp8.sh\n```\n\n## Common workflows\n\n### Workflow 1: Train LLaMA-style model with 3D parallelism\n\nCopy this checklist:\n\n```\nLLaMA Training Setup:\n- [ ] Step 1: Choose parallelism configuration\n- [ ] Step 2: Configure training hyperparameters\n- [ ] Step 3: Launch distributed training\n- [ ] Step 4: Monitor performance metrics\n```\n\n**Step 1: Choose parallelism configuration**\n\nModel size determines parallelism strategy:\n\n| Model Size | GPUs | Tensor Parallel | Pipeline Parallel | Data Parallel | Context Parallel |\n|------------|------|-----------------|-------------------|---------------|------------------|\n| 7B | 8 | 1 | 1 | 8 | 1 |\n| 13B | 8 | 2 | 1 | 4 | 1 |\n| 70B | 64 | 4 | 4 | 4 | 1 |\n| 405B | 128 | 8 | 8 | 2 | 2 |\n\n**Step 2: Configure training hyperparameters**\n\n```bash\n#!\u002Fbin\u002Fbash\n# train_llama_70b.sh\n\nGPUS_PER_NODE=8\nNNODES=8  # 64 GPUs total\nTP=4      # Tensor parallel\nPP=4      # Pipeline parallel\nCP=1      # Context parallel\n\n# LLaMA 70B configuration\nMODEL_SIZE=70  # Billion parameters\nHIDDEN_SIZE=8192\nNUM_LAYERS=80\nNUM_HEADS=64\nSEQ_LENGTH=4096\n\n# Training hyperparameters\nMICRO_BATCH=1\nGLOBAL_BATCH=1024\nLR=3e-4\n\ntorchrun \\\n  --nproc_per_node=$GPUS_PER_NODE \\\n  --nnodes=$NNODES \\\n  pretrain_gpt.py \\\n  --tensor-model-parallel-size $TP \\\n  --pipeline-model-parallel-size $PP \\\n  --context-parallel-size $CP \\\n  --sequence-parallel \\\n  --num-layers $NUM_LAYERS \\\n  --hidden-size $HIDDEN_SIZE \\\n  --num-attention-heads $NUM_HEADS \\\n  --seq-length $SEQ_LENGTH \\\n  --max-position-embeddings $SEQ_LENGTH \\\n  --micro-batch-size $MICRO_BATCH \\\n  --global-batch-size $GLOBAL_BATCH \\\n  --lr $LR \\\n  --train-iters 100000 \\\n  --lr-decay-style cosine \\\n  --lr-warmup-iters 2000 \\\n  --weight-decay 0.1 \\\n  --clip-grad 1.0 \\\n  --bf16 \\\n  --use-mcore-models \\\n  --transformer-impl transformer_engine \\\n  --data-path \u002Fpath\u002Fto\u002Fdata \\\n  --vocab-file \u002Fpath\u002Fto\u002Fvocab.json \\\n  --merge-file \u002Fpath\u002Fto\u002Fmerges.txt\n```\n\n**Step 3: Launch distributed training**\n\n```bash\n# Single node (8 GPUs)\nbash train_llama_70b.sh\n\n# Multi-node with SLURM\nsbatch --nodes=8 --gpus-per-node=8 train_llama_70b.sh\n```\n\n**Step 4: Monitor performance metrics**\n\nKey metrics to track:\n```\nModel FLOP Utilization (MFU): Target >40% on H100\nThroughput: Tokens\u002Fsec\u002FGPU\nMemory usage: \u003C80GB per GPU for 70B model\nLoss: Should decrease steadily\n```\n\n### Workflow 2: Configure Mixture of Experts (MoE) training\n\nFor sparse MoE models like Mixtral.\n\n```\nMoE Training:\n- [ ] Step 1: Configure expert parallelism\n- [ ] Step 2: Set MoE hyperparameters\n- [ ] Step 3: Launch training with EP\n```\n\n**Step 1: Configure expert parallelism**\n\n```bash\n# Mixtral 8x7B example\nTENSOR_PARALLEL=2\nPIPELINE_PARALLEL=1\nEXPERT_PARALLEL=4  # Split 8 experts across 4 GPUs\nDATA_PARALLEL=4\n\nTOTAL_GPUS=$((TENSOR_PARALLEL * PIPELINE_PARALLEL * EXPERT_PARALLEL * DATA_PARALLEL))\n# = 2 * 1 * 4 * 4 = 32 GPUs\n```\n\n**Step 2: Set MoE hyperparameters**\n\n```bash\ntorchrun \\\n  --nproc_per_node=8 \\\n  pretrain_gpt.py \\\n  --tensor-model-parallel-size 2 \\\n  --pipeline-model-parallel-size 1 \\\n  --expert-model-parallel-size 4 \\\n  --num-experts 8 \\\n  --moe-router-topk 2 \\\n  --moe-router-load-balancing-type aux_loss \\\n  --moe-aux-loss-coeff 0.01 \\\n  --hidden-size 4096 \\\n  --num-layers 32 \\\n  --num-attention-heads 32 \\\n  --seq-length 4096 \\\n  --max-position-embeddings 4096 \\\n  --bf16 \\\n  --use-mcore-models \\\n  --transformer-impl transformer_engine \\\n  --data-path \u002Fpath\u002Fto\u002Fdata \\\n  --vocab-file \u002Fpath\u002Fto\u002Fvocab.json \\\n  --merge-file \u002Fpath\u002Fto\u002Fmerges.txt\n```\n\n**Step 3: Launch training with EP**\n\nExpert parallelism distributes different experts across GPUs, reducing memory while maintaining capacity.\n\n```\nMemory without EP: 8 experts × 7B = 56GB per GPU\nMemory with EP=4: 2 experts × 7B = 14GB per GPU\nSavings: 75% memory reduction\n```\n\n### Workflow 3: Optimize for maximum throughput\n\nAchieve 47% MFU on H100.\n\n```\nPerformance Optimization:\n- [ ] Step 1: Enable Flash Attention\n- [ ] Step 2: Use FP8 precision (H100)\n- [ ] Step 3: Optimize micro-batch size\n- [ ] Step 4: Tune parallelism degrees\n```\n\n**Step 1: Enable optimizations**\n\n```bash\n--use-mcore-models  # Use Megatron Core models\n--transformer-impl transformer_engine  # Use Transformer Engine\n--sequence-parallel  # Reduce activation memory (use with TP)\n```\n\n**Step 2: Use FP8 precision (H100 only)**\n\n```bash\n--fp8-hybrid  # FP8 mixed precision training\n# Transformer Engine handles FP8 automatically\n```\n\nResult: 1.5-2x speedup on H100 vs BF16.\n\n**Step 3: Optimize micro-batch size**\n\nFind largest micro-batch that fits in memory:\n\n```bash\n# Start with 1, increase until OOM\nfor MBS in 1 2 4 8; do\n  echo \"Testing micro-batch-size=$MBS\"\n  torchrun ... --micro-batch-size $MBS\ndone\n```\n\nTypical values:\n- 7B model: 4-8\n- 70B model: 1-2\n- 405B model: 1\n\n**Step 4: Tune parallelism degrees**\n\nRules of thumb:\n```\nTensor Parallel: Use ≤8 (limited by NVLink within node)\nPipeline Parallel: Use for >70B models\nContext Parallel: Use for sequences >8K tokens\nData Parallel: Fill remaining GPUs\n```\n\nExample 405B on 128 H100s:\n```\nTP=8 (1 node)\nPP=8 (across nodes)\nCP=2 (long sequences)\nDP=1\nTotal = 8 × 8 × 2 × 1 = 128 GPUs\n```\n\n## When to use vs alternatives\n\n**Use Megatron-Core when:**\n- Training models >10B parameters\n- Need maximum efficiency (target >40% MFU)\n- Using NVIDIA GPUs (A100, H100)\n- Production training at scale\n- Want fine-grained parallelism control\n\n**Use alternatives instead:**\n- **PyTorch FSDP**: Models \u003C70B, simpler API, PyTorch native\n- **DeepSpeed**: Easier setup, good for \u003C100B models\n- **HuggingFace Accelerate**: Prototyping, simpler workflows\n- **LitGPT**: Educational, single-file implementations\n\n## Common issues\n\n**Issue: Low GPU utilization (\u003C30% MFU)**\n\nCauses:\n1. Micro-batch too small\n2. Too much parallelism overhead\n3. Not using Flash Attention\n\nFixes:\n```bash\n# Increase micro-batch\n--micro-batch-size 4  # Was 1\n\n# Enable optimizations\n--use-flash-attn\n--sequence-parallel\n\n# Reduce TP if >8\n--tensor-model-parallel-size 4  # Was 16\n```\n\n**Issue: Out of memory**\n\nReduce memory with:\n```bash\n--tensor-model-parallel-size 2  # Split model across GPUs\n--recompute-granularity full  # Gradient checkpointing\n--recompute-method block  # Checkpoint transformer blocks\n--recompute-num-layers 1  # Checkpoint every layer\n```\n\nOr use CPU\u002FNVMe offloading:\n```bash\n--cpu-optimizer  # Offload optimizer to CPU\n--cpu-optimizer-type ADAM  # CPU Adam variant\n```\n\n**Issue: Training slower than expected**\n\nCheck:\n1. **Network bottleneck**: Ensure InfiniBand\u002FNVLink enabled\n2. **Pipeline bubbles**: Use interleaved pipeline schedule\n   ```bash\n   --num-layers-per-virtual-pipeline-stage 2\n   ```\n3. **Data loading**: Use fast data loader\n   ```bash\n   --dataloader-type cyclic\n   ```\n\n**Issue: Diverging loss**\n\nStabilize training:\n```bash\n--lr-warmup-iters 2000  # Longer warmup\n--clip-grad 1.0  # Gradient clipping\n--init-method-std 0.006  # Smaller init\n--attention-dropout 0.0  # No dropout in attention\n--hidden-dropout 0.0  # No dropout in FFN\n```\n\n## Advanced topics\n\n**Parallelism strategies**: See [references\u002Fparallelism-guide.md](references\u002Fparallelism-guide.md) for detailed comparison of TP\u002FPP\u002FDP\u002FCP\u002FEP with performance analysis and when to use each.\n\n**Performance benchmarks**: See [references\u002Fbenchmarks.md](references\u002Fbenchmarks.md) for MFU numbers across different model sizes and GPU configurations.\n\n**Production configurations**: See [references\u002Fproduction-examples.md](references\u002Fproduction-examples.md) for real-world setups from LLaMA 3 405B, Nemotron-4 340B, and DeepSeek-V3 671B.\n\n**Training recipes**: See [references\u002Ftraining-recipes.md](references\u002Ftraining-recipes.md) for complete hyperparameter configurations for GPT\u002FLLaMA\u002FMixtral architectures.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA Ampere+ (A100, H100, B200)\n  - Turing works but slower\n  - FP8 requires Hopper\u002FAda\u002FBlackwell\n- **Network**: InfiniBand or 400Gb+ Ethernet for multi-node\n- **Memory per GPU**:\n  - 7B model: 40GB+\n  - 70B model: 80GB (with TP=4)\n  - 405B model: 80GB (with TP=8, PP=8)\n- **Storage**: Fast NVMe for checkpoints (1TB+ for 70B+ models)\n\n## Resources\n\n- Docs: https:\u002F\u002Fdocs.nvidia.com\u002Fmegatron-core\u002F\n- GitHub: https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM\n- Papers:\n  - \"Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism\" (2019)\n  - \"Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM\" (2021)\n- NeMo Framework: https:\u002F\u002Fdocs.nvidia.com\u002Fnemo-framework\u002F (built on Megatron-Core)\n\n\n",{"name":5,"tags":44,"author":53,"license":15,"version":54,"category":55,"description":7,"dependencies":56},[45,46,36,47,48,49,50,51,52],"Megatron-Core","Large-Scale Training","Tensor Parallelism","Pipeline Parallelism","Model Parallelism","H100","Distributed Training","Production","Orchestra Research","1.0.0","ai-research",[57,58,59,60],"megatron-core","torch","apex","transformer-engine","main","0da4485785ee599388900be50db1635489d53056",".agents\u002Fskills\u002Fqc-megatron-core","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6","https:\u002F\u002Fdisha6.6.thenitishkr.in\u002F","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh","www.thenitishkr","thenitishkr | AI • Cyber-security • Research\r\n","2026-11-08 03:33:25",{"source":8,"skillName":5,"commands":71,"prompt":81,"zipAvailable":82},[72,75,78],{"tool":73,"command":74},"npx","npx skills add Tashima-Tarsh\u002FDisha6.6 --skill training-llms-megatron",{"tool":76,"command":77},"bunx","bunx skills add Tashima-Tarsh\u002FDisha6.6 --skill training-llms-megatron",{"tool":79,"command":80},"pnpm","pnpm dlx skills add Tashima-Tarsh\u002FDisha6.6 --skill training-llms-megatron","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fqc-megatron-core\u002FSKILL.md",true,[84,90,94,98],{"path":85,"resourceType":86,"resourceTypeDesc":87,"sizeBytes":88,"sourceHtmlUrl":89},"references\u002Fbenchmarks.md","REFERENCE","参考文档",7363,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fqc-megatron-core\u002Freferences\u002Fbenchmarks.md",{"path":91,"resourceType":86,"resourceTypeDesc":87,"sizeBytes":92,"sourceHtmlUrl":93},"references\u002Fparallelism-guide.md",9591,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fqc-megatron-core\u002Freferences\u002Fparallelism-guide.md",{"path":95,"resourceType":86,"resourceTypeDesc":87,"sizeBytes":96,"sourceHtmlUrl":97},"references\u002Fproduction-examples.md",10903,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fqc-megatron-core\u002Freferences\u002Fproduction-examples.md",{"path":99,"resourceType":86,"resourceTypeDesc":87,"sizeBytes":100,"sourceHtmlUrl":101},"references\u002Ftraining-recipes.md",11588,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fqc-megatron-core\u002Freferences\u002Ftraining-recipes.md",[103,134,156,183,208,231],{"skillNo":104,"name":105,"nameZh":6,"description":106,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":107,"categories":108,"keywords":114,"publishedAt":39},"01M6QRD07D4TGNDK6VJ4EHF8BV","youtube-full","Use when the user needs YouTube transcripts, video search, channel browsing, playlist extraction, or content monitoring. Trigger phrases: 'get the transcript for', 'search YouTube for', 'what are the latest videos on', 'list this playlist', 'monitor this channel', or any request involving a YouTube URL, video ID, or @handle. Do NOT use for downloading video or audio files, YouTube engagement data (likes, comments), or private\u002Fage-restricted videos.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fyoutube-full\u002FSKILL.md",[109],{"categoryNo":110,"categoryName":111,"taxonomyType":112,"taxonomyTypeDesc":113},"01JBSEEDCAT000000000000016","图像与多媒体","SCENARIO","使用场景",[115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,25,131,132,133],"youtube full","youtube","full","transcript","search","monitoring","data","Video","Audio","list","files","do","get","age","trigger","URL","browsing","request","youtube transcripts",{"skillNo":135,"name":136,"nameZh":6,"description":137,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":138,"categories":139,"keywords":140,"publishedAt":39},"01M6QRCJ3112VYPA97YG7KGMC2","x-twitter-growth","X\u002FTwitter growth engine for building audience, crafting viral content, and analyzing engagement. Use when the user wants to grow on X\u002FTwitter, write tweets or threads, analyze their X profile, research competitors on X, plan a posting strategy, or optimize engagement. Complements social-content (generic multi-platform) with X-specific depth: algorithm mechanics, thread engineering, reply strategy, profile optimization, and competitive intelligence via web search.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fx-twitter-growth\u002FSKILL.md",[],[141,142,143,38,119,144,145,146,147,148,149,35,150,151,33,152,153,154,155,25],"twitter","x twitter growth","growth","content","engineering","plan","web search","competitors","analyze","write","strategy","platform","grow","posting","competitive",{"skillNo":157,"name":158,"nameZh":6,"description":159,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":160,"categories":161,"keywords":165,"publishedAt":39},"01M6QRBN3EHQNJY42RERP7A149","write-a-skill","Create new agent skills with proper structure, progressive disclosure, and bundled resources. Use when user wants to create, write, build, or author a new skill.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fwrite-a-skill\u002FSKILL.md",[162],{"categoryNo":163,"categoryName":164,"taxonomyType":112,"taxonomyTypeDesc":113},"01JBSEEDCAT000000000000023","Skill与智能体开发",[166,150,167,168,169,170,171,172,33,173,174,61,175,176,177,178,179,180,181,182],"write a skill","github","create","https","productivity","matt pocock","direct","example","build","new skill","structure","new agent","resources","with","tree","progressive disclosure","author",{"skillNo":184,"name":185,"nameZh":6,"description":186,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":187,"categories":188,"keywords":189,"publishedAt":39},"01M6QRB6G8P33PNGAF9Y69EJ9T","wrap-up","Close out a launched Claude Managed Agent — recap every primitive the founder now owns, regenerate the single-file overview page, and suggest the next 1-2 upgrades. Use when the user says \"wrap up\", \"close this out\", \"what do I own now\", \"give me the summary\", \"recap the agent\", or when the orchestrator routes phase=wrap-up. primitives_inventory.py tables everything owned (agent, environment, session, memory, outcome, deployment); overview_page.py regenerates a self-contained .\u002Fmy-agent\u002Fagent-overview.html; upgrade_suggester.py ranks the next moves from recorded deferrals plus standing hardening steps. Companion to run-without-you; the last stop before phase=done.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fwrap-up\u002FSKILL.md",[],[190,191,192,193,194,195,196,197,198,33,199,126,200,201,202,203,204,205,206,207],"wrap up","up","wrap","gemini","deployment","run","file","hardening","closeout","stop","upgrade","founder","close","companion","you","tables","directions","phase",{"skillNo":209,"name":210,"nameZh":6,"description":211,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":212,"categories":213,"keywords":215,"publishedAt":39},"01M6QRA5AYC1RBFZ39GTWGS3N8","workflow-builder","Design and write deterministic multi-agent workflow scripts (.js files in .claude\u002Fworkflows\u002F) for Claude Code's Workflow tool. Use when a user wants to build, create, author, scaffold, or run a custom Claude Code workflow, orchestrate sub-agents (fan-out, pipeline, loop, judge-panel), or automate a repeatable multi-step task across fresh-context agents.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fworkflow-builder\u002FSKILL.md",[214],{"categoryNo":163,"categoryName":164,"taxonomyType":112,"taxonomyTypeDesc":113},[216,217,218,219,220,168,195,221,222,223,224,225,150,34,125,226,227,228,229,230],"workflow","workflow builder","builder","design","workflows","code","orchestrate","pipeline","loop","multi-agent","across","fresh","scripts","judge","claude code workflow",{"skillNo":232,"name":233,"nameZh":6,"description":234,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":235,"categories":236,"keywords":237,"publishedAt":39},"01M6QR7QMCZESKQA0ERW1E657M","weekly-review","Use when someone wants to run a weekly review, close open loops, audit stalled projects and commitments, get their system back to trusted, restart a lapsed review habit, or says \"\u002Fcs:weekly-review\". Walks David Allen's three-phase loop — GET CLEAR, GET CURRENT, GET CREATIVE — with deterministic scripts that inventory open loops, gate the checklist with named gaps, and score commitment health 0-100.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fweekly-review\u002FSKILL.md",[],[238,239,240,241,242,243,244,245,246,224,247,248,249,250,251,252,127,253,254,207],"review","weekly review","weekly","deterministic scripts","reflect","feeds","inventory","open","team","projects","project-management","scanner","conversation","back","project","sprint","cadence"]