[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6PQFY2KPG3FAAS0SEXGVM9Y":3,"skill-same:01M6PQFY2KPG3FAAS0SEXGVM9Y":90},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":45,"commitSha":46,"skillDirectory":47,"forksCount":14,"repositoryHtmlUrl":48,"repositoryHomepageUrl":6,"ownerHtmlUrl":49,"ownerBlogUrl":50,"ownerDescription":6,"updatedAt":51,"install":52,"resources":65},"01M6PQFY2KPG3FAAS0SEXGVM9Y","nemo-mbridge-perf-memory-tuning",null,"Techniques for reducing peak GPU memory in Megatron Bridge — expandable segments, parallelism resizing, activation recompute, CPU offloading constraints, and common OOM fixes.","headmktlucca-ui\u002Fneuroads-lp","headmktlucca-ui","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F247379838?v=4","01M6PDVAV6M1W96D1SR1PDF8CS","NeuroAds","01M6PDVAZ4P281GNDKD45CYTF5",0,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fnemo-mbridge-perf-memory-tuning\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"nemo mbridge perf memory tuning","perf","NeMo","tuning","memory","bridge","fixes","reduce","for","techniques","errors","CPU","megatron","oom","common","regression","gpu","pytorch","commit","change","2026-11-10 10:12:30",false,"---\nname: nemo-mbridge-perf-memory-tuning\ndescription: Techniques for reducing peak GPU memory in Megatron Bridge — expandable segments, parallelism resizing, activation recompute, CPU offloading constraints, and common OOM fixes.\nlicense: Apache-2.0\nwhen_to_use: GPU OOM errors, reducing peak memory, or tracing an OOM regression to a specific commit or config change; 'out of memory', 'OOM', 'memory fragmentation', 'expandable_segments', 'reduce GPU memory', 'PYTORCH_CUDA_ALLOC_CONF'.\n---\n\n# Memory Tuning\n\nStable docs: @docs\u002Fparallelisms.md\nCard: @skills\u002Fnemo-mbridge-perf-memory-tuning\u002Fcard.yaml\n\n## What It Is\n\nGPU OOM failures during training often stem from memory **fragmentation** rather\nthan raw capacity.  PyTorch's default CUDA allocator can leave unusable gaps\nbetween allocations.  The single most effective fix is:\n\n```bash\nexport PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True\n```\n\nThis tells PyTorch to use expandable (non-fixed-size) memory segments, which\ndramatically reduces fragmentation and often eliminates borderline OOM without\nany model or parallelism changes.\n\nBeyond fragmentation, actual peak memory is determined by:\n\n- **Parameter + optimizer state memory** — controlled by TP, PP, DP sharding\n  (distributed optimizer, FSDP)\n- **Activation memory** — controlled by activation recompute, sequence length,\n  micro-batch size\n- **Temporary \u002F workspace memory** — CUDA kernels, NCCL buffers, CUDA graphs\n\nFor configuration planning, use the Bridge theoretical estimator before launching\nlarge jobs:\n\n```python\nfrom megatron.bridge.training.utils.theoretical_memory_utils import estimate_training_memory\n\nestimate = estimate_training_memory(cfg, num_microbatches=num_microbatches)\n```\n\nThe estimator reports the most-loaded GPU shard and separates dense\u002Fembedding,\nrouted MoE expert, and activation components. It does not include allocator\nfragmentation, CUDA\u002FNCCL workspace, CUDA graph buffers, token imbalance, or\ndispatcher workspace, so validate final configs with runtime memory metrics.\n\n## Quick Decision\n\nWhen a training run OOMs or is close to the memory limit:\n\n1. **Set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` first.** This fixes\n   fragmentation-induced OOM with zero performance cost. Most Slurm launch\n   templates already include it.\n2. **Add selective activation recompute** (`recompute_modules=[core_attn]`) if\n   not already enabled. See @skills\u002Fnemo-mbridge-perf-activation-recompute\u002FSKILL.md.\n3. **Avoid increasing TP** as a memory fix — doubling TP dramatically increases\n   NVLink all-reduce volume and often kills throughput (-28% on Llama3 70B).\n4. **Avoid increasing PP at the cost of DP** — halving DP doubles gradient\n   accumulation steps and hurts throughput (~6%).\n5. Consider `mlp` recompute if still OOM. Saves ~3 GB but costs ~16% GPU\n   utilization on large dense models (Llama3 70B).\n6. CPU offloading is **blocked when PP > 1**.\n\n## Enablement\n\n### Expandable segments (recommended first step)\n\nSet in the job's environment before launching:\n\n```bash\nexport PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True\n```\n\nIn Slurm scripts this is typically placed alongside other env vars:\n\n```bash\nexport CUDA_DEVICE_MAX_CONNECTIONS=1\nexport NVTE_ALLOW_NONDETERMINISTIC_ALGO=1\nexport PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True\n```\n\nNo model config changes needed. Zero throughput cost.\n\n### Parallelism resizing\n\nIf the model genuinely does not fit (not fragmentation), adjust parallelism:\n\n| Strategy | Memory effect | Throughput cost | Notes |\n|---|---|---|---|\n| Increase PP (keeping DP) | Fewer layers per stage | Moderate (~6% if DP halved) | Only if GPU count allows |\n| Increase TP | Fewer params per GPU | Severe (-28% on 70B) | Last resort |\n| Distributed optimizer | Shards optimizer state across DP ranks | ~1-2% | Recommended for large models |\n| FSDP | Shards params + grads + optimizer | Varies | See @skills\u002Fnemo-mbridge-perf-megatron-fsdp\u002FSKILL.md |\n\n### Activation recompute\n\nSee @skills\u002Fnemo-mbridge-perf-activation-recompute\u002FSKILL.md for full details.\n\n### CPU offloading\n\n```python\ncfg.model.cpu_offloading = True\n```\n\n**Incompatible with PP > 1.** Only usable when `pipeline_model_parallel_size = 1`.\n\n## A Note on VPP\n\nVirtual pipeline parallelism (VPP) is primarily a **throughput** optimization\nthat reduces pipeline bubble overhead by interleaving smaller model chunks. Its\neffect on peak memory is minimal — changing VPP does not meaningfully change\nthe total activation, parameter, or optimizer memory on a GPU.\n\nIn earlier experiments we incorrectly attributed an OOM fix to VPP tuning\n(VPP 5→10). The actual fix was `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`\nwhich eliminated memory fragmentation. The VPP=10 run actually used slightly\n**more** peak memory (60.2 GB vs 58.8 GB) but did not OOM because expandable\nsegments prevented fragmentation.\n\nVPP should be tuned for pipeline bubble reduction (see @docs\u002Fparallelisms.md),\nnot as a memory fix.\n\n## Compatibility and Constraints\n\n- `expandable_segments:True` is incompatible with `--use-nccl-ub` (NCCL\n  user-buffer registration). See Megatron-FSDP docs.\n- When using CUDA graphs with `expandable_segments:True`, set\n  `NCCL_GRAPH_REGISTER=0` (required on pre-Blackwell GPUs, enforced by MCore\n  `CudaGraphManager`).\n- CPU offloading requires `pipeline_model_parallel_size = 1`.\n- Distributed optimizer requires `use_distributed_optimizer = True` in the\n  optimizer config.\n\n## Measured Results\n\nLlama3 70B SFT on 32x H100 80GB, FP8 (Current Scaling):\n- Baseline: TP=4, PP=4, VPP=5, DP=2, MBS=1, GBS=32, seq_len=4096\n- Golden GPU utilization: 709.93 TFLOP\u002Fs\u002FGPU\n- Regression threshold: 5%\n\n### Strategy comparison: parallelism changes for memory reduction\n\n| Experiment | TP | PP | VPP | DP | TFLOP\u002Fs\u002FGPU | vs Golden | Peak Mem (GB) | Result |\n|---|---|---|---|---|---|---|---|---|\n| Baseline | 4 | 4 | 5 | 2 | ~704 | -0.8% | 58.8 | OOM (fragmentation) |\n| More PP | 4 | 8 | 5 | 1 | 668.0 | -5.9% | 53.2 | Borderline perf |\n| More TP | 8 | 4 | 5 | 1 | 508.7 | -28.4% | 50.2 | Severe regression |\n| Baseline + expandable_segments | 4 | 4 | 5 | 2 | ~704 | -0.8% | ~59 | **Passed** |\n\nKey takeaways:\n\n- **`expandable_segments:True` is the winner.** The baseline OOM was caused by\n  memory fragmentation, not insufficient capacity. Setting this env var\n  eliminated the OOM with zero throughput cost and no parallelism changes.\n- **PP=8 works for memory but loses DP** (2→1), meaning 32 gradient accumulation\n  steps per batch, which hurts throughput by ~6%.\n- **TP=8 is catastrophic** (-28%) because doubling TP increases all-reduce\n  communication volume proportionally across NVLink, and DP=1 means no\n  micro-batch overlap.\n\n### CPU offloading: blocked\n\n| Experiment | offload_layers | Result |\n|---|---|---|\n| Exp 4 | 2 | Incompatible (PP > 1) |\n| Exp 5 | 4 | Incompatible (PP > 1) |\n| Exp 6 | 6 | Incompatible (PP > 1) |\n\n`ValueError: Currently there is no support for Pipeline parallelism with CPU\noffloading.` This approach is blocked for any model using PP > 1.\n\n### Activation recompute: expensive alternative\n\nSelective activation recompute with `mlp` saved ~3 GB peak memory but cost\n~16% GPU utilization on this workload. See\n@skills\u002Fnemo-mbridge-perf-activation-recompute\u002FSKILL.md for full results.\n\n## Code Anchors\n\n### CPU offloading PP incompatibility (MCore)\n\n```1303:1306:3rdparty\u002FMegatron-LM\u002Fmegatron\u002Fcore\u002Ftransformer\u002Ftransformer_config.py\n        if self.cpu_offloading and self.pipeline_model_parallel_size > 1:\n            raise ValueError(\n                \"Currently there is no support for Pipeline parallelism with CPU offloading\"\n            )\n```\n\n### VPP config and layer divisibility validation (MCore)\n\n```1581:1592:3rdparty\u002FMegatron-LM\u002Fmegatron\u002Fcore\u002Ftransformer\u002Ftransformer_config.py\n            if pipeline_parallel_size and self.virtual_pipeline_model_parallel_size is not None:\n                num_layers_per_middle_pipeline_rank = num_layers \u002F\u002F pipeline_parallel_size\n                if (\n                    not num_layers_per_middle_pipeline_rank\n                    % self.virtual_pipeline_model_parallel_size\n                    == 0\n                ):\n                    raise ValueError(\n                        f\"number of layers on each middle pipeline rank:\"\n                        f\"{num_layers_per_middle_pipeline_rank} must be divisible by virtual\"\n                        f\"pipeline parallel degree {self.virtual_pipeline_model_parallel_size}\"\n                    )\n```\n\n### Parallelism docs on interleaved pipeline schedule\n\n```116:124:docs\u002Fparallelisms.md\nTo minimize the pipeline bubble, the computation on each GPU can be divided into multiple subsets of layers (referred to as model chunks), rather than a single contiguous block. Enable this by setting `virtual_pipeline_model_parallel_size`:\n\nmodel_config = GPTModelProvider(\n    pipeline_model_parallel_size=4,\n    virtual_pipeline_model_parallel_size=2,  # 2 model chunks per pipeline stage\n    # ... other model parameters\n)\n```\n\n## Failure Diagnosis\n\n| Symptom | Cause | Confirm | Fix |\n|---|---|---|---|\n| OOM on a single rank despite headroom on others | Memory fragmentation | check if `expandable_segments:True` is set | set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` |\n| OOM with `expandable_segments` already set | Genuine capacity limit | check `nvidia-smi` for param\u002Foptimizer memory | increase PP, use distributed optimizer, or add recompute |\n| Estimated memory exceeds GPU capacity before launch | model state or activations genuinely too large | run `estimate_training_memory` and inspect the largest component | adjust PP\u002FTP\u002FCP\u002FEP, distributed optimizer, or recompute before launching |\n| `ValueError: PP + CPU offloading` | using cpu_offloading with PP > 1 | check PP config | disable CPU offloading or set PP=1 |\n| `RuntimeError` with `--use-nccl-ub` + expandable segments | NCCL UB incompatible with expandable allocator | check env vars | remove `expandable_segments:True` or disable `--use-nccl-ub` |\n\n## Known Limitations\n\n- CPU offloading is blocked when PP > 1\n- Parallelism resizing (TP\u002FPP) often has significant throughput costs\n- The theoretical estimator is formula-based and does not replace runtime\n  profiling or CUDA memory reports\n\n## Verification\n\nQuick check that `expandable_segments:True` is active:\n\n```python\nimport os\nassert \"expandable_segments:True\" in os.environ.get(\"PYTORCH_CUDA_ALLOC_CONF\", \"\")\n```\n\nFor Slurm jobs, verify the env var is exported before the training command\nin the launch script.\n","# Memory Tuning\n\nStable docs: @docs\u002Fparallelisms.md\nCard: @skills\u002Fnemo-mbridge-perf-memory-tuning\u002Fcard.yaml\n\n## What It Is\n\nGPU OOM failures during training often stem from memory **fragmentation** rather\nthan raw capacity.  PyTorch's default CUDA allocator can leave unusable gaps\nbetween allocations.  The single most effective fix is:\n\n```bash\nexport PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True\n```\n\nThis tells PyTorch to use expandable (non-fixed-size) memory segments, which\ndramatically reduces fragmentation and often eliminates borderline OOM without\nany model or parallelism changes.\n\nBeyond fragmentation, actual peak memory is determined by:\n\n- **Parameter + optimizer state memory** — controlled by TP, PP, DP sharding\n  (distributed optimizer, FSDP)\n- **Activation memory** — controlled by activation recompute, sequence length,\n  micro-batch size\n- **Temporary \u002F workspace memory** — CUDA kernels, NCCL buffers, CUDA graphs\n\nFor configuration planning, use the Bridge theoretical estimator before launching\nlarge jobs:\n\n```python\nfrom megatron.bridge.training.utils.theoretical_memory_utils import estimate_training_memory\n\nestimate = estimate_training_memory(cfg, num_microbatches=num_microbatches)\n```\n\nThe estimator reports the most-loaded GPU shard and separates dense\u002Fembedding,\nrouted MoE expert, and activation components. It does not include allocator\nfragmentation, CUDA\u002FNCCL workspace, CUDA graph buffers, token imbalance, or\ndispatcher workspace, so validate final configs with runtime memory metrics.\n\n## Quick Decision\n\nWhen a training run OOMs or is close to the memory limit:\n\n1. **Set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` first.** This fixes\n   fragmentation-induced OOM with zero performance cost. Most Slurm launch\n   templates already include it.\n2. **Add selective activation recompute** (`recompute_modules=[core_attn]`) if\n   not already enabled. See @skills\u002Fnemo-mbridge-perf-activation-recompute\u002FSKILL.md.\n3. **Avoid increasing TP** as a memory fix — doubling TP dramatically increases\n   NVLink all-reduce volume and often kills throughput (-28% on Llama3 70B).\n4. **Avoid increasing PP at the cost of DP** — halving DP doubles gradient\n   accumulation steps and hurts throughput (~6%).\n5. Consider `mlp` recompute if still OOM. Saves ~3 GB but costs ~16% GPU\n   utilization on large dense models (Llama3 70B).\n6. CPU offloading is **blocked when PP > 1**.\n\n## Enablement\n\n### Expandable segments (recommended first step)\n\nSet in the job's environment before launching:\n\n```bash\nexport PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True\n```\n\nIn Slurm scripts this is typically placed alongside other env vars:\n\n```bash\nexport CUDA_DEVICE_MAX_CONNECTIONS=1\nexport NVTE_ALLOW_NONDETERMINISTIC_ALGO=1\nexport PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True\n```\n\nNo model config changes needed. Zero throughput cost.\n\n### Parallelism resizing\n\nIf the model genuinely does not fit (not fragmentation), adjust parallelism:\n\n| Strategy | Memory effect | Throughput cost | Notes |\n|---|---|---|---|\n| Increase PP (keeping DP) | Fewer layers per stage | Moderate (~6% if DP halved) | Only if GPU count allows |\n| Increase TP | Fewer params per GPU | Severe (-28% on 70B) | Last resort |\n| Distributed optimizer | Shards optimizer state across DP ranks | ~1-2% | Recommended for large models |\n| FSDP | Shards params + grads + optimizer | Varies | See @skills\u002Fnemo-mbridge-perf-megatron-fsdp\u002FSKILL.md |\n\n### Activation recompute\n\nSee @skills\u002Fnemo-mbridge-perf-activation-recompute\u002FSKILL.md for full details.\n\n### CPU offloading\n\n```python\ncfg.model.cpu_offloading = True\n```\n\n**Incompatible with PP > 1.** Only usable when `pipeline_model_parallel_size = 1`.\n\n## A Note on VPP\n\nVirtual pipeline parallelism (VPP) is primarily a **throughput** optimization\nthat reduces pipeline bubble overhead by interleaving smaller model chunks. Its\neffect on peak memory is minimal — changing VPP does not meaningfully change\nthe total activation, parameter, or optimizer memory on a GPU.\n\nIn earlier experiments we incorrectly attributed an OOM fix to VPP tuning\n(VPP 5→10). The actual fix was `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`\nwhich eliminated memory fragmentation. The VPP=10 run actually used slightly\n**more** peak memory (60.2 GB vs 58.8 GB) but did not OOM because expandable\nsegments prevented fragmentation.\n\nVPP should be tuned for pipeline bubble reduction (see @docs\u002Fparallelisms.md),\nnot as a memory fix.\n\n## Compatibility and Constraints\n\n- `expandable_segments:True` is incompatible with `--use-nccl-ub` (NCCL\n  user-buffer registration). See Megatron-FSDP docs.\n- When using CUDA graphs with `expandable_segments:True`, set\n  `NCCL_GRAPH_REGISTER=0` (required on pre-Blackwell GPUs, enforced by MCore\n  `CudaGraphManager`).\n- CPU offloading requires `pipeline_model_parallel_size = 1`.\n- Distributed optimizer requires `use_distributed_optimizer = True` in the\n  optimizer config.\n\n## Measured Results\n\nLlama3 70B SFT on 32x H100 80GB, FP8 (Current Scaling):\n- Baseline: TP=4, PP=4, VPP=5, DP=2, MBS=1, GBS=32, seq_len=4096\n- Golden GPU utilization: 709.93 TFLOP\u002Fs\u002FGPU\n- Regression threshold: 5%\n\n### Strategy comparison: parallelism changes for memory reduction\n\n| Experiment | TP | PP | VPP | DP | TFLOP\u002Fs\u002FGPU | vs Golden | Peak Mem (GB) | Result |\n|---|---|---|---|---|---|---|---|---|\n| Baseline | 4 | 4 | 5 | 2 | ~704 | -0.8% | 58.8 | OOM (fragmentation) |\n| More PP | 4 | 8 | 5 | 1 | 668.0 | -5.9% | 53.2 | Borderline perf |\n| More TP | 8 | 4 | 5 | 1 | 508.7 | -28.4% | 50.2 | Severe regression |\n| Baseline + expandable_segments | 4 | 4 | 5 | 2 | ~704 | -0.8% | ~59 | **Passed** |\n\nKey takeaways:\n\n- **`expandable_segments:True` is the winner.** The baseline OOM was caused by\n  memory fragmentation, not insufficient capacity. Setting this env var\n  eliminated the OOM with zero throughput cost and no parallelism changes.\n- **PP=8 works for memory but loses DP** (2→1), meaning 32 gradient accumulation\n  steps per batch, which hurts throughput by ~6%.\n- **TP=8 is catastrophic** (-28%) because doubling TP increases all-reduce\n  communication volume proportionally across NVLink, and DP=1 means no\n  micro-batch overlap.\n\n### CPU offloading: blocked\n\n| Experiment | offload_layers | Result |\n|---|---|---|\n| Exp 4 | 2 | Incompatible (PP > 1) |\n| Exp 5 | 4 | Incompatible (PP > 1) |\n| Exp 6 | 6 | Incompatible (PP > 1) |\n\n`ValueError: Currently there is no support for Pipeline parallelism with CPU\noffloading.` This approach is blocked for any model using PP > 1.\n\n### Activation recompute: expensive alternative\n\nSelective activation recompute with `mlp` saved ~3 GB peak memory but cost\n~16% GPU utilization on this workload. See\n@skills\u002Fnemo-mbridge-perf-activation-recompute\u002FSKILL.md for full results.\n\n## Code Anchors\n\n### CPU offloading PP incompatibility (MCore)\n\n```1303:1306:3rdparty\u002FMegatron-LM\u002Fmegatron\u002Fcore\u002Ftransformer\u002Ftransformer_config.py\n        if self.cpu_offloading and self.pipeline_model_parallel_size > 1:\n            raise ValueError(\n                \"Currently there is no support for Pipeline parallelism with CPU offloading\"\n            )\n```\n\n### VPP config and layer divisibility validation (MCore)\n\n```1581:1592:3rdparty\u002FMegatron-LM\u002Fmegatron\u002Fcore\u002Ftransformer\u002Ftransformer_config.py\n            if pipeline_parallel_size and self.virtual_pipeline_model_parallel_size is not None:\n                num_layers_per_middle_pipeline_rank = num_layers \u002F\u002F pipeline_parallel_size\n                if (\n                    not num_layers_per_middle_pipeline_rank\n                    % self.virtual_pipeline_model_parallel_size\n                    == 0\n                ):\n                    raise ValueError(\n                        f\"number of layers on each middle pipeline rank:\"\n                        f\"{num_layers_per_middle_pipeline_rank} must be divisible by virtual\"\n                        f\"pipeline parallel degree {self.virtual_pipeline_model_parallel_size}\"\n                    )\n```\n\n### Parallelism docs on interleaved pipeline schedule\n\n```116:124:docs\u002Fparallelisms.md\nTo minimize the pipeline bubble, the computation on each GPU can be divided into multiple subsets of layers (referred to as model chunks), rather than a single contiguous block. Enable this by setting `virtual_pipeline_model_parallel_size`:\n\nmodel_config = GPTModelProvider(\n    pipeline_model_parallel_size=4,\n    virtual_pipeline_model_parallel_size=2,  # 2 model chunks per pipeline stage\n    # ... other model parameters\n)\n```\n\n## Failure Diagnosis\n\n| Symptom | Cause | Confirm | Fix |\n|---|---|---|---|\n| OOM on a single rank despite headroom on others | Memory fragmentation | check if `expandable_segments:True` is set | set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` |\n| OOM with `expandable_segments` already set | Genuine capacity limit | check `nvidia-smi` for param\u002Foptimizer memory | increase PP, use distributed optimizer, or add recompute |\n| Estimated memory exceeds GPU capacity before launch | model state or activations genuinely too large | run `estimate_training_memory` and inspect the largest component | adjust PP\u002FTP\u002FCP\u002FEP, distributed optimizer, or recompute before launching |\n| `ValueError: PP + CPU offloading` | using cpu_offloading with PP > 1 | check PP config | disable CPU offloading or set PP=1 |\n| `RuntimeError` with `--use-nccl-ub` + expandable segments | NCCL UB incompatible with expandable allocator | check env vars | remove `expandable_segments:True` or disable `--use-nccl-ub` |\n\n## Known Limitations\n\n- CPU offloading is blocked when PP > 1\n- Parallelism resizing (TP\u002FPP) often has significant throughput costs\n- The theoretical estimator is formula-based and does not replace runtime\n  profiling or CUDA memory reports\n\n## Verification\n\nQuick check that `expandable_segments:True` is active:\n\n```python\nimport os\nassert \"expandable_segments:True\" in os.environ.get(\"PYTORCH_CUDA_ALLOC_CONF\", \"\")\n```\n\nFor Slurm jobs, verify the env var is exported before the training command\nin the launch script.\n",{"name":5,"license":15,"description":7,"when_to_use":44},"GPU OOM errors, reducing peak memory, or tracing an OOM regression to a specific commit or config change; 'out of memory', 'OOM', 'memory fragmentation', 'expandable_segments', 'reduce GPU memory', 'PYTORCH_CUDA_ALLOC_CONF'.","main","7a940b8e5af16c95e5c190a524421d32fc7445a9",".claude\u002Fskills\u002Fnemo-mbridge-perf-memory-tuning","https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp","https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui","https:\u002F\u002Fneuroads.com.br","2026-08-12 00:42:44",{"source":8,"skillName":5,"commands":53,"prompt":63,"zipAvailable":64},[54,57,60],{"tool":55,"command":56},"npx","npx skills add headmktlucca-ui\u002Fneuroads-lp --skill nemo-mbridge-perf-memory-tuning",{"tool":58,"command":59},"bunx","bunx skills add headmktlucca-ui\u002Fneuroads-lp --skill nemo-mbridge-perf-memory-tuning",{"tool":61,"command":62},"pnpm","pnpm dlx skills add headmktlucca-ui\u002Fneuroads-lp --skill nemo-mbridge-perf-memory-tuning","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fnemo-mbridge-perf-memory-tuning\u002FSKILL.md",true,[66,72,76,80,84],{"path":67,"resourceType":68,"resourceTypeDesc":69,"sizeBytes":70,"sourceHtmlUrl":71},"BENCHMARK.md","REFERENCE","参考文档",4331,"https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fnemo-mbridge-perf-memory-tuning\u002FBENCHMARK.md",{"path":73,"resourceType":68,"resourceTypeDesc":69,"sizeBytes":74,"sourceHtmlUrl":75},"card.yaml",6902,"https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fnemo-mbridge-perf-memory-tuning\u002Fcard.yaml",{"path":77,"resourceType":68,"resourceTypeDesc":69,"sizeBytes":78,"sourceHtmlUrl":79},"evals\u002Fevals.json",1474,"https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fnemo-mbridge-perf-memory-tuning\u002Fevals\u002Fevals.json",{"path":81,"resourceType":68,"resourceTypeDesc":69,"sizeBytes":82,"sourceHtmlUrl":83},"skill-card.md",3882,"https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fnemo-mbridge-perf-memory-tuning\u002Fskill-card.md",{"path":85,"resourceType":86,"resourceTypeDesc":87,"sizeBytes":88,"sourceHtmlUrl":89},"skill.oms.sig","OTHER","其他",4825,"https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fnemo-mbridge-perf-memory-tuning\u002Fskill.oms.sig",[91,118,142,168,186,221],{"skillNo":92,"name":93,"nameZh":6,"description":94,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":95,"licenseFileUrl":6,"sourceHtmlUrl":96,"categories":97,"keywords":106,"publishedAt":39},"01M6PTDCZTXTGSTX4RFK901YSZ","xunit-skill","Generates xUnit.net tests in C#. Covers Fact\u002FTheory, constructor injection, IClassFixture, and FluentAssertions. Use when user mentions \"xUnit\", \"[Fact]\", \"[Theory]\", \"Assert.Equal\", \"C# xUnit\". Triggers on: \"xUnit\", \"[Fact]\", \"[Theory]\", \"Assert.Equal C#\", \"xUnit.net\".","MIT","https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fxunit-skill\u002FSKILL.md",[98,103],{"categoryNo":99,"categoryName":100,"taxonomyType":101,"taxonomyTypeDesc":102},"01JBSEEDCAT000000000000008","测试与质量","SCENARIO","使用场景",{"categoryNo":104,"categoryName":105,"taxonomyType":101,"taxonomyTypeDesc":102},"01JBSEEDCAT000000000000023","Skill与智能体开发",[107,108,109,110,111,112,113,114,115,116,117],"xunit","xunit skill","testing","injection","user","use","tests","triggers","unit","unit-testing","C#",{"skillNo":119,"name":120,"nameZh":6,"description":121,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":95,"licenseFileUrl":6,"sourceHtmlUrl":122,"categories":123,"keywords":126,"publishedAt":39},"01M6PTCB59ZTG5FTMWGPYEMMBP","xcuitest-skill","Generates XCUITest UI tests for iOS\u002FiPadOS apps in Swift. Apple's native testing framework for reliable, fast UI automation. Supports local simulators and TestMu AI cloud real devices. Use when user mentions \"XCUITest\", \"XCTest\", \"iOS UI test\", \"Swift test\", \"XCUIApplication\". Triggers on: \"XCUITest\", \"XCTest UI\", \"iOS UI test\", \"Swift UI test\", \"XCUIApplication\", \"TestMu\".","https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fxcuitest-skill\u002FSKILL.md",[124,125],{"categoryNo":99,"categoryName":100,"taxonomyType":101,"taxonomyTypeDesc":102},{"categoryNo":104,"categoryName":105,"taxonomyType":101,"taxonomyTypeDesc":102},[127,128,109,129,130,131,132,133,112,134,135,136,137,138,113,114,139,140,27,141],"xcuitest","xcuitest skill","automation","test","testing framework","ios","devices","mobile","mobile-testing","ui test","local","apple","apps","native","objective",{"skillNo":143,"name":144,"nameZh":6,"description":145,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":95,"licenseFileUrl":6,"sourceHtmlUrl":146,"categories":147,"keywords":152,"publishedAt":39},"01M6PTATMNTY9NF4J7KSR76YK6","world-builder","Create and manage generative game zones for Dorothy's Pokemon-style world. Use this skill when creating, updating, or designing game worlds via MCP tools.","https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fworld-builder\u002FSKILL.md",[148,151],{"categoryNo":149,"categoryName":150,"taxonomyType":101,"taxonomyTypeDesc":102},"01JBSEEDCAT000000000000015","视觉设计",{"categoryNo":104,"categoryName":105,"taxonomyType":101,"taxonomyTypeDesc":102},[153,154,155,156,157,158,159,160,112,161,162,163,164,165,166,27,167],"builder","world","world builder","MCP","create","mcp tools","Tools","game","pokemon","manage","style","via","theme","zone","generative",{"skillNo":169,"name":170,"nameZh":6,"description":171,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":95,"licenseFileUrl":6,"sourceHtmlUrl":172,"categories":173,"keywords":176,"publishedAt":39},"01M6PT9Z92JTSQEHE1QR0NNW5F","webdriverio-skill","Generates WebdriverIO (WDIO) automation tests in JavaScript or TypeScript. Supports local and TestMu AI cloud. Use when user mentions \"WebdriverIO\", \"WDIO\", \"wdio.conf\", \"browser.url\", \"$\", \"$$\". Triggers on: \"WebdriverIO\", \"WDIO\", \"wdio\", \"browser.$\".","https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fwebdriverio-skill\u002FSKILL.md",[174,175],{"categoryNo":99,"categoryName":100,"taxonomyType":101,"taxonomyTypeDesc":102},{"categoryNo":104,"categoryName":105,"taxonomyType":101,"taxonomyTypeDesc":102},[177,109,178,129,179,180,181,182,111,112,183,137,113,114,184,185],"webdriverio skill","typescript","javascript","browser","cloud","e2e","e2e-testing","URL","conf",{"skillNo":187,"name":188,"nameZh":6,"description":189,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":95,"licenseFileUrl":6,"sourceHtmlUrl":190,"categories":191,"keywords":201,"publishedAt":39},"01M6PT9NVK36ZWCQ69YXK3FR4K","web-design-guidelines","Web platform design and accessibility guidelines. Use when building web interfaces, auditing accessibility, implementing responsive layouts, or reviewing web UI code. Triggers on tasks involving HTML, CSS, web components, WCAG compliance, responsive design, or web performance.","https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fweb\u002FSKILL.md",[192,195,196],{"categoryNo":193,"categoryName":194,"taxonomyType":101,"taxonomyTypeDesc":102},"01JBSEEDCAT000000000000009","前端开发",{"categoryNo":149,"categoryName":150,"taxonomyType":101,"taxonomyTypeDesc":102},{"categoryNo":197,"categoryName":198,"taxonomyType":199,"taxonomyTypeDesc":200},"01JBSEEDCAT000000000000032","设计师","PROFESSION","适用职业",[202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,114,220],"design","web","guidelines","web design guidelines","web design","css","performance","components","web components","tasks","code","web performance","ui","platform design","html","responsive design","responsive","platform","implementing",{"skillNo":222,"name":223,"nameZh":6,"description":224,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":95,"licenseFileUrl":6,"sourceHtmlUrl":225,"categories":226,"keywords":229,"publishedAt":39},"01M6PT9A3JKRE3X34276F2FW5R","watchos-design-guidelines","Apple Human Interface Guidelines for Apple Watch. Use when building watchOS apps, complications, or workout features. Triggers on tasks involving Watch UI, Digital Crown, glanceable interfaces, or wrist-based interactions.","https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fwatchos\u002FSKILL.md",[227,228],{"categoryNo":149,"categoryName":150,"taxonomyType":101,"taxonomyTypeDesc":102},{"categoryNo":197,"categoryName":198,"taxonomyType":199,"taxonomyTypeDesc":200},[202,204,230,211,214,112,231,232,138,219,233,114,139,234,27,235,236],"watchos design guidelines","workout","human","watch","interface","platform-design","features"]