[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6PQHGXTB3ENCDJMAFDHH6YX":3,"skill-same:01M6PQHGXTB3ENCDJMAFDHH6YX":90},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":45,"commitSha":46,"skillDirectory":47,"forksCount":14,"repositoryHtmlUrl":48,"repositoryHomepageUrl":6,"ownerHtmlUrl":49,"ownerBlogUrl":50,"ownerDescription":6,"updatedAt":51,"install":52,"resources":65},"01M6PQHGXTB3ENCDJMAFDHH6YX","nemo-mbridge-resiliency",null,"Resiliency features in Megatron Bridge including fault tolerance, straggler detection, in-process restart, preemption, and re-run state machine.","headmktlucca-ui\u002Fneuroads-lp","headmktlucca-ui","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F247379838?v=4","01M6PDVAV6M1W96D1SR1PDF8CS","NeuroAds","01M6PDVAZ4P281GNDKD45CYTF5",0,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fnemo-mbridge-resiliency\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"nemo mbridge resiliency","NeMo","restart","automatic","features","state machine","tolerance","re","megatron","machine","fault tolerance","commit","process","bridge","broken","detection","NVIDIA","state","run","training","2026-11-10 10:12:30",false,"---\nname: nemo-mbridge-resiliency\ndescription: Resiliency features in Megatron Bridge including fault tolerance, straggler detection, in-process restart, preemption, and re-run state machine.\nlicense: Apache-2.0\nwhen_to_use: Enabling resiliency features, or investigating a commit that caused training hangs, straggler detection failures, or broken restarts; 'fault tolerance', 'straggler detection', 'hang detection', 'automatic restart', 'in-process restart', 'preemption', 'nvidia-resiliency-ext'.\n---\n\n# Resiliency\n\nStable docs: @docs\u002Ftraining\u002Fresiliency.md, @docs\u002Ftraining\u002Fcheckpointing.md\nCard: @skills\u002Fnemo-mbridge-resiliency\u002Fcard.yaml\n\n## Enablement\n\n### Fault tolerance (Slurm only)\n\n#### Option 1: NeMo Run plugin (recommended)\n\n```python\nfrom megatron.bridge.recipes.run_plugins import FaultTolerancePlugin\nimport nemo_run as run\n\ntask = run.Script(...)\nrun_plugins = [\n    FaultTolerancePlugin(\n        enable_ft_package=True,\n        calc_ft_timeouts=True,\n        num_in_job_restarts=3,\n        num_job_retries_on_failure=2,\n        initial_rank_heartbeat_timeout=1800,\n        rank_heartbeat_timeout=300,\n    )\n]\nrun.run(task, plugins=run_plugins, executor=executor)\n```\n\n| Plugin parameter | Default | Description |\n|---|---|---|\n| `num_in_job_restarts` | 3 | Max restarts within same job |\n| `num_job_retries_on_failure` | 2 | Max new job launches on failure |\n| `initial_rank_heartbeat_timeout` | 1800 | First heartbeat timeout (seconds) |\n| `rank_heartbeat_timeout` | 300 | Subsequent heartbeat timeout (seconds) |\n\n#### Option 2: Direct config + ft_launcher\n\n```python\nfrom megatron.bridge.training.config import FaultToleranceConfig\n\ncfg.ft = FaultToleranceConfig(\n    enable_ft_package=True,\n    calc_ft_timeouts=True,\n    simulate_fault=False,\n    simulated_fault_type=\"random\",\n)\n```\n\nLaunch with `ft_launcher` (not `torchrun`):\n\n```bash\nexport GROUP_RANK=0  # required for non-Slurm\nft_launcher \\\n    --rdzv_backend=c10d --rdzv_endpoint=${MASTER_ADDR}:${MASTER_PORT} \\\n    --nnodes=${NUM_NODES} --nproc-per-node=${NUM_GPUS_PER_NODE} \\\n    --ft-rank_section_timeouts=setup:600,step:180,checkpointing:420 \\\n    --ft-rank_out_of_section_timeout=300 \\\n    your_training_script.py\n```\n\n| Config parameter | Default | Description |\n|---|---|---|\n| `enable_ft_package` | False | Enable fault tolerance |\n| `calc_ft_timeouts` | False | Auto-compute optimal timeouts |\n| `simulate_fault` | False | Enable fault simulation for testing |\n| `simulated_fault_type` | `\"random\"` | `\"rank_hung\"`, `\"rank_killed\"`, or `\"random\"` |\n| `simulated_fault_rank` | None | Specific rank to fault (random if None) |\n| `simulated_fault_base_delay` | 0 | Base delay before simulating fault |\n\nSection-based timeout monitoring covers setup, training steps, checkpointing,\nand out-of-section time independently. Timeouts are saved to `ft_state.json`\nfor subsequent runs when `calc_ft_timeouts=True`.\n\n### NVRx straggler detection\n\n```python\nfrom megatron.bridge.training.config import NVRxStragglerDetectionConfig\n\ncfg.nvrx_straggler = NVRxStragglerDetectionConfig(\n    enabled=True,\n    report_time_interval=300.0,\n    calc_relative_gpu_perf=True,\n    calc_individual_gpu_perf=True,\n    num_gpu_perf_scores_to_print=5,\n    gpu_relative_perf_threshold=0.7,\n    gpu_individual_perf_threshold=0.7,\n    stop_if_detected=False,\n    enable_logging=True,\n)\n```\n\n| Parameter | Default | Description |\n|---|---|---|\n| `enabled` | False | Enable straggler detection |\n| `report_time_interval` | 300.0 | Seconds between straggler checks |\n| `calc_relative_gpu_perf` | True | Compare ranks against each other |\n| `calc_individual_gpu_perf` | True | Track per-rank degradation over time |\n| `gpu_relative_perf_threshold` | 0.7 | Threshold for relative performance (0-1) |\n| `gpu_individual_perf_threshold` | 0.7 | Threshold for individual performance (0-1) |\n| `stop_if_detected` | False | Terminate training on straggler |\n| `num_gpu_perf_scores_to_print` | 5 | Number of best\u002Fworst scores to print |\n| `profiling_interval` | 1 | Profiling interval for detector |\n\n### Preemption\n\n#### Plugin (Slurm)\n\n```python\nfrom megatron.bridge.recipes.run_plugins import PreemptionPlugin\n\nplugins = [\n    PreemptionPlugin(\n        preempt_time=60,\n        enable_exit_handler=True,\n        enable_exit_handler_for_data_loader=False,\n    )\n]\n```\n\n| Plugin parameter | Default | Description |\n|---|---|---|\n| `preempt_time` | 60 | Seconds before job limit to send signal |\n| `enable_exit_handler` | True | Enable signal handler in training |\n| `enable_exit_handler_for_data_loader` | False | Enable for dataloader workers |\n\n#### Direct config\n\n```python\nimport signal\ncfg.train.exit_signal_handler = True\ncfg.train.exit_signal = signal.SIGTERM\ncfg.train.exit_signal_handler_for_dataloader = False\n```\n\n### Re-run state machine (experimental)\n\n```python\nfrom megatron.bridge.training.config import RerunStateMachineConfig\n\ncfg.rerun_state_machine = RerunStateMachineConfig(\n    rerun_mode=\"validate_results\",\n    check_for_nan_in_loss=True,\n    check_for_spiky_loss=False,\n    spiky_loss_factor=10.0,\n)\n```\n\n| Parameter | Default | Description |\n|---|---|---|\n| `rerun_mode` | `\"disabled\"` | `\"disabled\"`, `\"validate_results\"`, `\"report_determinism_stats\"` |\n| `check_for_nan_in_loss` | True | Check for NaN in loss |\n| `check_for_spiky_loss` | False | Check for unexpectedly large loss |\n| `spiky_loss_factor` | 10.0 | Loss flagged if > factor * max observed (increase for large models) |\n\nExit codes: 16 = resume to disambiguate, 17 = failed validation.\n\n### In-process restart (experimental)\n\n```python\nfrom megatron.bridge.training.config import InProcessRestartConfig\n\ncfg.inprocess_restart = InProcessRestartConfig(\n    enabled=True,\n    granularity=\"node\",\n    soft_timeout=60.0,\n    hard_timeout=90.0,\n)\n```\n\n| Parameter | Default | Description |\n|---|---|---|\n| `enabled` | False | Enable in-process restart |\n| `active_world_size` | None | Ranks executing workload (rest are warm reserves) |\n| `granularity` | `\"node\"` | `\"node\"` or `\"rank\"` restart granularity |\n| `max_iterations` | None | Max restart attempts (None = unlimited) |\n| `soft_timeout` | 60.0 | Detect GIL-released hangs (seconds) |\n| `hard_timeout` | 90.0 | Force-terminate hung ranks (seconds) |\n| `heartbeat_interval` | 30.0 | Heartbeat interval (seconds) |\n| `heartbeat_timeout` | 60.0 | Missing heartbeat timeout (seconds) |\n| `barrier_timeout` | 120.0 | Distributed barrier timeout (seconds) |\n| `completion_timeout` | 120.0 | Completion barrier timeout (seconds) |\n| `empty_cuda_cache` | True | Clear CUDA cache during restart |\n| `max_rank_faults` | None | Max rank faults before terminating |\n| `monitor_process_logdir` | None | Directory for monitor logs |\n\nRequired environment variables:\n\n```bash\nexport TORCH_CPP_LOG_LEVEL=error\nexport TORCH_NCCL_RETHROW_CUDA_ERRORS=0\nexport NCCL_NVLS_ENABLE=0\n```\n\nThe PyTorch NCCL watchdog timeout must exceed `hard_timeout`. NeMo-Run's\nSlurm Executor is not supported; launch directly with `srun --kill-on-bad-exit=0`.\n\n### Async checkpoint save\n\n```python\ncfg.checkpoint.async_save = True\ncfg.checkpoint.ckpt_format = \"torch_dist\"\n```\n\n### Local checkpointing (NVRx)\n\n```python\ncfg.checkpoint.non_persistent_local_ckpt_dir = \"\u002Flocal\u002Fscratch\u002Fckpt\"\ncfg.checkpoint.non_persistent_local_ckpt_algo = \"fully_parallel\"\n```\n\n## Code Anchors\n\n### Fault tolerance\n- Config: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fconfig.py` — `FaultToleranceConfig`\n- Runtime: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Ffault_tolerance.py`\n- Plugin: `src\u002Fmegatron\u002Fbridge\u002Frecipes\u002Frun_plugins.py` — `FaultTolerancePlugin`\n- Perf plugin: `scripts\u002Fperformance\u002Fnemo-mbridge-resiliency_plugins.py`\n- Tests: `tests\u002Funit_tests\u002Ftraining\u002Ftest_fault_tolerance.py`\n- Example: `examples\u002Ftraining_features\u002Fnemo-mbridge-resiliency\u002Ffault_tolerance\u002F`\n\n### Straggler detection\n- Config: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fconfig.py` — `NVRxStragglerDetectionConfig`\n- Runtime: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fnvrx_straggler.py`\n- Train loop: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Ftrain.py` — `check_nvrx_straggler_detection`\n- Tests: `tests\u002Funit_tests\u002Ftraining\u002Ftest_nvrx_straggler.py`, `tests\u002Ffunctional_tests\u002Ftraining\u002Ftest_nvrx_straggler.py`\n- Example: `examples\u002Ftraining_features\u002Fnemo-mbridge-resiliency\u002Fstraggler_detection\u002F`\n\n### In-process restart\n- Config: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fconfig.py` — `InProcessRestartConfig`\n- Runtime: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Finprocess_restart.py`\n- Entry point: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fpretrain.py` — `maybe_wrap_for_inprocess_restart`\n- Tests: `tests\u002Funit_tests\u002Ftraining\u002Ftest_inprocess_restart.py`, `tests\u002Ffunctional_tests\u002Ftraining\u002Ftest_inprocess_restart.py`\n\n### Preemption\n- Plugin: `src\u002Fmegatron\u002Fbridge\u002Frecipes\u002Frun_plugins.py` — `PreemptionPlugin`\n- Signal handler: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Futils\u002Fsig_utils.py`\n- Tests: `tests\u002Funit_tests\u002Frecipes\u002Ftest_run_plugins.py`\n\n### Re-run state machine\n- Config: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fconfig.py` — `RerunStateMachineConfig`\n- Init: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Finitialize.py` — `init_rerun_state`\n\n### Checkpointing\n- Async save: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fcheckpointing.py` — `schedule_async_save`\n- Local ckpt: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fcheckpointing.py` — `LocalCheckpointManager`\n- Tests: `tests\u002Ffunctional_tests\u002Ftraining\u002Ftest_local_checkpointing.py`\n\n## Pitfalls\n\n1. **ft_launcher, not torchrun**: Direct `FaultToleranceConfig` requires\n   `ft_launcher`. Using `torchrun` silently disables FT. For non-Slurm,\n   set `GROUP_RANK=0`.\n\n2. **Async save requires torch_dist**: `async_save=True` only works with\n   `ckpt_format=\"torch_dist\"`. Other formats silently fail or error.\n\n3. **IPR + NeMo-Run**: In-process restart is not compatible with NeMo-Run\n   or Slurm preemption plugins. Requires specific PyTorch\u002FNCCL versions\n   and env vars.\n\n4. **NVRx vs legacy straggler**: Two detectors exist. Use NVRx\n   (`nvrx_straggler`); do not enable both.\n\n5. **stop_if_detected default**: NVRx logs but does not stop training by\n   default. Set `stop_if_detected=True` for automatic termination.\n\n6. **NCCL watchdog vs hard_timeout**: For IPR, NCCL watchdog timeout must\n   exceed `hard_timeout` or PyTorch kills the process before recovery.\n\n7. **Rerun state machine is alpha**: Use `check_for_nan_in_loss=True` for\n   NaN detection, but don't rely on full rerun workflows yet.\n\n## Verification\n\n### Fault tolerance\n```bash\n.\u002Fexamples\u002Ftraining_features\u002Fnemo-mbridge-resiliency\u002Ffault_tolerance\u002Frun_fault_tolerance.sh\n.\u002Fexamples\u002Ftraining_features\u002Fnemo-mbridge-resiliency\u002Ffault_tolerance\u002Frun_fault_tolerance.sh --simulate-fault\n```\nLook for `[FaultTolerance]` \u002F `[RankMonitorServer]` log lines with section\ntimeouts. Simulated fault should trigger restart from checkpoint.\n\n### Straggler detection\n```bash\nuv run python -m torch.distributed.run --nproc_per_node=2 \\\n    examples\u002Ftraining_features\u002Fnemo-mbridge-resiliency\u002Fstraggler_detection\u002Fstraggler_detection_example.py\n```\nLook for `GPU relative performance` and `GPU individual performance` reports\nwith per-rank scores.\n\n### Async checkpoint\nLook for `Scheduling async checkpoint save` in logs. Training iterations\nshould continue while checkpoint files are being written.\n\n### In-process restart\n```bash\npytest tests\u002Ffunctional_tests\u002Ftraining\u002Ftest_inprocess_restart.py -v\n```\nRequires compatible PyTorch\u002FNCCL versions.\n","# Resiliency\n\nStable docs: @docs\u002Ftraining\u002Fresiliency.md, @docs\u002Ftraining\u002Fcheckpointing.md\nCard: @skills\u002Fnemo-mbridge-resiliency\u002Fcard.yaml\n\n## Enablement\n\n### Fault tolerance (Slurm only)\n\n#### Option 1: NeMo Run plugin (recommended)\n\n```python\nfrom megatron.bridge.recipes.run_plugins import FaultTolerancePlugin\nimport nemo_run as run\n\ntask = run.Script(...)\nrun_plugins = [\n    FaultTolerancePlugin(\n        enable_ft_package=True,\n        calc_ft_timeouts=True,\n        num_in_job_restarts=3,\n        num_job_retries_on_failure=2,\n        initial_rank_heartbeat_timeout=1800,\n        rank_heartbeat_timeout=300,\n    )\n]\nrun.run(task, plugins=run_plugins, executor=executor)\n```\n\n| Plugin parameter | Default | Description |\n|---|---|---|\n| `num_in_job_restarts` | 3 | Max restarts within same job |\n| `num_job_retries_on_failure` | 2 | Max new job launches on failure |\n| `initial_rank_heartbeat_timeout` | 1800 | First heartbeat timeout (seconds) |\n| `rank_heartbeat_timeout` | 300 | Subsequent heartbeat timeout (seconds) |\n\n#### Option 2: Direct config + ft_launcher\n\n```python\nfrom megatron.bridge.training.config import FaultToleranceConfig\n\ncfg.ft = FaultToleranceConfig(\n    enable_ft_package=True,\n    calc_ft_timeouts=True,\n    simulate_fault=False,\n    simulated_fault_type=\"random\",\n)\n```\n\nLaunch with `ft_launcher` (not `torchrun`):\n\n```bash\nexport GROUP_RANK=0  # required for non-Slurm\nft_launcher \\\n    --rdzv_backend=c10d --rdzv_endpoint=${MASTER_ADDR}:${MASTER_PORT} \\\n    --nnodes=${NUM_NODES} --nproc-per-node=${NUM_GPUS_PER_NODE} \\\n    --ft-rank_section_timeouts=setup:600,step:180,checkpointing:420 \\\n    --ft-rank_out_of_section_timeout=300 \\\n    your_training_script.py\n```\n\n| Config parameter | Default | Description |\n|---|---|---|\n| `enable_ft_package` | False | Enable fault tolerance |\n| `calc_ft_timeouts` | False | Auto-compute optimal timeouts |\n| `simulate_fault` | False | Enable fault simulation for testing |\n| `simulated_fault_type` | `\"random\"` | `\"rank_hung\"`, `\"rank_killed\"`, or `\"random\"` |\n| `simulated_fault_rank` | None | Specific rank to fault (random if None) |\n| `simulated_fault_base_delay` | 0 | Base delay before simulating fault |\n\nSection-based timeout monitoring covers setup, training steps, checkpointing,\nand out-of-section time independently. Timeouts are saved to `ft_state.json`\nfor subsequent runs when `calc_ft_timeouts=True`.\n\n### NVRx straggler detection\n\n```python\nfrom megatron.bridge.training.config import NVRxStragglerDetectionConfig\n\ncfg.nvrx_straggler = NVRxStragglerDetectionConfig(\n    enabled=True,\n    report_time_interval=300.0,\n    calc_relative_gpu_perf=True,\n    calc_individual_gpu_perf=True,\n    num_gpu_perf_scores_to_print=5,\n    gpu_relative_perf_threshold=0.7,\n    gpu_individual_perf_threshold=0.7,\n    stop_if_detected=False,\n    enable_logging=True,\n)\n```\n\n| Parameter | Default | Description |\n|---|---|---|\n| `enabled` | False | Enable straggler detection |\n| `report_time_interval` | 300.0 | Seconds between straggler checks |\n| `calc_relative_gpu_perf` | True | Compare ranks against each other |\n| `calc_individual_gpu_perf` | True | Track per-rank degradation over time |\n| `gpu_relative_perf_threshold` | 0.7 | Threshold for relative performance (0-1) |\n| `gpu_individual_perf_threshold` | 0.7 | Threshold for individual performance (0-1) |\n| `stop_if_detected` | False | Terminate training on straggler |\n| `num_gpu_perf_scores_to_print` | 5 | Number of best\u002Fworst scores to print |\n| `profiling_interval` | 1 | Profiling interval for detector |\n\n### Preemption\n\n#### Plugin (Slurm)\n\n```python\nfrom megatron.bridge.recipes.run_plugins import PreemptionPlugin\n\nplugins = [\n    PreemptionPlugin(\n        preempt_time=60,\n        enable_exit_handler=True,\n        enable_exit_handler_for_data_loader=False,\n    )\n]\n```\n\n| Plugin parameter | Default | Description |\n|---|---|---|\n| `preempt_time` | 60 | Seconds before job limit to send signal |\n| `enable_exit_handler` | True | Enable signal handler in training |\n| `enable_exit_handler_for_data_loader` | False | Enable for dataloader workers |\n\n#### Direct config\n\n```python\nimport signal\ncfg.train.exit_signal_handler = True\ncfg.train.exit_signal = signal.SIGTERM\ncfg.train.exit_signal_handler_for_dataloader = False\n```\n\n### Re-run state machine (experimental)\n\n```python\nfrom megatron.bridge.training.config import RerunStateMachineConfig\n\ncfg.rerun_state_machine = RerunStateMachineConfig(\n    rerun_mode=\"validate_results\",\n    check_for_nan_in_loss=True,\n    check_for_spiky_loss=False,\n    spiky_loss_factor=10.0,\n)\n```\n\n| Parameter | Default | Description |\n|---|---|---|\n| `rerun_mode` | `\"disabled\"` | `\"disabled\"`, `\"validate_results\"`, `\"report_determinism_stats\"` |\n| `check_for_nan_in_loss` | True | Check for NaN in loss |\n| `check_for_spiky_loss` | False | Check for unexpectedly large loss |\n| `spiky_loss_factor` | 10.0 | Loss flagged if > factor * max observed (increase for large models) |\n\nExit codes: 16 = resume to disambiguate, 17 = failed validation.\n\n### In-process restart (experimental)\n\n```python\nfrom megatron.bridge.training.config import InProcessRestartConfig\n\ncfg.inprocess_restart = InProcessRestartConfig(\n    enabled=True,\n    granularity=\"node\",\n    soft_timeout=60.0,\n    hard_timeout=90.0,\n)\n```\n\n| Parameter | Default | Description |\n|---|---|---|\n| `enabled` | False | Enable in-process restart |\n| `active_world_size` | None | Ranks executing workload (rest are warm reserves) |\n| `granularity` | `\"node\"` | `\"node\"` or `\"rank\"` restart granularity |\n| `max_iterations` | None | Max restart attempts (None = unlimited) |\n| `soft_timeout` | 60.0 | Detect GIL-released hangs (seconds) |\n| `hard_timeout` | 90.0 | Force-terminate hung ranks (seconds) |\n| `heartbeat_interval` | 30.0 | Heartbeat interval (seconds) |\n| `heartbeat_timeout` | 60.0 | Missing heartbeat timeout (seconds) |\n| `barrier_timeout` | 120.0 | Distributed barrier timeout (seconds) |\n| `completion_timeout` | 120.0 | Completion barrier timeout (seconds) |\n| `empty_cuda_cache` | True | Clear CUDA cache during restart |\n| `max_rank_faults` | None | Max rank faults before terminating |\n| `monitor_process_logdir` | None | Directory for monitor logs |\n\nRequired environment variables:\n\n```bash\nexport TORCH_CPP_LOG_LEVEL=error\nexport TORCH_NCCL_RETHROW_CUDA_ERRORS=0\nexport NCCL_NVLS_ENABLE=0\n```\n\nThe PyTorch NCCL watchdog timeout must exceed `hard_timeout`. NeMo-Run's\nSlurm Executor is not supported; launch directly with `srun --kill-on-bad-exit=0`.\n\n### Async checkpoint save\n\n```python\ncfg.checkpoint.async_save = True\ncfg.checkpoint.ckpt_format = \"torch_dist\"\n```\n\n### Local checkpointing (NVRx)\n\n```python\ncfg.checkpoint.non_persistent_local_ckpt_dir = \"\u002Flocal\u002Fscratch\u002Fckpt\"\ncfg.checkpoint.non_persistent_local_ckpt_algo = \"fully_parallel\"\n```\n\n## Code Anchors\n\n### Fault tolerance\n- Config: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fconfig.py` — `FaultToleranceConfig`\n- Runtime: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Ffault_tolerance.py`\n- Plugin: `src\u002Fmegatron\u002Fbridge\u002Frecipes\u002Frun_plugins.py` — `FaultTolerancePlugin`\n- Perf plugin: `scripts\u002Fperformance\u002Fnemo-mbridge-resiliency_plugins.py`\n- Tests: `tests\u002Funit_tests\u002Ftraining\u002Ftest_fault_tolerance.py`\n- Example: `examples\u002Ftraining_features\u002Fnemo-mbridge-resiliency\u002Ffault_tolerance\u002F`\n\n### Straggler detection\n- Config: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fconfig.py` — `NVRxStragglerDetectionConfig`\n- Runtime: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fnvrx_straggler.py`\n- Train loop: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Ftrain.py` — `check_nvrx_straggler_detection`\n- Tests: `tests\u002Funit_tests\u002Ftraining\u002Ftest_nvrx_straggler.py`, `tests\u002Ffunctional_tests\u002Ftraining\u002Ftest_nvrx_straggler.py`\n- Example: `examples\u002Ftraining_features\u002Fnemo-mbridge-resiliency\u002Fstraggler_detection\u002F`\n\n### In-process restart\n- Config: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fconfig.py` — `InProcessRestartConfig`\n- Runtime: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Finprocess_restart.py`\n- Entry point: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fpretrain.py` — `maybe_wrap_for_inprocess_restart`\n- Tests: `tests\u002Funit_tests\u002Ftraining\u002Ftest_inprocess_restart.py`, `tests\u002Ffunctional_tests\u002Ftraining\u002Ftest_inprocess_restart.py`\n\n### Preemption\n- Plugin: `src\u002Fmegatron\u002Fbridge\u002Frecipes\u002Frun_plugins.py` — `PreemptionPlugin`\n- Signal handler: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Futils\u002Fsig_utils.py`\n- Tests: `tests\u002Funit_tests\u002Frecipes\u002Ftest_run_plugins.py`\n\n### Re-run state machine\n- Config: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fconfig.py` — `RerunStateMachineConfig`\n- Init: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Finitialize.py` — `init_rerun_state`\n\n### Checkpointing\n- Async save: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fcheckpointing.py` — `schedule_async_save`\n- Local ckpt: `src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fcheckpointing.py` — `LocalCheckpointManager`\n- Tests: `tests\u002Ffunctional_tests\u002Ftraining\u002Ftest_local_checkpointing.py`\n\n## Pitfalls\n\n1. **ft_launcher, not torchrun**: Direct `FaultToleranceConfig` requires\n   `ft_launcher`. Using `torchrun` silently disables FT. For non-Slurm,\n   set `GROUP_RANK=0`.\n\n2. **Async save requires torch_dist**: `async_save=True` only works with\n   `ckpt_format=\"torch_dist\"`. Other formats silently fail or error.\n\n3. **IPR + NeMo-Run**: In-process restart is not compatible with NeMo-Run\n   or Slurm preemption plugins. Requires specific PyTorch\u002FNCCL versions\n   and env vars.\n\n4. **NVRx vs legacy straggler**: Two detectors exist. Use NVRx\n   (`nvrx_straggler`); do not enable both.\n\n5. **stop_if_detected default**: NVRx logs but does not stop training by\n   default. Set `stop_if_detected=True` for automatic termination.\n\n6. **NCCL watchdog vs hard_timeout**: For IPR, NCCL watchdog timeout must\n   exceed `hard_timeout` or PyTorch kills the process before recovery.\n\n7. **Rerun state machine is alpha**: Use `check_for_nan_in_loss=True` for\n   NaN detection, but don't rely on full rerun workflows yet.\n\n## Verification\n\n### Fault tolerance\n```bash\n.\u002Fexamples\u002Ftraining_features\u002Fnemo-mbridge-resiliency\u002Ffault_tolerance\u002Frun_fault_tolerance.sh\n.\u002Fexamples\u002Ftraining_features\u002Fnemo-mbridge-resiliency\u002Ffault_tolerance\u002Frun_fault_tolerance.sh --simulate-fault\n```\nLook for `[FaultTolerance]` \u002F `[RankMonitorServer]` log lines with section\ntimeouts. Simulated fault should trigger restart from checkpoint.\n\n### Straggler detection\n```bash\nuv run python -m torch.distributed.run --nproc_per_node=2 \\\n    examples\u002Ftraining_features\u002Fnemo-mbridge-resiliency\u002Fstraggler_detection\u002Fstraggler_detection_example.py\n```\nLook for `GPU relative performance` and `GPU individual performance` reports\nwith per-rank scores.\n\n### Async checkpoint\nLook for `Scheduling async checkpoint save` in logs. Training iterations\nshould continue while checkpoint files are being written.\n\n### In-process restart\n```bash\npytest tests\u002Ffunctional_tests\u002Ftraining\u002Ftest_inprocess_restart.py -v\n```\nRequires compatible PyTorch\u002FNCCL versions.\n",{"name":5,"license":15,"description":7,"when_to_use":44},"Enabling resiliency features, or investigating a commit that caused training hangs, straggler detection failures, or broken restarts; 'fault tolerance', 'straggler detection', 'hang detection', 'automatic restart', 'in-process restart', 'preemption', 'nvidia-resiliency-ext'.","main","7a940b8e5af16c95e5c190a524421d32fc7445a9",".claude\u002Fskills\u002Fnemo-mbridge-resiliency","https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp","https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui","https:\u002F\u002Fneuroads.com.br","2026-08-12 00:42:44",{"source":8,"skillName":5,"commands":53,"prompt":63,"zipAvailable":64},[54,57,60],{"tool":55,"command":56},"npx","npx skills add headmktlucca-ui\u002Fneuroads-lp --skill nemo-mbridge-resiliency",{"tool":58,"command":59},"bunx","bunx skills add headmktlucca-ui\u002Fneuroads-lp --skill nemo-mbridge-resiliency",{"tool":61,"command":62},"pnpm","pnpm dlx skills add headmktlucca-ui\u002Fneuroads-lp --skill nemo-mbridge-resiliency","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fnemo-mbridge-resiliency\u002FSKILL.md",true,[66,72,76,80,84],{"path":67,"resourceType":68,"resourceTypeDesc":69,"sizeBytes":70,"sourceHtmlUrl":71},"BENCHMARK.md","REFERENCE","参考文档",4267,"https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fnemo-mbridge-resiliency\u002FBENCHMARK.md",{"path":73,"resourceType":68,"resourceTypeDesc":69,"sizeBytes":74,"sourceHtmlUrl":75},"card.yaml",4910,"https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fnemo-mbridge-resiliency\u002Fcard.yaml",{"path":77,"resourceType":68,"resourceTypeDesc":69,"sizeBytes":78,"sourceHtmlUrl":79},"evals\u002Fevals.json",1310,"https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fnemo-mbridge-resiliency\u002Fevals\u002Fevals.json",{"path":81,"resourceType":68,"resourceTypeDesc":69,"sizeBytes":82,"sourceHtmlUrl":83},"skill-card.md",3896,"https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fnemo-mbridge-resiliency\u002Fskill-card.md",{"path":85,"resourceType":86,"resourceTypeDesc":87,"sizeBytes":88,"sourceHtmlUrl":89},"skill.oms.sig","OTHER","其他",4809,"https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fnemo-mbridge-resiliency\u002Fskill.oms.sig",[91,118,143,169,187,222],{"skillNo":92,"name":93,"nameZh":6,"description":94,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":95,"licenseFileUrl":6,"sourceHtmlUrl":96,"categories":97,"keywords":106,"publishedAt":39},"01M6PTDCZTXTGSTX4RFK901YSZ","xunit-skill","Generates xUnit.net tests in C#. Covers Fact\u002FTheory, constructor injection, IClassFixture, and FluentAssertions. Use when user mentions \"xUnit\", \"[Fact]\", \"[Theory]\", \"Assert.Equal\", \"C# xUnit\". Triggers on: \"xUnit\", \"[Fact]\", \"[Theory]\", \"Assert.Equal C#\", \"xUnit.net\".","MIT","https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fxunit-skill\u002FSKILL.md",[98,103],{"categoryNo":99,"categoryName":100,"taxonomyType":101,"taxonomyTypeDesc":102},"01JBSEEDCAT000000000000008","测试与质量","SCENARIO","使用场景",{"categoryNo":104,"categoryName":105,"taxonomyType":101,"taxonomyTypeDesc":102},"01JBSEEDCAT000000000000023","Skill与智能体开发",[107,108,109,110,111,112,113,114,115,116,117],"xunit","xunit skill","testing","injection","user","use","tests","triggers","unit","unit-testing","C#",{"skillNo":119,"name":120,"nameZh":6,"description":121,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":95,"licenseFileUrl":6,"sourceHtmlUrl":122,"categories":123,"keywords":126,"publishedAt":39},"01M6PTCB59ZTG5FTMWGPYEMMBP","xcuitest-skill","Generates XCUITest UI tests for iOS\u002FiPadOS apps in Swift. Apple's native testing framework for reliable, fast UI automation. Supports local simulators and TestMu AI cloud real devices. Use when user mentions \"XCUITest\", \"XCTest\", \"iOS UI test\", \"Swift test\", \"XCUIApplication\". Triggers on: \"XCUITest\", \"XCTest UI\", \"iOS UI test\", \"Swift UI test\", \"XCUIApplication\", \"TestMu\".","https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fxcuitest-skill\u002FSKILL.md",[124,125],{"categoryNo":99,"categoryName":100,"taxonomyType":101,"taxonomyTypeDesc":102},{"categoryNo":104,"categoryName":105,"taxonomyType":101,"taxonomyTypeDesc":102},[127,128,109,129,130,131,132,133,112,134,135,136,137,138,113,114,139,140,141,142],"xcuitest","xcuitest skill","automation","test","testing framework","ios","devices","mobile","mobile-testing","ui test","local","apple","apps","native","for","objective",{"skillNo":144,"name":145,"nameZh":6,"description":146,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":95,"licenseFileUrl":6,"sourceHtmlUrl":147,"categories":148,"keywords":153,"publishedAt":39},"01M6PTATMNTY9NF4J7KSR76YK6","world-builder","Create and manage generative game zones for Dorothy's Pokemon-style world. Use this skill when creating, updating, or designing game worlds via MCP tools.","https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fworld-builder\u002FSKILL.md",[149,152],{"categoryNo":150,"categoryName":151,"taxonomyType":101,"taxonomyTypeDesc":102},"01JBSEEDCAT000000000000015","视觉设计",{"categoryNo":104,"categoryName":105,"taxonomyType":101,"taxonomyTypeDesc":102},[154,155,156,157,158,159,160,161,112,162,163,164,165,166,167,141,168],"builder","world","world builder","MCP","create","mcp tools","Tools","game","pokemon","manage","style","via","theme","zone","generative",{"skillNo":170,"name":171,"nameZh":6,"description":172,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":95,"licenseFileUrl":6,"sourceHtmlUrl":173,"categories":174,"keywords":177,"publishedAt":39},"01M6PT9Z92JTSQEHE1QR0NNW5F","webdriverio-skill","Generates WebdriverIO (WDIO) automation tests in JavaScript or TypeScript. Supports local and TestMu AI cloud. Use when user mentions \"WebdriverIO\", \"WDIO\", \"wdio.conf\", \"browser.url\", \"$\", \"$$\". Triggers on: \"WebdriverIO\", \"WDIO\", \"wdio\", \"browser.$\".","https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fwebdriverio-skill\u002FSKILL.md",[175,176],{"categoryNo":99,"categoryName":100,"taxonomyType":101,"taxonomyTypeDesc":102},{"categoryNo":104,"categoryName":105,"taxonomyType":101,"taxonomyTypeDesc":102},[178,109,179,129,180,181,182,183,111,112,184,137,113,114,185,186],"webdriverio skill","typescript","javascript","browser","cloud","e2e","e2e-testing","URL","conf",{"skillNo":188,"name":189,"nameZh":6,"description":190,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":95,"licenseFileUrl":6,"sourceHtmlUrl":191,"categories":192,"keywords":202,"publishedAt":39},"01M6PT9NVK36ZWCQ69YXK3FR4K","web-design-guidelines","Web platform design and accessibility guidelines. Use when building web interfaces, auditing accessibility, implementing responsive layouts, or reviewing web UI code. Triggers on tasks involving HTML, CSS, web components, WCAG compliance, responsive design, or web performance.","https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fweb\u002FSKILL.md",[193,196,197],{"categoryNo":194,"categoryName":195,"taxonomyType":101,"taxonomyTypeDesc":102},"01JBSEEDCAT000000000000009","前端开发",{"categoryNo":150,"categoryName":151,"taxonomyType":101,"taxonomyTypeDesc":102},{"categoryNo":198,"categoryName":199,"taxonomyType":200,"taxonomyTypeDesc":201},"01JBSEEDCAT000000000000032","设计师","PROFESSION","适用职业",[203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,114,221],"design","web","guidelines","web design guidelines","web design","css","performance","components","web components","tasks","code","web performance","ui","platform design","html","responsive design","responsive","platform","implementing",{"skillNo":223,"name":224,"nameZh":6,"description":225,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":95,"licenseFileUrl":6,"sourceHtmlUrl":226,"categories":227,"keywords":230,"publishedAt":39},"01M6PT9A3JKRE3X34276F2FW5R","watchos-design-guidelines","Apple Human Interface Guidelines for Apple Watch. Use when building watchOS apps, complications, or workout features. Triggers on tasks involving Watch UI, Digital Crown, glanceable interfaces, or wrist-based interactions.","https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fwatchos\u002FSKILL.md",[228,229],{"categoryNo":150,"categoryName":151,"taxonomyType":101,"taxonomyTypeDesc":102},{"categoryNo":198,"categoryName":199,"taxonomyType":200,"taxonomyTypeDesc":201},[203,205,231,212,215,112,232,233,138,220,234,114,139,235,141,236,23],"watchos design guidelines","workout","human","watch","interface","platform-design"]