nemo-automodel-launcher-config
Configure NeMo AutoModel job launches for interactive runs, Slurm clusters, and SkyPilot cloud execution.
- License 许可
- Apache-2.0
- 最近更新
- 2026-08-12
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- nemo-automodel-launcher-config
- license
- Apache-2.0
- metadata
- { "tags": [ "nemo-automodel", "launcher-config" ], "author": "NVIDIA" }
- description
- Configure NeMo AutoModel job launches for interactive runs, Slurm clusters, and SkyPilot cloud execution.
- when_to_use
- Configuring Slurm or SkyPilot job submission, setting up multi-node launch scripts, debugging job submission failures, or switching between interactive and cluster launch modes.
02 资源文件 4
- 参考文档BENCHMARK.md4.1 KB
- 参考文档evals/evals.json3.4 KB
- 参考文档skill-card.md3.6 KB
- 其他skill.oms.sig4.5 KB
03 SKILL.md 正文
Launcher Configuration
NeMo AutoModel supports three launch methods: interactive (torchrun), Slurm (HPC clusters), and SkyPilot (cloud-agnostic).
Instructions
For launcher questions, answer directly from this skill without inspecting the repository unless the user asks you to edit files. Keep the answer focused on the relevant launch YAML, required fields, and the expected runtime behavior.
Use these compact answer patterns for common questions:
- Slurm multi-node: show a
slurm:YAML block withjob_name,nodes,ntasks_per_node,time,accountorpartition,container_image,hf_home, optionalextra_mounts,env_vars, andmaster_port; explain that the launcher derivesWORLD_SIZE = nodes * ntasks_per_nodeand setsMASTER_ADDRandMASTER_PORT. - SkyPilot spot: show a
skypilot:YAML block withcloud,accelerators,num_nodes,use_spot: true,disk_size,region,setup, andenv_vars; warn that spot instances can be preempted, set a shortstep_scheduler.checkpoint_interval, and resume withrestore_from.path. - Nsight Systems on Slurm: show
slurm.nsys_enabled: truealongside normal Slurm fields, say the launcher wraps the training command withnsys profile, and state that it produces a.nsys-repreport file. Treat profiling as diagnostic-only: use short profiling runs and disable it for normal production training because it adds overhead and large artifacts.
For Slurm answers, start with this minimal template and then adjust only the fields the user asked about:
slurm:
job_name: llm_finetune
nodes: 2
ntasks_per_node: 8
time: "04:00:00"
account: my_account
partition: batch
container_image: nvcr.io/nvidia/nemo:dev
hf_home: ~/.cache/huggingface
master_port: 13742
env_vars:
HF_TOKEN: "${HF_TOKEN}"
For Slurm-only questions, do not discuss SkyPilot or profiling unless the user
asks. For profiling questions, say the .nsys-rep report is written in the
Slurm job working or output directory, using the launcher's Nsys output setting
when one is configured.
Routing Boundary
Use this skill only for launch mechanics: interactive execution, Slurm, SkyPilot, containers, mounts, environment variables, rendezvous settings, and profiling.
Do not use this skill for implementing or registering new model architectures, Hugging Face state-dict adapters, model files, or capability flags. Those are model onboarding tasks, not launcher configuration tasks.
Launch Methods
- Interactive (default): runs torchrun on the current node. Suitable for single-node development and debugging.
- Slurm: submits a batch job to an HPC cluster scheduler. Handles multi-node setup, container management, and environment configuration.
- SkyPilot: cloud-agnostic job submission to AWS, GCP, Azure, Lambda, or Kubernetes. Supports spot instances.
Interactive Launch
# Single GPU
automodel finetune llm -c config.yaml
# Multi-GPU (all GPUs on current node)
torchrun --nproc_per_node=8 -m nemo_automodel._cli.app finetune llm -c config.yaml
No additional YAML section is needed for interactive mode. The CLI routes to torchrun automatically when no slurm: or skypilot: section is present in the config.
Slurm Configuration
The SlurmConfig dataclass generates an SBATCH script from a template.
YAML Example
slurm:
job_name: llm_finetune
nodes: 2
ntasks_per_node: 8
time: "04:00:00"
account: my_account
partition: batch
container_image: nvcr.io/nvidia/nemo:dev
hf_home: ~/.cache/huggingface
extra_mounts:
- source: /data
dest: /data
env_vars:
WANDB_API_KEY: "${WANDB_API_KEY}"
HF_TOKEN: "${HF_TOKEN}"
Key Fields
job_name: Slurm job identifiernodes: number of nodes to requestntasks_per_node: number of tasks (GPUs) per nodetime: wall-time limit in HH:MM:SS formataccount,partition: Slurm scheduling parameterscontainer_image: Enroot/Pyxis container image pathnemo_mount: mount point for NeMo AutoModel source inside the containerhf_home: HuggingFace cache directory pathextra_mounts: list ofVolumeMapping(source, dest)for additional container bind mountsmaster_port: port for distributed communication (default 13742)env_vars: environment variables passed into the jobnsys_enabled: when true, wraps the training command withnsys profilefor Nsight Systems profiling
SkyPilot Configuration
The SkyPilotConfig dataclass defines cloud job parameters.
YAML Example
skypilot:
cloud: aws
accelerators: "H100:8"
num_nodes: 2
use_spot: true
disk_size: 200
region: us-east-1
setup: "pip install nemo-automodel"
env_vars:
HF_TOKEN: "${HF_TOKEN}"
Key Fields
cloud: target cloud provider (aws,gcp,azure,lambda,kubernetes)accelerators: GPU type and count (e.g.,"H100:8","A100-80GB:4")num_nodes: number of cloud instancesuse_spot: use preemptible/spot instances for cost savingsdisk_size: disk size in GB per noderegion: cloud region for instance placementsetup: shell commands to run before the training job (e.g., install dependencies)env_vars: environment variables for the job
SkyPilot spot checklist
When using spot or preemptible instances:
- Set
use_spot: truein theskypilot:section. - Include
accelerators,num_nodes,disk_size,region,setup, and requiredenv_vars. - Use short checkpoint intervals in the recipe, for example
step_scheduler.checkpoint_interval, because spot instances can be preempted. - Resume from the most recent checkpoint after preemption with the recipe's
restore_fromsetting.
Minimal spot-resume recipe keys:
step_scheduler:
checkpoint_interval: 100
restore_from:
path: /checkpoints/latest
Multi-Node Environment
For multi-node training (both Slurm and SkyPilot), the launcher automatically configures:
MASTER_ADDR: hostname of the first nodeMASTER_PORT: port for rendezvous (default 13742)WORLD_SIZE: total number of processes (nodes * ntasks_per_node)- NCCL environment variables for optimized collective communication
Nsys Profiling
Enable Nsight Systems profiling in Slurm jobs:
slurm:
job_name: llm_profile
nodes: 1
ntasks_per_node: 8
time: "00:30:00"
account: my_account
partition: batch
container_image: nvcr.io/nvidia/nemo:dev
nsys_enabled: true
This is a Slurm launcher setting. Normal Slurm fields such as job_name,
nodes, ntasks_per_node, time, account or partition, and
container_image still apply.
When nsys_enabled: true, the launcher wraps the training command with
nsys profile and writes a .nsys-rep report file for performance analysis
in the Slurm job working or output directory.
Profiling is diagnostic-only: run it for a short investigation, expect overhead
and large artifacts, and turn it off for normal production training.
Code Anchors
components/launcher/slurm/config.py- SlurmConfig dataclass, VolumeMappingcomponents/launcher/slurm/template.py- SBATCH script template generationcomponents/launcher/slurm/utils.py- Slurm submission utilitiescomponents/launcher/skypilot/config.py- SkyPilotConfig dataclass_cli/app.py- CLI entry point and launcher routing logic
Pitfalls
- Port collisions: if the default
master_port(13742) is in use by another job on the same node, change it to avoid connection failures. - Container mounts: the
sourcepath inextra_mountsmust exist on all nodes in the allocation. Missing paths cause container startup failures. - Slurm fault tolerance: the fault tolerance plugin is Slurm-specific and does not work with SkyPilot or interactive mode.
- SkyPilot spot preemption: spot instances (
use_spot: true) may be preempted by the cloud provider. Enable checkpointing with short intervals to minimize lost work. - Environment variable syntax: use
${VAR}syntax in YAML for shell variable expansion. Bare variable names will not be expanded. - Time limit vs async checkpoint: if the Slurm
timelimit is too short, an in-progress async checkpoint write may be killed before completion, resulting in a corrupted checkpoint. Leave at least 5-10 minutes of margin.
04 同仓库更多 Skills
xunit-skill
Generates xUnit.net tests in C#. Covers Fact/Theory, constructor injection, IClassFixture, and FluentAssertions. Use when user mentions "xUnit", "[Fact]", "[Theory]", "Assert.Equal", "C# xUnit". Triggers on: "xUnit", "[Fact]", "[Theory]", "Assert.Equal C#", "xUnit.net".
xcuitest-skill
Generates XCUITest UI tests for iOS/iPadOS apps in Swift. Apple's native testing framework for reliable, fast UI automation. Supports local simulators and TestMu AI cloud real devices. Use when user mentions "XCUITest", "XCTest", "iOS UI test", "Swift test", "XCUIApplication". Triggers on: "XCUITest", "XCTest UI", "iOS UI test", "Swift UI test", "XCUIApplication", "TestMu".
world-builder
Create and manage generative game zones for Dorothy's Pokemon-style world. Use this skill when creating, updating, or designing game worlds via MCP tools.
webdriverio-skill
Generates WebdriverIO (WDIO) automation tests in JavaScript or TypeScript. Supports local and TestMu AI cloud. Use when user mentions "WebdriverIO", "WDIO", "wdio.conf", "browser.url", "$", "$$". Triggers on: "WebdriverIO", "WDIO", "wdio", "browser.$".
web-design-guidelines
Web platform design and accessibility guidelines. Use when building web interfaces, auditing accessibility, implementing responsive layouts, or reviewing web UI code. Triggers on tasks involving HTML, CSS, web components, WCAG compliance, responsive design, or web performance.
watchos-design-guidelines
Apple Human Interface Guidelines for Apple Watch. Use when building watchOS apps, complications, or workout features. Triggers on tasks involving Watch UI, Digital Crown, glanceable interfaces, or wrist-based interactions.