[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6PSFR1JG3AB08RWXXJSS0F9":3,"skill-same:01M6PSFR1JG3AB08RWXXJSS0F9":113},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":52,"commitSha":53,"skillDirectory":54,"forksCount":14,"repositoryHtmlUrl":55,"repositoryHomepageUrl":6,"ownerHtmlUrl":56,"ownerBlogUrl":57,"ownerDescription":6,"updatedAt":58,"install":59,"resources":72},"01M6PSFR1JG3AB08RWXXJSS0F9","tao-run-on-slurm",null,"Remote SLURM GPU cluster execution over SSH with sbatch\u002Fsrun, Pyxis\u002FEnroot containers, and Lustre-backed results. Use when running TAO training\u002Feval\u002Finference jobs on an on-prem or DGX SLURM cluster. Trigger phrases include \"run on SLURM\", \"submit sbatch\", \"DGX SLURM cluster\", \"Pyxis\u002FEnroot container\", \"Lustre dataset\".","headmktlucca-ui\u002Fneuroads-lp","headmktlucca-ui","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F247379838?v=4","01M6PDVAV6M1W96D1SR1PDF8CS","NeuroAds","01M6PDVAZ4P281GNDKD45CYTF5",0,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Ftao-run-on-slurm\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"tao run on slurm","run","Slurm","tao","only","nvidia-tao","login","you","remote","submit","dataset","hostname","auth","read","eval","node","access","Bash","passwordless","training","2026-11-10 10:12:30",false,"---\nname: tao-run-on-slurm\ndescription: Remote SLURM GPU cluster execution over SSH with sbatch\u002Fsrun, Pyxis\u002FEnroot containers, and Lustre-backed\n  results. Use when running TAO training\u002Feval\u002Finference jobs on an on-prem or DGX SLURM cluster. Trigger phrases include\n  \"run on SLURM\", \"submit sbatch\", \"DGX SLURM cluster\", \"Pyxis\u002FEnroot container\", \"Lustre dataset\".\nlicense: Apache-2.0\ncompatibility: Requires SSH access to a SLURM login node (passwordless via key auth) and SLURM_USER + SLURM_HOSTNAME env vars.\n  The TAO SDK with the slurm extra (pip install 'nvidia-tao-sdk[slurm]') is needed only if you want Job handles, S3 I\u002FO wrapping,\n  or run-folder durability via ActionWorkflow.\nmetadata:\n  author: NVIDIA Corporation\n  version: \"0.2.0\"\nallowed-tools: Read Bash\ntags:\n- platform\n- slurm\n---\n\n# SLURM\n\nRemote GPU compute platform for clusters managed by SLURM. Jobs are submitted\nfrom the TAO service or SDK host to a login node over SSH, staged on a shared\nfilesystem, submitted with `sbatch`, and executed with `srun` container support.\n\nUse SLURM when the user has access to a managed GPU cluster, shared Lustre\nstorage, and scheduler-owned GPU allocation. Do not use SLURM for local files\nthat exist only on the agent machine; data and outputs must be reachable from\nthe cluster.\n\n## Preflight\n\n```bash\n# 1. SSH to the login node works without a password prompt\nSLURM_HOST=\"${SLURM_HOSTNAME%%,*}\"\n[ -n \"$SLURM_USER\" ] && [ -n \"$SLURM_HOST\" ] || {\n  echo \"MISSING: set SLURM_USER and SLURM_HOSTNAME (comma-separated for failover) in your env (~\u002F.config\u002Ftao\u002F.env).\"\n  exit 1\n}\nssh -o BatchMode=yes -o ConnectTimeout=10 \"${SLURM_USER}@${SLURM_HOST}\" \"true\" 2>\u002Fdev\u002Fnull || {\n  echo \"MISSING: passwordless SSH to ${SLURM_USER}@${SLURM_HOST} not working. See references\u002Fssh-setup.md.\"\n  exit 1\n}\n\n# 2. Optional: TAO SDK wrapper for Job handles + S3 wrapping.\n# nvidia-tao-sdk is on public PyPI; pin lives in versions.yaml (wheels.tao_sdk_slurm).\nPIN=$(\"${TAO_SKILL_BANK_PATH:?}\u002Fscripts\u002Fresolve_versions_key.py\" wheels.tao_sdk_slurm)\npython -c \"import tao_sdk\" 2>\u002Fdev\u002Fnull || {\n  echo \"MISSING: nvidia-tao-sdk not installed. Run:\"\n  echo \"  pip install \\\"$PIN\\\"\"\n  exit 1\n}\n```\n\nIf a check fails, the agent prompts the user to authorize the install\u002Ffix via Bash.\n\nA third preflight step applies only for **private `nvcr.io` images**: Pyxis on\nthe compute nodes needs persistent enroot credentials in\n`~\u002F.config\u002Fenroot\u002F.credentials` on the cluster (it does NOT read `NGC_KEY` from\nthe job env). Without them, auth-gated pulls fail with \"Could not process JSON\ninput\" at job startup. This runs once per (cluster, user). See\n`references\u002Fssh-setup.md` for the full check and the `printf | ssh` install\npattern that keeps `NGC_KEY` out of history, files, and chat output. Skip it for\npublic images.\n\n## Prerequisites\n\nBefore any job is submitted, the host running the TAO service or SDK must log in\nto at least one host from `SLURM_HOSTNAME` over SSH **without an interactive\npassword prompt**. The handler runs `sbatch`, `squeue`, `sacct`, `scancel`, and\nlog tails non-interactively, so password or 2FA prompts will fail the job at\nsubmit or status time.\n\nSet this up once per (host, login node, user) tuple: create an SSH keypair,\ninstall the public key on each login host, trust the host key, lock private-key\npermissions to `chmod 600`, and verify with `ssh -o BatchMode=yes ...`. See\n`references\u002Fssh-setup.md` for the full step-by-step (including the `~\u002F.ssh\u002Fconfig`\nalias, the container key-mount note, and the 2FA \u002F `SSH_AUTH_SOCK` fallback). The\nsame file holds the **SSH failure remediation prompt** to show the user when\npasswordless SSH fails.\n\n## Credentials\n\n- **SLURM_USER** (required): SSH username for the login node. In microservices\n  workspace metadata this is `cloud_specific_details.slurm_user`.\n- **SLURM_HOSTNAME** (required): Comma-separated login hostnames for failover.\n  Microservices schema stores this as the list field\n  `cloud_specific_details.slurm_hostname`.\n- **SLURM_PARTITION** (required): Partition list for GPU job submission. Ask\n  for this in the mandatory SLURM intake list. The packaged default is\n  `polar,polar3,polar4,grizzly`, which are treated as 4-hour queues.\n- **SSH_KEY_PATH** (preferred and expected before launch): private key path for\n  non-interactive public-key auth to the login node. If passwordless SSH fails,\n  ask the user for `SSH_KEY_PATH=\u002Fpath\u002Fto\u002Fprivate_key` and show the setup steps\n  in `references\u002Fssh-setup.md`; do not bury this behind several alternate choices.\n- **SSH_AUTH_SOCK** (advanced fallback): SSH agent socket with an accepted key\n  already loaded. Prefer `SSH_KEY_PATH` in user-facing remediation prompts.\n- **SLURM_BASE_RESULTS_DIR** (optional): Base shared filesystem path. Default\n  convention from `tao-core` is `\u002Flustre\u002Ffsw\u002Fportfolios\u002Fedgeai\u002F\u003Cyour-dir>`,\n  where `\u003Cyour-dir>` is your per-user directory on the cluster.\n- **SLURM_ACCOUNT** (usually required by site policy): Account charged by\n  `#SBATCH --account`.\n\nDo not ask for `SLURM_ACCOUNT` or `SLURM_BASE_RESULTS_DIR` in the initial\nintake unless the user says their site requires an account, wants a custom\nresults root, or the workflow cannot proceed without overriding defaults.\n\n## Backend Details\n\nUse `backend_details.backend_type = \"slurm\"` when routing a job to this\nplatform. Supported backend details from the microservices schema:\n\n```json\n{\n  \"backend_type\": \"slurm\",\n  \"partition\": \"polar,polar3,polar4,grizzly\",\n  \"cluster_name\": \"optional-name\"\n}\n```\n\nRuntime metadata is stored under `backend_details.slurm_metadata`, especially\n`slurm_job_id` and `job_dir`. Do not invent these values. They are written\nafter `sbatch` returns a scheduler job id.\n\n## Storage\n\nSLURM jobs run on the cluster, so local paths from the API host are not valid\ndataset paths. Prefer shared filesystem URIs:\n\n- Use `lustre:\u002F\u002F\u002Fabsolute\u002Fpath` for user-provided datasets on Lustre.\n- `slurm:\u002F\u002F` paths may appear in microservices metadata and are converted to\n  actual Lustre paths before the container starts.\n- Avoid bare `\u002Flocal\u002Fpath` and `file:\u002F\u002F` dataset URIs for SLURM. Validation in\n  `tao-core` rejects local and file paths for remote backends.\n\nAccept either dataset roots or direct spec-key paths:\n\n- Root mode: `\u002Flustre\u002F...\u002F\u003Cmodel>\u002Ftrain`, which model skills map to required\n  files such as `\u003Croot>\u002Fannotations.json` and `\u003Croot>` as media path.\n- Direct spec mode: exact fields such as\n  `custom.train_dataset.annotation_path=\u002Flustre\u002F...\u002Ftrain.json` and\n  `custom.train_dataset.media_path=\u002Flustre\u002F...\u002Fvideos.tar.gz`.\n\nAfter passwordless SSH succeeds and before generating scripts, validate each\nrequired dataset file\u002Fpath from the login host:\n\n```bash\nssh -o BatchMode=yes \u003CSLURM_USER>@\u003Cworking-login-host> \\\n  'test -e \u002Flustre\u002F...\u002Fannotations.json && test -e \u002Flustre\u002F...\u002Fmedia_or_archive'\n```\n\nIf the remote `test -e` fails, stop and ask for corrected paths or for the data\nto be staged onto shared cluster storage. Do not create runner scripts that will\nfail inside the first training job.\n\nResults default to:\n\n```text\n\u002Flustre\u002Ffsw\u002Fportfolios\u002Fedgeai\u002F\u003Cyour-dir>\u002Fresults\u002F\u003Cjob_id>\n```\n\n`\u003Cyour-dir>` is your per-user directory on the cluster.\n\nThe runner sets `TAO_API_RESULTS_DIR` to the parent results directory because\ncontainer code appends the job id when writing status and artifacts.\n\n> **Use Lustre, not S3, for SLURM job inputs.** SLURM's scheduler enforces a\n> GPU-idle timeout — a long `s3:\u002F\u002F` download at the top of the script can burn\n> the allocation before training begins, and the scheduler may kill the job.\n> Stage training data onto Lustre first; S3 \u002F HF \u002F NGC pre-fetch is fine only\n> for small auxiliary inputs (checkpoints, configs). See `references\u002Fsdk-usage.md`\n> for the full rationale.\n\n## Container Execution\n\n`tao-core` uses the SLURM handler to run TAO containers through Pyxis\u002FEnroot:\n\n1. Stage compact JSON files for specs, environment, and cloud metadata under\n   `\u003Cjob_dir>\u002Fspecs`, `\u003Cjob_dir>\u002Fenv`, and `\u003Cjob_dir>\u002Fmeta`.\n2. Optionally convert the Docker image to a cached SQSH image with\n   `srun -n1 -p \u003Cconversion_partition> enroot import`.\n3. Write an sbatch script under `\u003Cjob_dir>\u002Fsbatch\u002Fjob_\u003Cjob_id>.sbatch`.\n4. Submit `sbatch --export=ALL \u003Cscript>`.\n5. Run the container with `srun --container-image=\u003Cimage> --container-mounts=\u002Flustre`.\n\nImage formats accepted by the handler:\n\n- `\u002Fpath\u002Fto\u002Fimage.sqsh`\n- `registry#image:tag`\n- `docker:\u002F\u002Fregistry#image:tag`\n- ordinary `registry\u002Fimage:tag`, which is converted to Pyxis form when needed\n\nSQSH conversion is cached by image name. For `:latest` images, cached SQSH is\nused unless `force_reconvert_latest` is enabled.\n\n## Resource Mapping\n\nDefaults from `tao-core`:\n\n- `num_nodes`: 1\n- `num_gpus`: 4\n- `max_num_gpus_per_node`: 8\n- `cpus_per_task`: 16\n- `time_hours`: 4\n- `timeout_hours`: 3.8\n- `max_time_hours`: 4\n- `container_mounts`: `\u002Flustre`\n- `use_requeue`: true\n- `use_sqsh`: true\n\nWhen generating launchers or wrapper scripts for SLURM, set the wall-time\ndefaults explicitly from the packaged platform resource defaults:\n\n```bash\nexport SLURM_TIME_HOURS=\"${SLURM_TIME_HOURS:-4}\"\nexport SLURM_TIMEOUT_HOURS=\"${SLURM_TIMEOUT_HOURS:-3.8}\"\n```\n\nDo not default to 12 hours on SLURM. If the user supplies a longer\n`SLURM_TIME_HOURS`, verify that the selected partition supports it before\nsubmitting. For the packaged default partition list\n`polar,polar3,polar4,grizzly`, reject requests above 4 hours and ask for a\ndifferent partition only if the user actually wants a longer wall time.\n\nWhen `num_gpus` is greater than or equal to `max_num_gpus_per_node`, the\nhandler treats the request as exclusive per node and computes additional nodes\nfrom total GPU count when necessary.\n\nFor multi-node jobs (`num_nodes > 1`), the sbatch script exports `WORLD_SIZE`,\n`MASTER_ADDR`, `MASTER_PORT`, `NODE_RANK`, and `NUM_GPU_PER_NODE`, and Cosmos-RL\nhas special multi-node role handling for controller, policy, and rollout\nworkers. See `references\u002Fmulti-node.md` for the full sbatch directives, the\nrendezvous env-var table and contract, and cluster requirements.\n\n## Monitoring\n\n- Scheduler status comes from the stored SLURM job id via `squeue` or `sacct`.\n- TAO terminal status comes from `status.json` in the shared results folder.\n- If the user enabled chat monitoring, continue polling at the requested\n  interval while the job is `PENDING`, `RUNNING`, or otherwise non-terminal.\n  Do not stop after a fixed elapsed time such as 30 minutes; long queue waits\n  are normal on shared GPU partitions.\n- Do not send a final response for a non-terminal SLURM job when chat\n  monitoring is enabled. A final response is a detach action; use it only if\n  the user asked to detach\u002Fstop or the job reached terminal state.\n- Logs are read over SSH from:\n\n```text\n\u003Cjob_dir>\u002Fslurm-logs\u002F\u003Cslurm_job_name>-\u003Cslurm_job_id>\u002Fmain.out\n\u003Cjob_dir>\u002Fslurm-logs\u002F\u003Cslurm_job_name>-\u003Cslurm_job_id>\u002Fmain.err\n```\n\nStatus mapping:\n\n- `PENDING` -> `Pending`\n- `RUNNING` or `COMPLETING` -> `Running`\n- `COMPLETED` -> check `status.json`\n- `FAILED`, `BOOT_FAIL`, `DEADLINE`, `OUT_OF_MEMORY`, `NODE_FAIL` -> retry if\n  logs match retriable infrastructure patterns, otherwise `Error`\n- `CANCELLED`, `PREEMPTED`, `REVOKED` -> `Canceled`\n- `TIMEOUT` -> `Error`\n- `SUSPENDED`, `STOPPED` -> `Paused`\n\n## Cancellation\n\nCancel by looking up `backend_details.slurm_metadata.slurm_job_id` and running\n`scancel \u003Cslurm_job_id>` over SSH. Treat missing or already terminated SLURM\njobs as successful cancellation.\n\n## Multi-node training (distributed)\n\nSLURM is the platform of choice for large multi-node runs — pass `num_nodes > 1`\nand the SDK handles the sbatch directives and PyTorch-distributed env vars\nautomatically. See `references\u002Fmulti-node.md` for a worked `create_job` example,\nthe generated sbatch directives, the rendezvous env-var table (`WORLD_SIZE`,\n`NUM_GPU_PER_NODE`, `NODE_RANK`, `MASTER_ADDR`, `MASTER_PORT`), the Cosmos-RL\nrole note, cluster requirements (Pyxis\u002FEnroot, InfiniBand\u002FNVLink, Lustre), and\nupstream reference links.\n\n## Running via the TAO SDK\n\nThe SDK install is covered in Preflight — `pip install 'nvidia-tao-sdk[slurm]'`.\nUse it when you want Job handles, the sbatch\u002F`squeue`\u002F`sacct` plumbing handled\nfor you, run-folder durability via `ActionWorkflow`, or convenient cloud-storage\nI\u002FO (`s3:\u002F\u002F`, `hf_model:\u002F\u002F`, `ngc:\u002F\u002F`). Without the SDK, drive `sbatch` and\n`srun` yourself.\n\nAuto-retry is **fully automatic**: a background monitor polls `squeue`\u002F`sacct`\nand re-`sbatch`'s the staged script on infrastructure-looking failures up to\n`MAX_JOB_RETRIES = 10`, while plain training failures surface immediately. In\naddition, `#SBATCH --requeue` is set by default (`SLURM_USE_REQUEUE`, defaults\nto `true`). See `references\u002Fsdk-usage.md` for the `SlurmSDK` \u002F `build_entrypoint`\ncode example, the Lustre-not-S3 rule, the retriable-failure classification, and\nthe full auto-retry and requeue behavior.\n\n## Failure Modes\n\nCommon failures: SSH auth failure, local dataset path rejected, SQSH conversion\ntimeout, Pyxis\u002FEnroot unavailable, and bad-node \u002F transient GPU failures (which\nthe handler retries up to the configured limit). See\n`references\u002Ftroubleshooting.md` for the diagnosis and remediation of each.\n","# SLURM\n\nRemote GPU compute platform for clusters managed by SLURM. Jobs are submitted\nfrom the TAO service or SDK host to a login node over SSH, staged on a shared\nfilesystem, submitted with `sbatch`, and executed with `srun` container support.\n\nUse SLURM when the user has access to a managed GPU cluster, shared Lustre\nstorage, and scheduler-owned GPU allocation. Do not use SLURM for local files\nthat exist only on the agent machine; data and outputs must be reachable from\nthe cluster.\n\n## Preflight\n\n```bash\n# 1. SSH to the login node works without a password prompt\nSLURM_HOST=\"${SLURM_HOSTNAME%%,*}\"\n[ -n \"$SLURM_USER\" ] && [ -n \"$SLURM_HOST\" ] || {\n  echo \"MISSING: set SLURM_USER and SLURM_HOSTNAME (comma-separated for failover) in your env (~\u002F.config\u002Ftao\u002F.env).\"\n  exit 1\n}\nssh -o BatchMode=yes -o ConnectTimeout=10 \"${SLURM_USER}@${SLURM_HOST}\" \"true\" 2>\u002Fdev\u002Fnull || {\n  echo \"MISSING: passwordless SSH to ${SLURM_USER}@${SLURM_HOST} not working. See references\u002Fssh-setup.md.\"\n  exit 1\n}\n\n# 2. Optional: TAO SDK wrapper for Job handles + S3 wrapping.\n# nvidia-tao-sdk is on public PyPI; pin lives in versions.yaml (wheels.tao_sdk_slurm).\nPIN=$(\"${TAO_SKILL_BANK_PATH:?}\u002Fscripts\u002Fresolve_versions_key.py\" wheels.tao_sdk_slurm)\npython -c \"import tao_sdk\" 2>\u002Fdev\u002Fnull || {\n  echo \"MISSING: nvidia-tao-sdk not installed. Run:\"\n  echo \"  pip install \\\"$PIN\\\"\"\n  exit 1\n}\n```\n\nIf a check fails, the agent prompts the user to authorize the install\u002Ffix via Bash.\n\nA third preflight step applies only for **private `nvcr.io` images**: Pyxis on\nthe compute nodes needs persistent enroot credentials in\n`~\u002F.config\u002Fenroot\u002F.credentials` on the cluster (it does NOT read `NGC_KEY` from\nthe job env). Without them, auth-gated pulls fail with \"Could not process JSON\ninput\" at job startup. This runs once per (cluster, user). See\n`references\u002Fssh-setup.md` for the full check and the `printf | ssh` install\npattern that keeps `NGC_KEY` out of history, files, and chat output. Skip it for\npublic images.\n\n## Prerequisites\n\nBefore any job is submitted, the host running the TAO service or SDK must log in\nto at least one host from `SLURM_HOSTNAME` over SSH **without an interactive\npassword prompt**. The handler runs `sbatch`, `squeue`, `sacct`, `scancel`, and\nlog tails non-interactively, so password or 2FA prompts will fail the job at\nsubmit or status time.\n\nSet this up once per (host, login node, user) tuple: create an SSH keypair,\ninstall the public key on each login host, trust the host key, lock private-key\npermissions to `chmod 600`, and verify with `ssh -o BatchMode=yes ...`. See\n`references\u002Fssh-setup.md` for the full step-by-step (including the `~\u002F.ssh\u002Fconfig`\nalias, the container key-mount note, and the 2FA \u002F `SSH_AUTH_SOCK` fallback). The\nsame file holds the **SSH failure remediation prompt** to show the user when\npasswordless SSH fails.\n\n## Credentials\n\n- **SLURM_USER** (required): SSH username for the login node. In microservices\n  workspace metadata this is `cloud_specific_details.slurm_user`.\n- **SLURM_HOSTNAME** (required): Comma-separated login hostnames for failover.\n  Microservices schema stores this as the list field\n  `cloud_specific_details.slurm_hostname`.\n- **SLURM_PARTITION** (required): Partition list for GPU job submission. Ask\n  for this in the mandatory SLURM intake list. The packaged default is\n  `polar,polar3,polar4,grizzly`, which are treated as 4-hour queues.\n- **SSH_KEY_PATH** (preferred and expected before launch): private key path for\n  non-interactive public-key auth to the login node. If passwordless SSH fails,\n  ask the user for `SSH_KEY_PATH=\u002Fpath\u002Fto\u002Fprivate_key` and show the setup steps\n  in `references\u002Fssh-setup.md`; do not bury this behind several alternate choices.\n- **SSH_AUTH_SOCK** (advanced fallback): SSH agent socket with an accepted key\n  already loaded. Prefer `SSH_KEY_PATH` in user-facing remediation prompts.\n- **SLURM_BASE_RESULTS_DIR** (optional): Base shared filesystem path. Default\n  convention from `tao-core` is `\u002Flustre\u002Ffsw\u002Fportfolios\u002Fedgeai\u002F\u003Cyour-dir>`,\n  where `\u003Cyour-dir>` is your per-user directory on the cluster.\n- **SLURM_ACCOUNT** (usually required by site policy): Account charged by\n  `#SBATCH --account`.\n\nDo not ask for `SLURM_ACCOUNT` or `SLURM_BASE_RESULTS_DIR` in the initial\nintake unless the user says their site requires an account, wants a custom\nresults root, or the workflow cannot proceed without overriding defaults.\n\n## Backend Details\n\nUse `backend_details.backend_type = \"slurm\"` when routing a job to this\nplatform. Supported backend details from the microservices schema:\n\n```json\n{\n  \"backend_type\": \"slurm\",\n  \"partition\": \"polar,polar3,polar4,grizzly\",\n  \"cluster_name\": \"optional-name\"\n}\n```\n\nRuntime metadata is stored under `backend_details.slurm_metadata`, especially\n`slurm_job_id` and `job_dir`. Do not invent these values. They are written\nafter `sbatch` returns a scheduler job id.\n\n## Storage\n\nSLURM jobs run on the cluster, so local paths from the API host are not valid\ndataset paths. Prefer shared filesystem URIs:\n\n- Use `lustre:\u002F\u002F\u002Fabsolute\u002Fpath` for user-provided datasets on Lustre.\n- `slurm:\u002F\u002F` paths may appear in microservices metadata and are converted to\n  actual Lustre paths before the container starts.\n- Avoid bare `\u002Flocal\u002Fpath` and `file:\u002F\u002F` dataset URIs for SLURM. Validation in\n  `tao-core` rejects local and file paths for remote backends.\n\nAccept either dataset roots or direct spec-key paths:\n\n- Root mode: `\u002Flustre\u002F...\u002F\u003Cmodel>\u002Ftrain`, which model skills map to required\n  files such as `\u003Croot>\u002Fannotations.json` and `\u003Croot>` as media path.\n- Direct spec mode: exact fields such as\n  `custom.train_dataset.annotation_path=\u002Flustre\u002F...\u002Ftrain.json` and\n  `custom.train_dataset.media_path=\u002Flustre\u002F...\u002Fvideos.tar.gz`.\n\nAfter passwordless SSH succeeds and before generating scripts, validate each\nrequired dataset file\u002Fpath from the login host:\n\n```bash\nssh -o BatchMode=yes \u003CSLURM_USER>@\u003Cworking-login-host> \\\n  'test -e \u002Flustre\u002F...\u002Fannotations.json && test -e \u002Flustre\u002F...\u002Fmedia_or_archive'\n```\n\nIf the remote `test -e` fails, stop and ask for corrected paths or for the data\nto be staged onto shared cluster storage. Do not create runner scripts that will\nfail inside the first training job.\n\nResults default to:\n\n```text\n\u002Flustre\u002Ffsw\u002Fportfolios\u002Fedgeai\u002F\u003Cyour-dir>\u002Fresults\u002F\u003Cjob_id>\n```\n\n`\u003Cyour-dir>` is your per-user directory on the cluster.\n\nThe runner sets `TAO_API_RESULTS_DIR` to the parent results directory because\ncontainer code appends the job id when writing status and artifacts.\n\n> **Use Lustre, not S3, for SLURM job inputs.** SLURM's scheduler enforces a\n> GPU-idle timeout — a long `s3:\u002F\u002F` download at the top of the script can burn\n> the allocation before training begins, and the scheduler may kill the job.\n> Stage training data onto Lustre first; S3 \u002F HF \u002F NGC pre-fetch is fine only\n> for small auxiliary inputs (checkpoints, configs). See `references\u002Fsdk-usage.md`\n> for the full rationale.\n\n## Container Execution\n\n`tao-core` uses the SLURM handler to run TAO containers through Pyxis\u002FEnroot:\n\n1. Stage compact JSON files for specs, environment, and cloud metadata under\n   `\u003Cjob_dir>\u002Fspecs`, `\u003Cjob_dir>\u002Fenv`, and `\u003Cjob_dir>\u002Fmeta`.\n2. Optionally convert the Docker image to a cached SQSH image with\n   `srun -n1 -p \u003Cconversion_partition> enroot import`.\n3. Write an sbatch script under `\u003Cjob_dir>\u002Fsbatch\u002Fjob_\u003Cjob_id>.sbatch`.\n4. Submit `sbatch --export=ALL \u003Cscript>`.\n5. Run the container with `srun --container-image=\u003Cimage> --container-mounts=\u002Flustre`.\n\nImage formats accepted by the handler:\n\n- `\u002Fpath\u002Fto\u002Fimage.sqsh`\n- `registry#image:tag`\n- `docker:\u002F\u002Fregistry#image:tag`\n- ordinary `registry\u002Fimage:tag`, which is converted to Pyxis form when needed\n\nSQSH conversion is cached by image name. For `:latest` images, cached SQSH is\nused unless `force_reconvert_latest` is enabled.\n\n## Resource Mapping\n\nDefaults from `tao-core`:\n\n- `num_nodes`: 1\n- `num_gpus`: 4\n- `max_num_gpus_per_node`: 8\n- `cpus_per_task`: 16\n- `time_hours`: 4\n- `timeout_hours`: 3.8\n- `max_time_hours`: 4\n- `container_mounts`: `\u002Flustre`\n- `use_requeue`: true\n- `use_sqsh`: true\n\nWhen generating launchers or wrapper scripts for SLURM, set the wall-time\ndefaults explicitly from the packaged platform resource defaults:\n\n```bash\nexport SLURM_TIME_HOURS=\"${SLURM_TIME_HOURS:-4}\"\nexport SLURM_TIMEOUT_HOURS=\"${SLURM_TIMEOUT_HOURS:-3.8}\"\n```\n\nDo not default to 12 hours on SLURM. If the user supplies a longer\n`SLURM_TIME_HOURS`, verify that the selected partition supports it before\nsubmitting. For the packaged default partition list\n`polar,polar3,polar4,grizzly`, reject requests above 4 hours and ask for a\ndifferent partition only if the user actually wants a longer wall time.\n\nWhen `num_gpus` is greater than or equal to `max_num_gpus_per_node`, the\nhandler treats the request as exclusive per node and computes additional nodes\nfrom total GPU count when necessary.\n\nFor multi-node jobs (`num_nodes > 1`), the sbatch script exports `WORLD_SIZE`,\n`MASTER_ADDR`, `MASTER_PORT`, `NODE_RANK`, and `NUM_GPU_PER_NODE`, and Cosmos-RL\nhas special multi-node role handling for controller, policy, and rollout\nworkers. See `references\u002Fmulti-node.md` for the full sbatch directives, the\nrendezvous env-var table and contract, and cluster requirements.\n\n## Monitoring\n\n- Scheduler status comes from the stored SLURM job id via `squeue` or `sacct`.\n- TAO terminal status comes from `status.json` in the shared results folder.\n- If the user enabled chat monitoring, continue polling at the requested\n  interval while the job is `PENDING`, `RUNNING`, or otherwise non-terminal.\n  Do not stop after a fixed elapsed time such as 30 minutes; long queue waits\n  are normal on shared GPU partitions.\n- Do not send a final response for a non-terminal SLURM job when chat\n  monitoring is enabled. A final response is a detach action; use it only if\n  the user asked to detach\u002Fstop or the job reached terminal state.\n- Logs are read over SSH from:\n\n```text\n\u003Cjob_dir>\u002Fslurm-logs\u002F\u003Cslurm_job_name>-\u003Cslurm_job_id>\u002Fmain.out\n\u003Cjob_dir>\u002Fslurm-logs\u002F\u003Cslurm_job_name>-\u003Cslurm_job_id>\u002Fmain.err\n```\n\nStatus mapping:\n\n- `PENDING` -> `Pending`\n- `RUNNING` or `COMPLETING` -> `Running`\n- `COMPLETED` -> check `status.json`\n- `FAILED`, `BOOT_FAIL`, `DEADLINE`, `OUT_OF_MEMORY`, `NODE_FAIL` -> retry if\n  logs match retriable infrastructure patterns, otherwise `Error`\n- `CANCELLED`, `PREEMPTED`, `REVOKED` -> `Canceled`\n- `TIMEOUT` -> `Error`\n- `SUSPENDED`, `STOPPED` -> `Paused`\n\n## Cancellation\n\nCancel by looking up `backend_details.slurm_metadata.slurm_job_id` and running\n`scancel \u003Cslurm_job_id>` over SSH. Treat missing or already terminated SLURM\njobs as successful cancellation.\n\n## Multi-node training (distributed)\n\nSLURM is the platform of choice for large multi-node runs — pass `num_nodes > 1`\nand the SDK handles the sbatch directives and PyTorch-distributed env vars\nautomatically. See `references\u002Fmulti-node.md` for a worked `create_job` example,\nthe generated sbatch directives, the rendezvous env-var table (`WORLD_SIZE`,\n`NUM_GPU_PER_NODE`, `NODE_RANK`, `MASTER_ADDR`, `MASTER_PORT`), the Cosmos-RL\nrole note, cluster requirements (Pyxis\u002FEnroot, InfiniBand\u002FNVLink, Lustre), and\nupstream reference links.\n\n## Running via the TAO SDK\n\nThe SDK install is covered in Preflight — `pip install 'nvidia-tao-sdk[slurm]'`.\nUse it when you want Job handles, the sbatch\u002F`squeue`\u002F`sacct` plumbing handled\nfor you, run-folder durability via `ActionWorkflow`, or convenient cloud-storage\nI\u002FO (`s3:\u002F\u002F`, `hf_model:\u002F\u002F`, `ngc:\u002F\u002F`). Without the SDK, drive `sbatch` and\n`srun` yourself.\n\nAuto-retry is **fully automatic**: a background monitor polls `squeue`\u002F`sacct`\nand re-`sbatch`'s the staged script on infrastructure-looking failures up to\n`MAX_JOB_RETRIES = 10`, while plain training failures surface immediately. In\naddition, `#SBATCH --requeue` is set by default (`SLURM_USE_REQUEUE`, defaults\nto `true`). See `references\u002Fsdk-usage.md` for the `SlurmSDK` \u002F `build_entrypoint`\ncode example, the Lustre-not-S3 rule, the retriable-failure classification, and\nthe full auto-retry and requeue behavior.\n\n## Failure Modes\n\nCommon failures: SSH auth failure, local dataset path rejected, SQSH conversion\ntimeout, Pyxis\u002FEnroot unavailable, and bad-node \u002F transient GPU failures (which\nthe handler retries up to the configured limit). See\n`references\u002Ftroubleshooting.md` for the diagnosis and remediation of each.\n",{"name":5,"tags":44,"license":15,"metadata":47,"description":7,"allowed-tools":50,"compatibility":51},[45,46],"platform","slurm",{"author":48,"version":49},"NVIDIA Corporation","0.2.0","Read Bash","Requires SSH access to a SLURM login node (passwordless via key auth) and SLURM_USER + SLURM_HOSTNAME env vars. The TAO SDK with the slurm extra (pip install 'nvidia-tao-sdk[slurm]') is needed only if you want Job handles, S3 I\u002FO wrapping, or run-folder durability via ActionWorkflow.","main","7a940b8e5af16c95e5c190a524421d32fc7445a9",".claude\u002Fskills\u002Ftao-run-on-slurm","https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp","https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui","https:\u002F\u002Fneuroads.com.br","2026-08-12 00:42:44",{"source":8,"skillName":5,"commands":60,"prompt":70,"zipAvailable":71},[61,64,67],{"tool":62,"command":63},"npx","npx skills add headmktlucca-ui\u002Fneuroads-lp --skill tao-run-on-slurm",{"tool":65,"command":66},"bunx","bunx skills add headmktlucca-ui\u002Fneuroads-lp --skill tao-run-on-slurm",{"tool":68,"command":69},"pnpm","pnpm dlx skills add headmktlucca-ui\u002Fneuroads-lp --skill tao-run-on-slurm","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Ftao-run-on-slurm\u002FSKILL.md",true,[73,79,83,87,91,95,99,103,107],{"path":74,"resourceType":75,"resourceTypeDesc":76,"sizeBytes":77,"sourceHtmlUrl":78},"BENCHMARK.md","REFERENCE","参考文档",4284,"https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Ftao-run-on-slurm\u002FBENCHMARK.md",{"path":80,"resourceType":75,"resourceTypeDesc":76,"sizeBytes":81,"sourceHtmlUrl":82},"evals\u002Fevals.json",762,"https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Ftao-run-on-slurm\u002Fevals\u002Fevals.json",{"path":84,"resourceType":75,"resourceTypeDesc":76,"sizeBytes":85,"sourceHtmlUrl":86},"references\u002Fmulti-node.md",3705,"https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Ftao-run-on-slurm\u002Freferences\u002Fmulti-node.md",{"path":88,"resourceType":75,"resourceTypeDesc":76,"sizeBytes":89,"sourceHtmlUrl":90},"references\u002Fsdk-usage.md",4156,"https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Ftao-run-on-slurm\u002Freferences\u002Fsdk-usage.md",{"path":92,"resourceType":75,"resourceTypeDesc":76,"sizeBytes":93,"sourceHtmlUrl":94},"references\u002Fskill_info.yaml",1282,"https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Ftao-run-on-slurm\u002Freferences\u002Fskill_info.yaml",{"path":96,"resourceType":75,"resourceTypeDesc":76,"sizeBytes":97,"sourceHtmlUrl":98},"references\u002Fssh-setup.md",5253,"https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Ftao-run-on-slurm\u002Freferences\u002Fssh-setup.md",{"path":100,"resourceType":75,"resourceTypeDesc":76,"sizeBytes":101,"sourceHtmlUrl":102},"references\u002Ftroubleshooting.md",1092,"https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Ftao-run-on-slurm\u002Freferences\u002Ftroubleshooting.md",{"path":104,"resourceType":75,"resourceTypeDesc":76,"sizeBytes":105,"sourceHtmlUrl":106},"skill-card.md",3823,"https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Ftao-run-on-slurm\u002Fskill-card.md",{"path":108,"resourceType":109,"resourceTypeDesc":110,"sizeBytes":111,"sourceHtmlUrl":112},"skill.oms.sig","OTHER","其他",5777,"https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Ftao-run-on-slurm\u002Fskill.oms.sig",[114,141,166,192,210,244],{"skillNo":115,"name":116,"nameZh":6,"description":117,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":118,"licenseFileUrl":6,"sourceHtmlUrl":119,"categories":120,"keywords":129,"publishedAt":39},"01M6PTDCZTXTGSTX4RFK901YSZ","xunit-skill","Generates xUnit.net tests in C#. Covers Fact\u002FTheory, constructor injection, IClassFixture, and FluentAssertions. Use when user mentions \"xUnit\", \"[Fact]\", \"[Theory]\", \"Assert.Equal\", \"C# xUnit\". Triggers on: \"xUnit\", \"[Fact]\", \"[Theory]\", \"Assert.Equal C#\", \"xUnit.net\".","MIT","https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fxunit-skill\u002FSKILL.md",[121,126],{"categoryNo":122,"categoryName":123,"taxonomyType":124,"taxonomyTypeDesc":125},"01JBSEEDCAT000000000000008","测试与质量","SCENARIO","使用场景",{"categoryNo":127,"categoryName":128,"taxonomyType":124,"taxonomyTypeDesc":125},"01JBSEEDCAT000000000000023","Skill与智能体开发",[130,131,132,133,134,135,136,137,138,139,140],"xunit","xunit skill","testing","injection","user","use","tests","triggers","unit","unit-testing","C#",{"skillNo":142,"name":143,"nameZh":6,"description":144,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":118,"licenseFileUrl":6,"sourceHtmlUrl":145,"categories":146,"keywords":149,"publishedAt":39},"01M6PTCB59ZTG5FTMWGPYEMMBP","xcuitest-skill","Generates XCUITest UI tests for iOS\u002FiPadOS apps in Swift. Apple's native testing framework for reliable, fast UI automation. Supports local simulators and TestMu AI cloud real devices. Use when user mentions \"XCUITest\", \"XCTest\", \"iOS UI test\", \"Swift test\", \"XCUIApplication\". Triggers on: \"XCUITest\", \"XCTest UI\", \"iOS UI test\", \"Swift UI test\", \"XCUIApplication\", \"TestMu\".","https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fxcuitest-skill\u002FSKILL.md",[147,148],{"categoryNo":122,"categoryName":123,"taxonomyType":124,"taxonomyTypeDesc":125},{"categoryNo":127,"categoryName":128,"taxonomyType":124,"taxonomyTypeDesc":125},[150,151,132,152,153,154,155,156,135,157,158,159,160,161,136,137,162,163,164,165],"xcuitest","xcuitest skill","automation","test","testing framework","ios","devices","mobile","mobile-testing","ui test","local","apple","apps","native","for","objective",{"skillNo":167,"name":168,"nameZh":6,"description":169,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":118,"licenseFileUrl":6,"sourceHtmlUrl":170,"categories":171,"keywords":176,"publishedAt":39},"01M6PTATMNTY9NF4J7KSR76YK6","world-builder","Create and manage generative game zones for Dorothy's Pokemon-style world. Use this skill when creating, updating, or designing game worlds via MCP tools.","https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fworld-builder\u002FSKILL.md",[172,175],{"categoryNo":173,"categoryName":174,"taxonomyType":124,"taxonomyTypeDesc":125},"01JBSEEDCAT000000000000015","视觉设计",{"categoryNo":127,"categoryName":128,"taxonomyType":124,"taxonomyTypeDesc":125},[177,178,179,180,181,182,183,184,135,185,186,187,188,189,190,164,191],"builder","world","world builder","MCP","create","mcp tools","Tools","game","pokemon","manage","style","via","theme","zone","generative",{"skillNo":193,"name":194,"nameZh":6,"description":195,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":118,"licenseFileUrl":6,"sourceHtmlUrl":196,"categories":197,"keywords":200,"publishedAt":39},"01M6PT9Z92JTSQEHE1QR0NNW5F","webdriverio-skill","Generates WebdriverIO (WDIO) automation tests in JavaScript or TypeScript. Supports local and TestMu AI cloud. Use when user mentions \"WebdriverIO\", \"WDIO\", \"wdio.conf\", \"browser.url\", \"$\", \"$$\". Triggers on: \"WebdriverIO\", \"WDIO\", \"wdio\", \"browser.$\".","https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fwebdriverio-skill\u002FSKILL.md",[198,199],{"categoryNo":122,"categoryName":123,"taxonomyType":124,"taxonomyTypeDesc":125},{"categoryNo":127,"categoryName":128,"taxonomyType":124,"taxonomyTypeDesc":125},[201,132,202,152,203,204,205,206,134,135,207,160,136,137,208,209],"webdriverio skill","typescript","javascript","browser","cloud","e2e","e2e-testing","URL","conf",{"skillNo":211,"name":212,"nameZh":6,"description":213,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":118,"licenseFileUrl":6,"sourceHtmlUrl":214,"categories":215,"keywords":225,"publishedAt":39},"01M6PT9NVK36ZWCQ69YXK3FR4K","web-design-guidelines","Web platform design and accessibility guidelines. Use when building web interfaces, auditing accessibility, implementing responsive layouts, or reviewing web UI code. Triggers on tasks involving HTML, CSS, web components, WCAG compliance, responsive design, or web performance.","https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fweb\u002FSKILL.md",[216,219,220],{"categoryNo":217,"categoryName":218,"taxonomyType":124,"taxonomyTypeDesc":125},"01JBSEEDCAT000000000000009","前端开发",{"categoryNo":173,"categoryName":174,"taxonomyType":124,"taxonomyTypeDesc":125},{"categoryNo":221,"categoryName":222,"taxonomyType":223,"taxonomyTypeDesc":224},"01JBSEEDCAT000000000000032","设计师","PROFESSION","适用职业",[226,227,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,45,137,243],"design","web","guidelines","web design guidelines","web design","css","performance","components","web components","tasks","code","web performance","ui","platform design","html","responsive design","responsive","implementing",{"skillNo":245,"name":246,"nameZh":6,"description":247,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":118,"licenseFileUrl":6,"sourceHtmlUrl":248,"categories":249,"keywords":252,"publishedAt":39},"01M6PT9A3JKRE3X34276F2FW5R","watchos-design-guidelines","Apple Human Interface Guidelines for Apple Watch. Use when building watchOS apps, complications, or workout features. Triggers on tasks involving Watch UI, Digital Crown, glanceable interfaces, or wrist-based interactions.","https:\u002F\u002Fgithub.com\u002Fheadmktlucca-ui\u002Fneuroads-lp\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fwatchos\u002FSKILL.md",[250,251],{"categoryNo":173,"categoryName":174,"taxonomyType":124,"taxonomyTypeDesc":125},{"categoryNo":221,"categoryName":222,"taxonomyType":223,"taxonomyTypeDesc":224},[226,228,253,235,238,135,254,255,161,45,256,137,162,257,164,258,259],"watchos design guidelines","workout","human","watch","interface","platform-design","features"]