AGENT SKILL

modelopt-monitor

Monitor submitted jobs (PTQ, evaluation, deployment) on SLURM clusters. Use when the user asks "check job status", "is my job done", "monitor my evaluation", "what's the status of the PTQ", "check on job" followed by a SLURM job ID, or after any skill submits a long-running job. Also triggers on "nel status", "squeue", or any request to check progress of a previously submitted job.

License 许可
Apache-2.0
最近更新
2026-11-15
Stars
★ 5.6k
Forks
806

01 前言 Front Matter

name
modelopt-monitor
description
Monitor submitted jobs (PTQ, evaluation, deployment) on SLURM clusters. Use when the user asks "check job status", "is my job done", "monitor my evaluation", "what's the status of the PTQ", "check on job" followed by a SLURM job ID, or after any skill submits a long-running job. Also triggers on "nel status", "squeue", or any request to check progress of a previously submitted job.

03 SKILL.md 正文

Job Monitor

Monitor jobs submitted to SLURM clusters — PTQ quantization, NEL evaluation, model deployment, or raw SLURM jobs.

When to use

  1. Auto-monitor — another skill (PTQ, evaluation, deployment) just submitted a job. Register the job and set up monitoring immediately.
  2. User-initiated — user asks about a job status. Check the current session registry first; if the job is not registered there, use the discovery steps below.

Job Registry

Active jobs are tracked in per-session registries under .claude/agents/. This avoids multiple agents clobbering one shared registry when they run at the same time.

Use the current agent session id as <session_id>:

  • Claude Code: $CLAUDE_CODE_SESSION_ID, or the session_id field from hook input
  • Codex: $CODEX_THREAD_ID
  • If no session id is available, create a stable id for the current terminal session and reuse it for every job registered by that agent

Registry layout:

.claude/agents/
  <session_id>/
    active_jobs.json

Each session's active_jobs.json is a JSON array:

[
  {
    "type": "nel",
    "id": "<invocation_id or slurm_job_id>",
    "host": "<cluster_hostname>",
    "user": "<ssh_user>",
    "submitted": "YYYY-MM-DD HH:MM",
    "description": "<what this job does>",
    "last_status": "<last known status>",
    "owner": {
      "agent": "claude-code|codex|manual",
      "session_id": "<session_id>"
    }
  }
]

type is one of: nel, slurm, launcher.


On Job Submission

Every time a job is submitted (by any skill or manually):

  1. Add an entry to .claude/agents/<session_id>/active_jobs.json. Create the session directory and file if they don't exist.
  2. Start a durable monitor (if one isn't already watching the registry) that polls this session's registered jobs until they reach terminal status. Prefer the Claude Code Monitor tool when it is available: write a small watcher that reads .claude/agents/<session_id>/active_jobs.json, checks every job with the appropriate method below, prints state-change events, updates last_status, removes terminal jobs from the session registry, and exits when no active jobs remain for this session.

The monitor should terminate naturally when every registered job has reached a terminal state. If the Monitor tool is not available in the current harness, run an equivalent background process that implements the same loop and lets the agent resume/restart when the process exits.

Always do both steps. Don't try to predict job duration.


On Monitor Event / Status Check

Whether triggered by monitor output or by the user asking "check status":

  1. Read the registry from .claude/agents/<session_id>/active_jobs.json
  2. Check each job using the appropriate method (see below)
  3. Report only state changes — compare against last_status in registry
  4. Update last_status in the session registry
  5. Remove completed jobs — any job in a terminal state (COMPLETED, FAILED, CANCELLED, KILLED, TIMEOUT, NODE_FAIL, OUT_OF_MEMORY, PREEMPTED, BOOT_FAIL, DEADLINE)
  6. If no active jobs remain — let the monitor exit

How to Check Each Job Type

Each check method has its own status vocabulary. A watcher that mixes them (e.g. uses SLURM's COMPLETED terminal-state regex against nel status output) will silently never fire terminal transitions. Always match against the vocabulary of the source you're polling.

NEL jobs (type: nel)

  • Check: nel status <id>.
extract_nel_state() {
  local jid="$1" nel_bin="${NEL:-nel}" output state_col
  output=$("$nel_bin" status "$jid" 2>&1)
  state_col=$(echo "$output" \
    | awk -F'|' -v prefix="$jid." 'index($1, prefix) == 1 { print $2; exit }')
  [ -z "$state_col" ] && state_col="$output"
  echo "$state_col" \
    | LC_ALL=C tr '[:lower:]' '[:upper:]' \
    | awk 'match($0, /(PENDING|RUNNING|SUCCESS|FAILED|KILLED|ERROR|NOT[[:space:]]+FOUND)/) { print substr($0, RSTART, RLENGTH); exit }' \
    | sed 's/[[:space:]][[:space:]]*/ /g'
}

is_nel_terminal() {
  case "$(extract_nel_state "$1")" in
    SUCCESS|FAILED|KILLED|ERROR|"NOT FOUND") return 0 ;;
    *) return 1 ;;
  esac
}
  • On completion: nel info <id> to fetch results.
  • On failure: nel info <id> --logs then inspect server/client/SLURM logs via SSH.

Launcher jobs (type: launcher)

  • Check: Tail the launcher's background output file for key events.
  • Key events: experiment ID, SLURM job ID, container import, calibration progress, export path, final status.
  • On failure: Look for Traceback, Error, or FAILED in the output.

Raw SLURM jobs (type: slurm)

  • Check: sacct; use sacct for the termination check because squeue can lag in COMPLETING after sacct reports a terminal state.
extract_slurm_state() {
  local jid="$1" host="$2"
  ssh "$host" "sacct -j $jid -X --format=State --noheader -P 2>/dev/null | head -1" \
    | sed 's/^[[:space:]]*//;s/[[:space:]]*$//' \
    | sed 's/^CANCELLED by .*/CANCELLED/'
}

is_slurm_terminal() {
  case "$(extract_slurm_state "$1" "$2")" in
    COMPLETED|FAILED|CANCELLED|TIMEOUT|NODE_FAIL|OUT_OF_MEMORY|PREEMPTED|BOOT_FAIL|DEADLINE) return 0 ;;
    *) return 1 ;;
  esac
}
  • On completion: ssh <host> "sacct -j <id> --format=State,ExitCode,Elapsed -n".
  • On failure: Check the job's output log file.

Identifying Jobs (user-initiated, no ID given)

When the user asks about a job without specifying an ID, check in order:

  1. .claude/agents/<current_session_id>/active_jobs.json — current agent's jobs
  2. nel ls runs --since 1d — recent NEL runs
  3. ssh <host> "squeue -u <user>" — active SLURM jobs
  4. ls -lt tools/launcher/experiments/cicd/ | head -10 — recent launcher experiments

Reporting Guidelines

  • Report state changes proactively — PENDING → RUNNING, or job completes
  • Aggregate multiple jobs — "2 of 4 completed (MMLU-Pro: 42.3%, GSM8K: 67.1%), 1 running, 1 pending"
  • Summarize, don't echo — interpret events ("Calibration complete, exporting checkpoint") not raw logs
  • On failure, diagnose immediately — check logs and report root cause without waiting for user to ask
  • Minimize noise — don't report "still running" unless the user is actively asking

04 同仓库更多 Skills

NVIDIA
NVIDIA/Model-Optimizer

modelopt-speculative-decoding

Train, debug, and validate a speculative decoding draft model (EAGLE3, DFlash, DSpark, Domino) through the ModelOpt launcher pipeline. Use when the user wants to add a new model to a draft-training pipeline, asks why a pipeline run failed, wants experiment logs reviewed, or wants to check whether a run's acceptance rate meets threshold. Triggers on "EAGLE3", "DFlash", "DSpark", "Domino", "draft model", "acceptance rate", "speculative decoding pipeline". Do NOT use for quantizing a model (use modelopt-ptq) or serving a checkpoint (use modelopt-deployment).

★ 5.6k
NVIDIA
NVIDIA/Model-Optimizer

modelopt-run-agentx

Run the AgentX agentic serving benchmark with the SemiAnalysis harness. Use for AgentX setup, dataset downloads, concurrency sweeps, or serving latency and throughput comparisons.

★ 5.6k
NVIDIA
NVIDIA/Model-Optimizer

modelopt-release-cherry-pick

Audit merged bug-fix PRs and release NVBugs for missing cherry-pick labels, then cherry-pick labeled PRs into a release branch and open a PR. Use when asked to "cherry-pick PRs for release/X.Y.Z", "pick PRs to release branch", "verify cherry-pick labels", or "cherry-pick labeled PRs".

★ 5.6k
NVIDIA
NVIDIA/Model-Optimizer

modelopt-quant-recipe-search

Use when the user asks to find, search for, or optimize the best quantization recipe for a model, including direct requests like "find the best quantization recipe and generate a PTQ checkpoint." Guides the multi-candidate loop: choose compute-vs-memory success metrics, select ModelOpt recipe baselines, design AutoQuant/manual recipe deltas, interpret sensitivity, and decide next candidates. Do NOT use for a single known PTQ recipe run (use modelopt-ptq), serving (use modelopt-deployment), creating/running evals (use modelopt-evaluation or modelopt-launching-evals), monitoring jobs (use modelopt-monitor), MLflow browsing (use modelopt-accessing-mlflow), or comparing completed baseline-vs-candidate scores only (use modelopt-compare-results).

★ 5.6k
NVIDIA
NVIDIA/Model-Optimizer

modelopt-qad

Run explicitly requested ModelOpt Quantization-Aware Distillation (QAD) through Megatron Bridge (recommended) or Hugging Face Transformers to recover a measured BF16-to-PTQ accuracy gap. Use only when the user explicitly asks for QAD, including its topology, data preparation, launch, resume, checkpoint export, or recovery decisions.

★ 5.6k
NVIDIA
NVIDIA/Model-Optimizer

modelopt-ptq

Use when the user asks to "quantize a model", "run PTQ", "post-training quantization", "NVFP4 quantization", "FP8 quantization", "INT8 quantization", "INT4 AWQ", "quantize LLM", "quantize MoE", "quantize VLM", or needs to produce a quantized HuggingFace checkpoint from a pretrained model using ModelOpt. Do NOT use for multi-candidate recipe exploration or optimization (use modelopt-quant-recipe-search).

★ 5.6k