modelopt-launching-evals
Run, monitor, analyze, and debug LLM evaluations via nemo-evaluator-launcher. Covers running evaluations, checking status and live progress, debugging failed runs, exporting artifacts and logs, and analyzing results. ALWAYS triggers on mentions of running evaluations, checking progress, debugging failed evals, analyzing or analysing runs or results, run directories or artifact paths on clusters, Slurm job issues, invocation IDs, or inspecting logs (client logs, server logs, SSH to cluster, tail logs, grep logs). Do NOT use for creating or modifying evaluation configs.
- License 许可
- Apache-2.0
- 最近更新
- 2026-11-15
- Stars
- ★ 5.6k
- Forks
- 806
01 前言 Front Matter
- name
- modelopt-launching-evals
- license
- Apache-2.0
- description
- Run, monitor, analyze, and debug LLM evaluations via nemo-evaluator-launcher. Covers running evaluations, checking status and live progress, debugging failed runs, exporting artifacts and logs, and analyzing results. ALWAYS triggers on mentions of running evaluations, checking progress, debugging failed evals, analyzing or analysing runs or results, run directories or artifact paths on clusters, Slurm job issues, invocation IDs, or inspecting logs (client logs, server logs, SSH to cluster, tail logs, grep logs). Do NOT use for creating or modifying evaluation configs.
02 资源文件 8
- 参考文档references/analyze-results.md4.5 KB
- 参考文档references/benchmarks/swebench-general-info.md11.5 KB
- 参考文档references/benchmarks/terminal-bench-general-info.md7.7 KB
- 参考文档references/benchmarks/terminal-bench-trace-analysis.md5.7 KB
- 参考文档references/check-progress.md1.2 KB
- 参考文档references/debug-failed-runs.md6.7 KB
- 参考文档references/run-evaluation.md1.1 KB
- 参考文档tests.json2.7 KB
03 SKILL.md 正文
NeMo Evaluator Skill
Quick Reference
nemo-evaluator-launcher CLI
# Run evaluation
uv run nemo-evaluator-launcher run --config <path.yaml>
uv run nemo-evaluator-launcher run --config <path.yaml> -t <a_single_task_to_be_run_by_name>
uv run nemo-evaluator-launcher run --config <path.yaml> -t <task_name_1> -t <task_name_2> ...
uv run nemo-evaluator-launcher run --config <path.yaml> -o evaluation.nemo_evaluator_config.config.params.limit_samples=10 ...
# Preview the resolved config and the sbatch script without running the evaluation
uv run nemo-evaluator-launcher run --config <path.yaml> --dry-run
# Check status (--json for machine-readable output)
uv run nemo-evaluator-launcher status <invocation_id> --json
# Get evaluation run info (output paths, slurm job IDs, cluster hostname, etc.)
uv run nemo-evaluator-launcher info <invocation_id>
# Copy just the logs (quick — good for debugging)
uv run nemo-evaluator-launcher info <invocation_id> --copy-logs ./evaluation-results/
# For artifacts: use `nel info` to discover paths. If remote, SSH to explore and rsync what you need.
# If local, just read directly from the paths shown by `nel info`.
# ssh <user>@<hostname> "ls <artifacts_path>/"
# rsync -avzP <user>@<hostname>:<artifacts_path>/{results.yml,eval_factory_metrics.json,config.yml} ./evaluation-results/<invocation_id>.<job_index>/artifacts/
# Resume a failed/interrupted run (re-sbatches existing run.sub in the original run directory)
uv run nemo-evaluator-launcher resume <invocation_id>
# List past runs
uv run nemo-evaluator-launcher ls runs --since 1d
# List available evaluation tasks (by default, only shows tasks from the latest released containers)
uv run nemo-evaluator-launcher ls tasks
uv run nemo-evaluator-launcher ls tasks --from_container nvcr.io/nvidia/eval-factory/simple-evals:26.03
Workflow
The complete evaluation workflow is divided into the following steps you should follow IN ORDER.
- Create or modify a config using the
nel-assistantskill. If the user provides a past run, use itsconfig.ymlartifact as a starting point. - Run the evaluation. See
references/run-evaluation.mdwhen executing this step. - Monitor progress (MANDATORY after every
nel run): poll status repeatedly until SUCCESS/FAILED. Seereferences/check-progress.md. - Post-run actions (when terminal state reached):
- When the evaluation status is
SUCCESS, analyze the results. Seereferences/analyze-results.mdwhen executing this step. - When the evaluation status is
FAILED, debug the failed run. Seereferences/debug-failed-runs.mdwhen executing this step.
- When the evaluation status is
Key Facts
- Benchmark-specific info learned during launching/analyzing evals should be added to
references/benchmarks/ - PPP = Slurm account / project portfolio code (the
accountfield in cluster_config.yaml). When the user says "change PPP to X", update the account value (e.g.,<old_account>→<new_account>). - Slurm job pairs: NEL (nemo-evaluator-launcher) submits paired Slurm jobs — a RUNNING job + a PENDING restart job (for when the 4h walltime expires). Never cancel the pending restart jobs — they are expected and necessary.
- HF cache requirement: For configs with
HF_HUB_OFFLINE=1, models must be pre-downloaded to the HF cache on each cluster before launching. Before running a model on a new cluster, always ask the user if the model is already cached there. If not, on the cluster login node:python3 -m venv hf_cli && source hf_cli/bin/activate && pip install huggingface_hubthenHF_HOME=<your_hf_cache_path> hf download <model>(on lustre-style HPC clusters this is typically under/lustre/.../<group>/users/<username>/cache/huggingface). Without this, vLLM will fail withLocalEntryNotFoundError. data_parallel_sizeis per node:dp_size=1withnum_nodes=8means 8 model instances total (one per node), load-balanced by haproxy. Do NOT interpretdp_sizeas the global replica count.payload_modifierinterceptor: Theparams_to_removelist (e.g.[max_tokens, max_completion_tokens]) strips those fields from the outgoing payload, intentionally lifting output length limits so reasoning models can think as long as they need.- Auto-export git workaround: The export container (
python:3.12-slim) lacksgit. When installing the launcher from a git URL, setauto_export.launcher_install_cmdto install git first (e.g.,apt-get update -qq && apt-get install -qq -y git && pip install "nemo-evaluator-launcher[all] @ git+...#subdirectory=packages/nemo-evaluator-launcher"). - Do NOT use
nemo-evaluator-launcher export --dest local— it only writes a summary JSON (processed_results.json), it does NOT copy actual logs or artifacts despite accepting--copy_logsand--copy-artifactsflags.nel info --copy-artifactsworks but copies everything (very slow for large benchmarks). Preferred approach: usenel infoto discover paths — if local, read directly; if remote, SSH to explore and rsync only what you need. Note thatnel infoprints standard artifacts but benchmarks produce additional artifacts in subdirs — explore to find them.
04 同仓库更多 Skills
modelopt-speculative-decoding
Train, debug, and validate a speculative decoding draft model (EAGLE3, DFlash, DSpark, Domino) through the ModelOpt launcher pipeline. Use when the user wants to add a new model to a draft-training pipeline, asks why a pipeline run failed, wants experiment logs reviewed, or wants to check whether a run's acceptance rate meets threshold. Triggers on "EAGLE3", "DFlash", "DSpark", "Domino", "draft model", "acceptance rate", "speculative decoding pipeline". Do NOT use for quantizing a model (use modelopt-ptq) or serving a checkpoint (use modelopt-deployment).
modelopt-run-agentx
Run the AgentX agentic serving benchmark with the SemiAnalysis harness. Use for AgentX setup, dataset downloads, concurrency sweeps, or serving latency and throughput comparisons.
modelopt-release-cherry-pick
Audit merged bug-fix PRs and release NVBugs for missing cherry-pick labels, then cherry-pick labeled PRs into a release branch and open a PR. Use when asked to "cherry-pick PRs for release/X.Y.Z", "pick PRs to release branch", "verify cherry-pick labels", or "cherry-pick labeled PRs".
modelopt-quant-recipe-search
Use when the user asks to find, search for, or optimize the best quantization recipe for a model, including direct requests like "find the best quantization recipe and generate a PTQ checkpoint." Guides the multi-candidate loop: choose compute-vs-memory success metrics, select ModelOpt recipe baselines, design AutoQuant/manual recipe deltas, interpret sensitivity, and decide next candidates. Do NOT use for a single known PTQ recipe run (use modelopt-ptq), serving (use modelopt-deployment), creating/running evals (use modelopt-evaluation or modelopt-launching-evals), monitoring jobs (use modelopt-monitor), MLflow browsing (use modelopt-accessing-mlflow), or comparing completed baseline-vs-candidate scores only (use modelopt-compare-results).
modelopt-qad
Run explicitly requested ModelOpt Quantization-Aware Distillation (QAD) through Megatron Bridge (recommended) or Hugging Face Transformers to recover a measured BF16-to-PTQ accuracy gap. Use only when the user explicitly asks for QAD, including its topology, data preparation, launch, resume, checkpoint export, or recovery decisions.
modelopt-ptq
Use when the user asks to "quantize a model", "run PTQ", "post-training quantization", "NVFP4 quantization", "FP8 quantization", "INT8 quantization", "INT4 AWQ", "quantize LLM", "quantize MoE", "quantize VLM", or needs to produce a quantized HuggingFace checkpoint from a pretrained model using ModelOpt. Do NOT use for multi-candidate recipe exploration or optimization (use modelopt-quant-recipe-search).