[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7B5VT1Q0S4ADWDXN52KTBQP":3,"skill-same:01M7B5VT1Q0S4ADWDXN52KTBQP":87},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":24,"commitSha":25,"skillDirectory":26,"forksCount":27,"repositoryHtmlUrl":28,"repositoryHomepageUrl":29,"ownerHtmlUrl":30,"ownerBlogUrl":31,"ownerDescription":6,"updatedAt":32,"install":33,"resources":46},"01M7B5VT1Q0S4ADWDXN52KTBQP","modelopt-deployment",null,"Serve a quantized or unquantized LLM checkpoint as an OpenAI-compatible API endpoint using vLLM, SGLang, or TRT-LLM. Use when user says \"deploy model\", \"serve model\", \"start vLLM server\", \"launch SGLang\", \"TRT-LLM deploy\", \"AutoDeploy\", \"benchmark throughput\", \"serve checkpoint\", or needs an inference endpoint from a HuggingFace or ModelOpt-quantized checkpoint. Do NOT use for quantizing models (use modelopt-ptq) or evaluating accuracy (use modelopt-evaluation).","NVIDIA\u002FModel-Optimizer","NVIDIA","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F1728152?v=4","01M6BXXZYTRNBXSTRXNNBB356H","NVIDIA Corporation","01M6BXY02X3PRVHNNATDHA2D0W",5604,"Apache-2.0","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FModel-Optimizer\u002Fblob\u002Fmain\u002Fskills\u002Fmodelopt-deployment\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: modelopt-deployment\ndescription: Serve a quantized or unquantized LLM checkpoint as an OpenAI-compatible API endpoint using vLLM, SGLang, or TRT-LLM. Use when user says \"deploy model\", \"serve model\", \"start vLLM server\", \"launch SGLang\", \"TRT-LLM deploy\", \"AutoDeploy\", \"benchmark throughput\", \"serve checkpoint\", or needs an inference endpoint from a HuggingFace or ModelOpt-quantized checkpoint. Do NOT use for quantizing models (use modelopt-ptq) or evaluating accuracy (use modelopt-evaluation).\nlicense: Apache-2.0\n---\n\n# Deployment Skill\n\nServe a model checkpoint as an OpenAI-compatible inference endpoint. Supports vLLM, SGLang, and TRT-LLM (including AutoDeploy).\n\n## Quick Start\n\nPrefer `$SKILL_DIR\u002Fscripts\u002Fdeploy.sh` for standard local deployments — it handles quant detection, health checks, and server lifecycle. Use the raw framework commands in Step 4 when you need flags the script doesn't support, or for remote deployment.\n\n```bash\n# Start vLLM server with a ModelOpt checkpoint\n\"$SKILL_DIR\u002Fscripts\u002Fdeploy.sh\" start --model .\u002Fqwen3-0.6b-fp8\n\n# Start with SGLang and tensor parallelism\n\"$SKILL_DIR\u002Fscripts\u002Fdeploy.sh\" start --model .\u002Fllama-70b-nvfp4 --framework sglang --tp 4\n\n# Start from HuggingFace hub\n\"$SKILL_DIR\u002Fscripts\u002Fdeploy.sh\" start --model nvidia\u002FLlama-3.1-8B-Instruct-FP8\n\n# Test the API\n\"$SKILL_DIR\u002Fscripts\u002Fdeploy.sh\" test\n\n# Check status\n\"$SKILL_DIR\u002Fscripts\u002Fdeploy.sh\" status\n\n# Stop\n\"$SKILL_DIR\u002Fscripts\u002Fdeploy.sh\" stop\n```\n\nThe script handles: GPU detection, quantization flag auto-detection (FP8 vs FP4), server lifecycle (start\u002Fstop\u002Frestart\u002Fstatus), health check polling, and API testing.\n\n## Decision Flow\n\n### 0. Check workspace (multi-user \u002F Slack bot)\n\nIf `MODELOPT_WORKSPACE_ROOT` is set, use the common skill's `workspace-management.md`. Before creating a new workspace, check the current session for existing model workspaces — especially if deploying a checkpoint from a prior PTQ run:\n\n```bash\nls \"$MODELOPT_WORKSPACE_ROOT\u002F\u003Csession_id>\u002F\" 2>\u002Fdev\u002Fnull\n```\n\nIf the user says \"deploy the model I just quantized\" or references a previous PTQ, find the matching workspace and `cd` into it. The checkpoint should be in that workspace's output directory.\n\n### 1. Identify the checkpoint\n\nDetermine what the user wants to deploy:\n\n- **Local quantized checkpoint** (from ptq skill or manual export): look for `hf_quant_config.json` in the directory. If coming from a prior PTQ run in the same workspace, check common output locations: `output\u002F`, `outputs\u002F`, `exported_model\u002F`, or the `--export_path` used in the PTQ command.\n- **HuggingFace model hub** (e.g., `nvidia\u002FLlama-3.1-8B-Instruct-FP8`): use directly\n- **Unquantized model**: deploy as-is (BF16) or suggest quantizing first with the ptq skill\n\n> **Note:** This skill expects HF-format checkpoints (from PTQ with `--export_fmt hf`). TRT-LLM format checkpoints should be deployed directly with TRT-LLM — see `references\u002Ftrtllm.md`.\n\nCheck the quantization format if applicable:\n\n```bash\ncat \u003Ccheckpoint_path>\u002Fhf_quant_config.json 2>\u002Fdev\u002Fnull || echo \"No hf_quant_config.json\"\n```\n\nIf not found, also check `config.json` for a `quantization_config` section with `quant_method: \"modelopt\"`. If neither exists, the checkpoint is unquantized.\n\n### 2. Choose the framework\n\nIf the user hasn't specified a framework, recommend based on this priority:\n\n| Situation | Recommended | Why |\n|-----------|-------------|-----|\n| General use | **vLLM** | Widest ecosystem, easy setup, OpenAI-compatible |\n| Best SGLang model support | **SGLang** | Strong DeepSeek\u002FLlama 4 support |\n| Maximum optimization | **TRT-LLM** | Best throughput via engine compilation |\n| Mixed-precision \u002F AutoQuant | **TRT-LLM AutoDeploy** | Only option for AutoQuant checkpoints |\n\nCheck the support matrix in `references\u002Fsupport-matrix.md` to confirm the model + format + framework combination is supported.\n\n### 3. Check the environment\n\nUse the common skill's `environment-setup.md` for GPU detection, local vs remote, and SLURM\u002FDocker\u002Fbare metal detection. After completing it you should know: GPU model\u002Fcount, local or remote, and execution environment.\n\nThen check the **deployment framework** is installed:\n\n```bash\npython -c \"import vllm; print(f'vLLM {vllm.__version__}')\" 2>\u002Fdev\u002Fnull || echo \"vLLM not installed\"\npython -c \"import sglang; print(f'SGLang {sglang.__version__}')\" 2>\u002Fdev\u002Fnull || echo \"SGLang not installed\"\npython -c \"import tensorrt_llm; print(f'TRT-LLM {tensorrt_llm.__version__}')\" 2>\u002Fdev\u002Fnull || echo \"TRT-LLM not installed\"\n```\n\nIf not installed, consult `references\u002Fsetup.md`.\n\n**GPU memory estimate** (to determine tensor parallelism):\n\n- BF16: `params × 2 bytes` (8B ≈ 16 GB)\n- FP8: `params × 1 byte` (8B ≈ 8 GB)\n- FP4: `params × 0.5 bytes` (8B ≈ 4 GB)\n- Add ~2-4 GB for KV cache and framework overhead\n\nIf the model exceeds single GPU memory, use tensor parallelism (`-tp \u003Cnum_gpus>`).\n\n### 4. Deploy\n\nRead the framework-specific reference for detailed instructions:\n\n| Framework | Reference file |\n|-----------|---------------|\n| vLLM | `references\u002Fvllm.md` |\n| SGLang | `references\u002Fsglang.md` |\n| TRT-LLM | `references\u002Ftrtllm.md` |\n\n**Quick-start commands** (for common cases):\n\n#### vLLM\n\n```bash\n# Serve as OpenAI-compatible endpoint\npython -m vllm.entrypoints.openai.api_server \\\n    --model \u003Ccheckpoint_path> \\\n    --quantization modelopt \\\n    --tensor-parallel-size \u003Cnum_gpus> \\\n    --host 0.0.0.0 --port 8000\n```\n\nFor NVFP4 checkpoints, use `--quantization modelopt_fp4`.\n\n> **NVFP4 on Blackwell B300\u002FGB300 (sm_103) needs a CUDA-13 image.** From v0.20.0\n> on, release tags are CUDA-13 unsuffixed (e.g. `vllm\u002Fvllm-openai:v0.26.0`) with\n> `-cu129` the CUDA-12 opt-out; v0.19.x and earlier were the other way round\n> (`-cu130` = CUDA 13), and `-cu130` no longer exists after v0.20.0. Don't trust\n> the tag name — select a tag reporting `CUDA_VERSION` >= 13 in the config blob\n> of your platform's child manifest (arm64 Grace\u002FGB300, amd64 x86);\n> `TORCH_CUDA_ARCH_LIST` differs between the two. A cu12 build has **no sm_103\n> FP4 kernel**, so vLLM loads the checkpoint then dies at engine init with `CUDA\n> error: no kernel image is available for execution on the device` (affects the\n> `flashinfer` and `cutlass` NVFP4 backends; `marlin` separately fails on\n> non-64-divisible layer dims).\n> Cross-check via\n> `recipes.vllm.ai\u002F\u003Corg>\u002F\u003Cmodel>?hardware=b300` (JS-rendered — fetch the raw\n> markdown at `github.com\u002Fvllm-project\u002Frecipes\u002Fblob\u002Fmain\u002F\u003Corg>\u002F\u003Cmodel>.md`). For\n> multimodal models on sm_103, also pass `--mm-encoder-attn-backend TRITON_ATTN`\n> (the default CuTe ViT flash-attn asserts \"Only SM 10.x and 11.x\").\n\n#### SGLang\n\n```bash\npython -m sglang.launch_server \\\n    --model-path \u003Ccheckpoint_path> \\\n    --quantization modelopt \\\n    --tp \u003Cnum_gpus> \\\n    --host 0.0.0.0 --port 8000\n```\n\nFor NVFP4 checkpoints, use `--quantization modelopt_fp4`.\n\n> **Cross-check SGLang launch flags via the SGLang cookbook** (the SGLang analog\n> of `recipes.vllm.ai`): `docs.sglang.io\u002Fcookbook\u002F\u003Ccategory>\u002F\u003Corg>\u002F\u003Cmodel>` (e.g.\n> `...\u002Fautoregressive\u002FDeepSeek\u002FDeepSeek-V4`) — authoritative for parallelism, MoE\n> backends, strategy flags, Docker image, and min version. Select the variant via\n> the URL fragment `#hw=...&variant=...&quant=...&strategy=...&nodes=...`. The\n> page is **JS-rendered** — fetch the raw markdown at\n> `raw.githubusercontent.com\u002Fsgl-project\u002Fsglang\u002Fmain\u002Fdocs_new\u002Fcookbook\u002F\u003Ccategory>\u002F\u003Corg>\u002F\u003Cmodel>.mdx`.\n> SM120 (RTX PRO 6000) needs the `lmsysorg\u002Fsglang:dev` nightly (`:latest` lacks\n> SM120). See `references\u002Fsglang.md` for the full backend\u002Fflag matrix.\n\n#### TRT-LLM (direct)\n\n```python\nfrom tensorrt_llm import LLM, SamplingParams\nllm = LLM(model=\"\u003Ccheckpoint_path>\")\noutputs = llm.generate([\"Hello, my name is\"], SamplingParams(temperature=0.8, top_p=0.95))\n```\n\n#### TRT-LLM AutoDeploy\n\nFor AutoQuant or mixed-precision checkpoints, see `references\u002Ftrtllm.md`.\n\n### 5. Verify the deployment\n\nAfter the server starts, verify it's healthy:\n\n```bash\n# Health check\ncurl -s http:\u002F\u002Flocalhost:8000\u002Fhealth\n\n# List models\ncurl -s http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fmodels | python -m json.tool\n\n# Test generation\ncurl -s http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fcompletions \\\n    -H \"Content-Type: application\u002Fjson\" \\\n    -d '{\n        \"model\": \"\u003Cmodel_name>\",\n        \"prompt\": \"The capital of France is\",\n        \"max_tokens\": 32\n    }' | python -m json.tool\n```\n\nAll checks must pass before reporting success to the user.\n\n### 5b. Benchmark throughput\u002Flatency (optional)\n\nIf the user asks to benchmark, measure throughput\u002Flatency, or compare precisions,\nuse **AIPerf** (Apache-2.0, OpenAI-compatible client benchmark). See\n`references\u002Fbenchmarking.md` for install, the pre-benchmark coherence gate, the\n`aiperf profile` flags (notably `--extra-inputs ignore_eos:true`), suggested\ntoken shapes, and how to read `profile_export_aiperf.json`.\n\n### 6. Remote deployment (SSH\u002FSLURM)\n\nIf a cluster config exists (`~\u002F.config\u002Fmodelopt\u002Fclusters.yaml`, `.agents\u002Fclusters.yaml`, or `.claude\u002Fclusters.yaml`), or the user mentions running on a remote machine:\n\n0. **Check container registry auth** — before submitting any SLURM job with a container image, verify credentials exist on the cluster per the common skill's `slurm-setup.md` section 6. If credentials are missing for the image's registry, ask the user to fix auth or switch to an image on an authenticated registry (e.g., NGC). **Do not submit until auth is confirmed.**\n\n1. **Source remote utilities:** Load the common skill, then resolve\n   `remote_exec.sh` from that skill's root.\n\n   ```bash\n   source \"\u003Ccommon-skill-dir>\u002Fremote_exec.sh\"\n   remote_load_cluster\n   remote_check_ssh\n   remote_detect_env\n   ```\n\n2. **Sync the checkpoint** (only if it was produced locally):\n\n   If the checkpoint path is a remote\u002Fabsolute path (e.g., from a prior PTQ run on the cluster), skip sync — it's already there. Verify with `remote_run \"ls \u003Ccheckpoint_path>\u002Fconfig.json\"`. Only sync if the checkpoint is local:\n\n   ```bash\n   remote_sync_to \u003Clocal_checkpoint_path> \u003Csession_id>\u002F\u003Cmodel>\u002Fcheckpoints\u002F\n   ```\n\n3. **Deploy based on remote environment:**\n\n   - **SLURM** — see the common skill's `slurm-setup.md` for job script templates (container setup, account\u002Fpartition discovery). The server command inside the container is the same as Step 4 (e.g., `python -m vllm.entrypoints.openai.api_server --model \u003Cpath> --quantization modelopt`). After submitting, register the job and set up monitoring per the **modelopt-monitor skill**. Get the node hostname from `squeue -j $JOBID -o %N`.\n\n   - **Bare metal \u002F Docker** — use `remote_run` to start the server directly:\n\n     ```bash\n     remote_run \"nohup python -m vllm.entrypoints.openai.api_server --model \u003Cpath> --port 8000 > deploy.log 2>&1 &\"\n     ```\n\n4. **Verify remotely:**\n\n   ```bash\n   remote_run \"curl -s http:\u002F\u002Flocalhost:8000\u002Fhealth\"\n   remote_run \"curl -s http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fmodels\"\n   ```\n\n5. **Report the endpoint** — include the remote hostname and port so the user can connect (e.g., `http:\u002F\u002F\u003Cnode_hostname>:8000`). For SLURM, note that the port is only reachable from within the cluster network.\n\nFor NEL-managed deployment (evaluation with self-deployment), use the evaluation skill instead — NEL handles SLURM container deployment, health checks, and teardown automatically.\n\n## Error Handling\n\n| Error | Cause | Fix |\n|-------|-------|-----|\n| `CUDA error: an illegal memory access` on an **NVFP4 MoE** (`trtllm_fused_moe_dev_kernel.cu`, `deepgemm`) | Fused-MoE FP4 kernel fault on long-context loads; engine dies, all requests 500 | Try `VLLM_USE_FLASHINFER_MOE_FP4=1` + `VLLM_FLASHINFER_MOE_BACKEND=throughput`. **Not always a fix** — on DeepSeek-V4 it moved the fault from TRT-LLM to DeepGEMM. It also makes quantized vs baseline not kernel-matched; record that when reporting deltas. |\n| `CUDA out of memory` | Model too large for GPU(s) | Increase `--tensor-parallel-size` or use a smaller model |\n| `quantization=\"modelopt\" not recognized` | vLLM\u002FSGLang version too old | Upgrade: vLLM >= 0.10.1, SGLang >= 0.4.10 |\n| `hf_quant_config.json not found` | Not a ModelOpt-exported checkpoint | Re-export with `export_hf_checkpoint()`, or remove `--quantization` flag |\n| `Connection refused` on health check | Server still starting | Wait 30-60s for large models; check logs for errors |\n| `modelopt_fp4 not supported` | Framework doesn't support FP4 for this model | Check support matrix in `references\u002Fsupport-matrix.md` |\n\n## Unsupported Models\n\nIf the model is not in the validated support matrix (`references\u002Fsupport-matrix.md`), deployment may fail due to weight key mismatches, missing architecture mappings, or quantized\u002Funquantized layer confusion. Read `references\u002Funsupported-models.md` for the iterative debug loop: **run → read error → diagnose → patch framework source → re-run**. For kernel-level issues, escalate to the framework team rather than attempting fixes.\n\n## Success Criteria\n\n1. Server process is running and healthy (`\u002Fhealth` returns 200)\n2. Model is listed at `\u002Fv1\u002Fmodels`\n3. Test generation produces coherent output\n4. Server URL and port are reported to the user\n5. If benchmarking was requested, throughput\u002Flatency numbers are reported\n","# Deployment Skill\n\nServe a model checkpoint as an OpenAI-compatible inference endpoint. Supports vLLM, SGLang, and TRT-LLM (including AutoDeploy).\n\n## Quick Start\n\nPrefer `$SKILL_DIR\u002Fscripts\u002Fdeploy.sh` for standard local deployments — it handles quant detection, health checks, and server lifecycle. Use the raw framework commands in Step 4 when you need flags the script doesn't support, or for remote deployment.\n\n```bash\n# Start vLLM server with a ModelOpt checkpoint\n\"$SKILL_DIR\u002Fscripts\u002Fdeploy.sh\" start --model .\u002Fqwen3-0.6b-fp8\n\n# Start with SGLang and tensor parallelism\n\"$SKILL_DIR\u002Fscripts\u002Fdeploy.sh\" start --model .\u002Fllama-70b-nvfp4 --framework sglang --tp 4\n\n# Start from HuggingFace hub\n\"$SKILL_DIR\u002Fscripts\u002Fdeploy.sh\" start --model nvidia\u002FLlama-3.1-8B-Instruct-FP8\n\n# Test the API\n\"$SKILL_DIR\u002Fscripts\u002Fdeploy.sh\" test\n\n# Check status\n\"$SKILL_DIR\u002Fscripts\u002Fdeploy.sh\" status\n\n# Stop\n\"$SKILL_DIR\u002Fscripts\u002Fdeploy.sh\" stop\n```\n\nThe script handles: GPU detection, quantization flag auto-detection (FP8 vs FP4), server lifecycle (start\u002Fstop\u002Frestart\u002Fstatus), health check polling, and API testing.\n\n## Decision Flow\n\n### 0. Check workspace (multi-user \u002F Slack bot)\n\nIf `MODELOPT_WORKSPACE_ROOT` is set, use the common skill's `workspace-management.md`. Before creating a new workspace, check the current session for existing model workspaces — especially if deploying a checkpoint from a prior PTQ run:\n\n```bash\nls \"$MODELOPT_WORKSPACE_ROOT\u002F\u003Csession_id>\u002F\" 2>\u002Fdev\u002Fnull\n```\n\nIf the user says \"deploy the model I just quantized\" or references a previous PTQ, find the matching workspace and `cd` into it. The checkpoint should be in that workspace's output directory.\n\n### 1. Identify the checkpoint\n\nDetermine what the user wants to deploy:\n\n- **Local quantized checkpoint** (from ptq skill or manual export): look for `hf_quant_config.json` in the directory. If coming from a prior PTQ run in the same workspace, check common output locations: `output\u002F`, `outputs\u002F`, `exported_model\u002F`, or the `--export_path` used in the PTQ command.\n- **HuggingFace model hub** (e.g., `nvidia\u002FLlama-3.1-8B-Instruct-FP8`): use directly\n- **Unquantized model**: deploy as-is (BF16) or suggest quantizing first with the ptq skill\n\n> **Note:** This skill expects HF-format checkpoints (from PTQ with `--export_fmt hf`). TRT-LLM format checkpoints should be deployed directly with TRT-LLM — see `references\u002Ftrtllm.md`.\n\nCheck the quantization format if applicable:\n\n```bash\ncat \u003Ccheckpoint_path>\u002Fhf_quant_config.json 2>\u002Fdev\u002Fnull || echo \"No hf_quant_config.json\"\n```\n\nIf not found, also check `config.json` for a `quantization_config` section with `quant_method: \"modelopt\"`. If neither exists, the checkpoint is unquantized.\n\n### 2. Choose the framework\n\nIf the user hasn't specified a framework, recommend based on this priority:\n\n| Situation | Recommended | Why |\n|-----------|-------------|-----|\n| General use | **vLLM** | Widest ecosystem, easy setup, OpenAI-compatible |\n| Best SGLang model support | **SGLang** | Strong DeepSeek\u002FLlama 4 support |\n| Maximum optimization | **TRT-LLM** | Best throughput via engine compilation |\n| Mixed-precision \u002F AutoQuant | **TRT-LLM AutoDeploy** | Only option for AutoQuant checkpoints |\n\nCheck the support matrix in `references\u002Fsupport-matrix.md` to confirm the model + format + framework combination is supported.\n\n### 3. Check the environment\n\nUse the common skill's `environment-setup.md` for GPU detection, local vs remote, and SLURM\u002FDocker\u002Fbare metal detection. After completing it you should know: GPU model\u002Fcount, local or remote, and execution environment.\n\nThen check the **deployment framework** is installed:\n\n```bash\npython -c \"import vllm; print(f'vLLM {vllm.__version__}')\" 2>\u002Fdev\u002Fnull || echo \"vLLM not installed\"\npython -c \"import sglang; print(f'SGLang {sglang.__version__}')\" 2>\u002Fdev\u002Fnull || echo \"SGLang not installed\"\npython -c \"import tensorrt_llm; print(f'TRT-LLM {tensorrt_llm.__version__}')\" 2>\u002Fdev\u002Fnull || echo \"TRT-LLM not installed\"\n```\n\nIf not installed, consult `references\u002Fsetup.md`.\n\n**GPU memory estimate** (to determine tensor parallelism):\n\n- BF16: `params × 2 bytes` (8B ≈ 16 GB)\n- FP8: `params × 1 byte` (8B ≈ 8 GB)\n- FP4: `params × 0.5 bytes` (8B ≈ 4 GB)\n- Add ~2-4 GB for KV cache and framework overhead\n\nIf the model exceeds single GPU memory, use tensor parallelism (`-tp \u003Cnum_gpus>`).\n\n### 4. Deploy\n\nRead the framework-specific reference for detailed instructions:\n\n| Framework | Reference file |\n|-----------|---------------|\n| vLLM | `references\u002Fvllm.md` |\n| SGLang | `references\u002Fsglang.md` |\n| TRT-LLM | `references\u002Ftrtllm.md` |\n\n**Quick-start commands** (for common cases):\n\n#### vLLM\n\n```bash\n# Serve as OpenAI-compatible endpoint\npython -m vllm.entrypoints.openai.api_server \\\n    --model \u003Ccheckpoint_path> \\\n    --quantization modelopt \\\n    --tensor-parallel-size \u003Cnum_gpus> \\\n    --host 0.0.0.0 --port 8000\n```\n\nFor NVFP4 checkpoints, use `--quantization modelopt_fp4`.\n\n> **NVFP4 on Blackwell B300\u002FGB300 (sm_103) needs a CUDA-13 image.** From v0.20.0\n> on, release tags are CUDA-13 unsuffixed (e.g. `vllm\u002Fvllm-openai:v0.26.0`) with\n> `-cu129` the CUDA-12 opt-out; v0.19.x and earlier were the other way round\n> (`-cu130` = CUDA 13), and `-cu130` no longer exists after v0.20.0. Don't trust\n> the tag name — select a tag reporting `CUDA_VERSION` >= 13 in the config blob\n> of your platform's child manifest (arm64 Grace\u002FGB300, amd64 x86);\n> `TORCH_CUDA_ARCH_LIST` differs between the two. A cu12 build has **no sm_103\n> FP4 kernel**, so vLLM loads the checkpoint then dies at engine init with `CUDA\n> error: no kernel image is available for execution on the device` (affects the\n> `flashinfer` and `cutlass` NVFP4 backends; `marlin` separately fails on\n> non-64-divisible layer dims).\n> Cross-check via\n> `recipes.vllm.ai\u002F\u003Corg>\u002F\u003Cmodel>?hardware=b300` (JS-rendered — fetch the raw\n> markdown at `github.com\u002Fvllm-project\u002Frecipes\u002Fblob\u002Fmain\u002F\u003Corg>\u002F\u003Cmodel>.md`). For\n> multimodal models on sm_103, also pass `--mm-encoder-attn-backend TRITON_ATTN`\n> (the default CuTe ViT flash-attn asserts \"Only SM 10.x and 11.x\").\n\n#### SGLang\n\n```bash\npython -m sglang.launch_server \\\n    --model-path \u003Ccheckpoint_path> \\\n    --quantization modelopt \\\n    --tp \u003Cnum_gpus> \\\n    --host 0.0.0.0 --port 8000\n```\n\nFor NVFP4 checkpoints, use `--quantization modelopt_fp4`.\n\n> **Cross-check SGLang launch flags via the SGLang cookbook** (the SGLang analog\n> of `recipes.vllm.ai`): `docs.sglang.io\u002Fcookbook\u002F\u003Ccategory>\u002F\u003Corg>\u002F\u003Cmodel>` (e.g.\n> `...\u002Fautoregressive\u002FDeepSeek\u002FDeepSeek-V4`) — authoritative for parallelism, MoE\n> backends, strategy flags, Docker image, and min version. Select the variant via\n> the URL fragment `#hw=...&variant=...&quant=...&strategy=...&nodes=...`. The\n> page is **JS-rendered** — fetch the raw markdown at\n> `raw.githubusercontent.com\u002Fsgl-project\u002Fsglang\u002Fmain\u002Fdocs_new\u002Fcookbook\u002F\u003Ccategory>\u002F\u003Corg>\u002F\u003Cmodel>.mdx`.\n> SM120 (RTX PRO 6000) needs the `lmsysorg\u002Fsglang:dev` nightly (`:latest` lacks\n> SM120). See `references\u002Fsglang.md` for the full backend\u002Fflag matrix.\n\n#### TRT-LLM (direct)\n\n```python\nfrom tensorrt_llm import LLM, SamplingParams\nllm = LLM(model=\"\u003Ccheckpoint_path>\")\noutputs = llm.generate([\"Hello, my name is\"], SamplingParams(temperature=0.8, top_p=0.95))\n```\n\n#### TRT-LLM AutoDeploy\n\nFor AutoQuant or mixed-precision checkpoints, see `references\u002Ftrtllm.md`.\n\n### 5. Verify the deployment\n\nAfter the server starts, verify it's healthy:\n\n```bash\n# Health check\ncurl -s http:\u002F\u002Flocalhost:8000\u002Fhealth\n\n# List models\ncurl -s http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fmodels | python -m json.tool\n\n# Test generation\ncurl -s http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fcompletions \\\n    -H \"Content-Type: application\u002Fjson\" \\\n    -d '{\n        \"model\": \"\u003Cmodel_name>\",\n        \"prompt\": \"The capital of France is\",\n        \"max_tokens\": 32\n    }' | python -m json.tool\n```\n\nAll checks must pass before reporting success to the user.\n\n### 5b. Benchmark throughput\u002Flatency (optional)\n\nIf the user asks to benchmark, measure throughput\u002Flatency, or compare precisions,\nuse **AIPerf** (Apache-2.0, OpenAI-compatible client benchmark). See\n`references\u002Fbenchmarking.md` for install, the pre-benchmark coherence gate, the\n`aiperf profile` flags (notably `--extra-inputs ignore_eos:true`), suggested\ntoken shapes, and how to read `profile_export_aiperf.json`.\n\n### 6. Remote deployment (SSH\u002FSLURM)\n\nIf a cluster config exists (`~\u002F.config\u002Fmodelopt\u002Fclusters.yaml`, `.agents\u002Fclusters.yaml`, or `.claude\u002Fclusters.yaml`), or the user mentions running on a remote machine:\n\n0. **Check container registry auth** — before submitting any SLURM job with a container image, verify credentials exist on the cluster per the common skill's `slurm-setup.md` section 6. If credentials are missing for the image's registry, ask the user to fix auth or switch to an image on an authenticated registry (e.g., NGC). **Do not submit until auth is confirmed.**\n\n1. **Source remote utilities:** Load the common skill, then resolve\n   `remote_exec.sh` from that skill's root.\n\n   ```bash\n   source \"\u003Ccommon-skill-dir>\u002Fremote_exec.sh\"\n   remote_load_cluster\n   remote_check_ssh\n   remote_detect_env\n   ```\n\n2. **Sync the checkpoint** (only if it was produced locally):\n\n   If the checkpoint path is a remote\u002Fabsolute path (e.g., from a prior PTQ run on the cluster), skip sync — it's already there. Verify with `remote_run \"ls \u003Ccheckpoint_path>\u002Fconfig.json\"`. Only sync if the checkpoint is local:\n\n   ```bash\n   remote_sync_to \u003Clocal_checkpoint_path> \u003Csession_id>\u002F\u003Cmodel>\u002Fcheckpoints\u002F\n   ```\n\n3. **Deploy based on remote environment:**\n\n   - **SLURM** — see the common skill's `slurm-setup.md` for job script templates (container setup, account\u002Fpartition discovery). The server command inside the container is the same as Step 4 (e.g., `python -m vllm.entrypoints.openai.api_server --model \u003Cpath> --quantization modelopt`). After submitting, register the job and set up monitoring per the **modelopt-monitor skill**. Get the node hostname from `squeue -j $JOBID -o %N`.\n\n   - **Bare metal \u002F Docker** — use `remote_run` to start the server directly:\n\n     ```bash\n     remote_run \"nohup python -m vllm.entrypoints.openai.api_server --model \u003Cpath> --port 8000 > deploy.log 2>&1 &\"\n     ```\n\n4. **Verify remotely:**\n\n   ```bash\n   remote_run \"curl -s http:\u002F\u002Flocalhost:8000\u002Fhealth\"\n   remote_run \"curl -s http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fmodels\"\n   ```\n\n5. **Report the endpoint** — include the remote hostname and port so the user can connect (e.g., `http:\u002F\u002F\u003Cnode_hostname>:8000`). For SLURM, note that the port is only reachable from within the cluster network.\n\nFor NEL-managed deployment (evaluation with self-deployment), use the evaluation skill instead — NEL handles SLURM container deployment, health checks, and teardown automatically.\n\n## Error Handling\n\n| Error | Cause | Fix |\n|-------|-------|-----|\n| `CUDA error: an illegal memory access` on an **NVFP4 MoE** (`trtllm_fused_moe_dev_kernel.cu`, `deepgemm`) | Fused-MoE FP4 kernel fault on long-context loads; engine dies, all requests 500 | Try `VLLM_USE_FLASHINFER_MOE_FP4=1` + `VLLM_FLASHINFER_MOE_BACKEND=throughput`. **Not always a fix** — on DeepSeek-V4 it moved the fault from TRT-LLM to DeepGEMM. It also makes quantized vs baseline not kernel-matched; record that when reporting deltas. |\n| `CUDA out of memory` | Model too large for GPU(s) | Increase `--tensor-parallel-size` or use a smaller model |\n| `quantization=\"modelopt\" not recognized` | vLLM\u002FSGLang version too old | Upgrade: vLLM >= 0.10.1, SGLang >= 0.4.10 |\n| `hf_quant_config.json not found` | Not a ModelOpt-exported checkpoint | Re-export with `export_hf_checkpoint()`, or remove `--quantization` flag |\n| `Connection refused` on health check | Server still starting | Wait 30-60s for large models; check logs for errors |\n| `modelopt_fp4 not supported` | Framework doesn't support FP4 for this model | Check support matrix in `references\u002Fsupport-matrix.md` |\n\n## Unsupported Models\n\nIf the model is not in the validated support matrix (`references\u002Fsupport-matrix.md`), deployment may fail due to weight key mismatches, missing architecture mappings, or quantized\u002Funquantized layer confusion. Read `references\u002Funsupported-models.md` for the iterative debug loop: **run → read error → diagnose → patch framework source → re-run**. For kernel-level issues, escalate to the framework team rather than attempting fixes.\n\n## Success Criteria\n\n1. Server process is running and healthy (`\u002Fhealth` returns 200)\n2. Model is listed at `\u002Fv1\u002Fmodels`\n3. Test generation produces coherent output\n4. Server URL and port are reported to the user\n5. If benchmarking was requested, throughput\u002Flatency numbers are reported\n",{"name":5,"license":15,"description":7},"main","e169b4c2bc62a32c45aaef793cde7599dee2c1b2","skills\u002Fmodelopt-deployment",806,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FModel-Optimizer","https:\u002F\u002Fnvidia.github.io\u002FModel-Optimizer\u002F","https:\u002F\u002Fgithub.com\u002FNVIDIA","https:\u002F\u002Fnvidia.com","2026-11-15 02:45:59",{"source":8,"skillName":5,"commands":34,"prompt":44,"zipAvailable":45},[35,38,41],{"tool":36,"command":37},"npx","npx skills add NVIDIA\u002FModel-Optimizer --skill modelopt-deployment",{"tool":39,"command":40},"bunx","bunx skills add NVIDIA\u002FModel-Optimizer --skill modelopt-deployment",{"tool":42,"command":43},"pnpm","pnpm dlx skills add NVIDIA\u002FModel-Optimizer --skill modelopt-deployment","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FModel-Optimizer\u002Fblob\u002Fmain\u002Fskills\u002Fmodelopt-deployment\u002FSKILL.md",true,[47,53,57,61,65,69,73,77,83],{"path":48,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":51,"sourceHtmlUrl":52},"references\u002Fbenchmarking.md","REFERENCE","参考文档",5841,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FModel-Optimizer\u002Fblob\u002Fmain\u002Fskills\u002Fmodelopt-deployment\u002Freferences\u002Fbenchmarking.md",{"path":54,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":55,"sourceHtmlUrl":56},"references\u002Fsetup.md",2491,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FModel-Optimizer\u002Fblob\u002Fmain\u002Fskills\u002Fmodelopt-deployment\u002Freferences\u002Fsetup.md",{"path":58,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":59,"sourceHtmlUrl":60},"references\u002Fsglang.md",4765,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FModel-Optimizer\u002Fblob\u002Fmain\u002Fskills\u002Fmodelopt-deployment\u002Freferences\u002Fsglang.md",{"path":62,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":63,"sourceHtmlUrl":64},"references\u002Fsupport-matrix.md",3286,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FModel-Optimizer\u002Fblob\u002Fmain\u002Fskills\u002Fmodelopt-deployment\u002Freferences\u002Fsupport-matrix.md",{"path":66,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":67,"sourceHtmlUrl":68},"references\u002Ftrtllm.md",3382,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FModel-Optimizer\u002Fblob\u002Fmain\u002Fskills\u002Fmodelopt-deployment\u002Freferences\u002Ftrtllm.md",{"path":70,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":71,"sourceHtmlUrl":72},"references\u002Funsupported-models.md",4886,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FModel-Optimizer\u002Fblob\u002Fmain\u002Fskills\u002Fmodelopt-deployment\u002Freferences\u002Funsupported-models.md",{"path":74,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":75,"sourceHtmlUrl":76},"references\u002Fvllm.md",2586,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FModel-Optimizer\u002Fblob\u002Fmain\u002Fskills\u002Fmodelopt-deployment\u002Freferences\u002Fvllm.md",{"path":78,"resourceType":79,"resourceTypeDesc":80,"sizeBytes":81,"sourceHtmlUrl":82},"scripts\u002Fdeploy.sh","SCRIPT","脚本",19486,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FModel-Optimizer\u002Fblob\u002Fmain\u002Fskills\u002Fmodelopt-deployment\u002Fscripts\u002Fdeploy.sh",{"path":84,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":85,"sourceHtmlUrl":86},"tests\u002Fevals.json",3082,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FModel-Optimizer\u002Fblob\u002Fmain\u002Fskills\u002Fmodelopt-deployment\u002Ftests\u002Fevals.json",[88,95,102,109,116,123],{"skillNo":89,"name":90,"nameZh":6,"description":91,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":92,"categories":93,"keywords":94,"publishedAt":19},"01M7B5XD9M965H2TAV4ZJMPF6N","modelopt-speculative-decoding","Train, debug, and validate a speculative decoding draft model (EAGLE3, DFlash, DSpark, Domino) through the ModelOpt launcher pipeline. Use when the user wants to add a new model to a draft-training pipeline, asks why a pipeline run failed, wants experiment logs reviewed, or wants to check whether a run's acceptance rate meets threshold. Triggers on \"EAGLE3\", \"DFlash\", \"DSpark\", \"Domino\", \"draft model\", \"acceptance rate\", \"speculative decoding pipeline\". Do NOT use for quantizing a model (use modelopt-ptq) or serving a checkpoint (use modelopt-deployment).","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FModel-Optimizer\u002Fblob\u002Fmain\u002Fskills\u002Fmodelopt-speculative-decoding\u002FSKILL.md",[],[],{"skillNo":96,"name":97,"nameZh":6,"description":98,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":99,"categories":100,"keywords":101,"publishedAt":19},"01M7B5X9SG5FHW174VFK979DQ9","modelopt-run-agentx","Run the AgentX agentic serving benchmark with the SemiAnalysis harness. Use for AgentX setup, dataset downloads, concurrency sweeps, or serving latency and throughput comparisons.","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FModel-Optimizer\u002Fblob\u002Fmain\u002Fskills\u002Fmodelopt-run-agentx\u002FSKILL.md",[],[],{"skillNo":103,"name":104,"nameZh":6,"description":105,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":106,"categories":107,"keywords":108,"publishedAt":19},"01M7B5X4C0XGVA8N0BXF9N4HCX","modelopt-release-cherry-pick","Audit merged bug-fix PRs and release NVBugs for missing cherry-pick labels, then cherry-pick labeled PRs into a release branch and open a PR. Use when asked to \"cherry-pick PRs for release\u002FX.Y.Z\", \"pick PRs to release branch\", \"verify cherry-pick labels\", or \"cherry-pick labeled PRs\".","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FModel-Optimizer\u002Fblob\u002Fmain\u002Fskills\u002Fmodelopt-release-cherry-pick\u002FSKILL.md",[],[],{"skillNo":110,"name":111,"nameZh":6,"description":112,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":113,"categories":114,"keywords":115,"publishedAt":19},"01M7B5WXMDYXCK43YA5ZEF7ECY","modelopt-quant-recipe-search","Use when the user asks to find, search for, or optimize the best quantization recipe for a model, including direct requests like \"find the best quantization recipe and generate a PTQ checkpoint.\" Guides the multi-candidate loop: choose compute-vs-memory success metrics, select ModelOpt recipe baselines, design AutoQuant\u002Fmanual recipe deltas, interpret sensitivity, and decide next candidates. Do NOT use for a single known PTQ recipe run (use modelopt-ptq), serving (use modelopt-deployment), creating\u002Frunning evals (use modelopt-evaluation or modelopt-launching-evals), monitoring jobs (use modelopt-monitor), MLflow browsing (use modelopt-accessing-mlflow), or comparing completed baseline-vs-candidate scores only (use modelopt-compare-results).","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FModel-Optimizer\u002Fblob\u002Fmain\u002Fskills\u002Fmodelopt-quant-recipe-search\u002FSKILL.md",[],[],{"skillNo":117,"name":118,"nameZh":6,"description":119,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":120,"categories":121,"keywords":122,"publishedAt":19},"01M7B5WRMPZE23YXCFMZBM8KJA","modelopt-qad","Run explicitly requested ModelOpt Quantization-Aware Distillation (QAD) through Megatron Bridge (recommended) or Hugging Face Transformers to recover a measured BF16-to-PTQ accuracy gap. Use only when the user explicitly asks for QAD, including its topology, data preparation, launch, resume, checkpoint export, or recovery decisions.","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FModel-Optimizer\u002Fblob\u002Fmain\u002Fskills\u002Fmodelopt-qad\u002FSKILL.md",[],[],{"skillNo":124,"name":125,"nameZh":6,"description":126,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":127,"categories":128,"keywords":129,"publishedAt":19},"01M7B5WJHB4D787EZCY4RNN44T","modelopt-ptq","Use when the user asks to \"quantize a model\", \"run PTQ\", \"post-training quantization\", \"NVFP4 quantization\", \"FP8 quantization\", \"INT8 quantization\", \"INT4 AWQ\", \"quantize LLM\", \"quantize MoE\", \"quantize VLM\", or needs to produce a quantized HuggingFace checkpoint from a pretrained model using ModelOpt. Do NOT use for multi-candidate recipe exploration or optimization (use modelopt-quant-recipe-search).","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FModel-Optimizer\u002Fblob\u002Fmain\u002Fskills\u002Fmodelopt-ptq\u002FSKILL.md",[],[]]