modelopt-speculative-decoding
Train, debug, and validate a speculative decoding draft model (EAGLE3, DFlash, DSpark, Domino) through the ModelOpt launcher pipeline. Use when the user wants to add a new model to a draft-training pipeline, asks why a pipeline run failed, wants experiment logs reviewed, or wants to check whether a run's acceptance rate meets threshold. Triggers on "EAGLE3", "DFlash", "DSpark", "Domino", "draft model", "acceptance rate", "speculative decoding pipeline". Do NOT use for quantizing a model (use modelopt-ptq) or serving a checkpoint (use modelopt-deployment).
- License 许可
- Apache-2.0
- 最近更新
- 2026-11-15
- Stars
- ★ 5.6k
- Forks
- 806
01 前言 Front Matter
- name
- modelopt-speculative-decoding
- description
- Train, debug, and validate a speculative decoding draft model (EAGLE3, DFlash, DSpark, Domino) through the ModelOpt launcher pipeline. Use when the user wants to add a new model to a draft-training pipeline, asks why a pipeline run failed, wants experiment logs reviewed, or wants to check whether a run's acceptance rate meets threshold. Triggers on "EAGLE3", "DFlash", "DSpark", "Domino", "draft model", "acceptance rate", "speculative decoding pipeline". Do NOT use for quantizing a model (use modelopt-ptq) or serving a checkpoint (use modelopt-deployment).
- user_invocable
- true
02 资源文件 11
- 参考文档references/algorithms/dflash.md11.4 KB
- 参考文档references/algorithms/dflash2.md9.6 KB
- 参考文档references/algorithms/domino.md7.0 KB
- 参考文档references/algorithms/dspark.md10.7 KB
- 参考文档references/algorithms/eagle3.md6.2 KB
- 参考文档references/algorithms/lilicorr.md13.0 KB
- 参考文档references/algorithms/README.md2.5 KB
- 参考文档references/stages/configure.md2.2 KB
- 参考文档references/stages/review-logs.md2.9 KB
- 参考文档references/stages/triage.md6.2 KB
- 参考文档references/stages/validate.md4.9 KB
03 SKILL.md 正文
Speculative Decoding Draft-Model Training
Everything needed to take a target model from "no draft head" to "validated acceptance rate" lives in this directory. Work through the stages below in order for a new model; jump straight to a stage when you already know which one you need.
Two axes: stage and algorithm
The pipeline is the same shape for every draft-model algorithm — synthesize data, dump base-model hidden states, train the draft, benchmark acceptance rate. What changes between algorithms is which training script and recipe run, which knobs matter, and which failures are typical.
So this skill is split along those two axes, and you almost always read one file from each:
| Axis | Directory | What it holds |
|---|---|---|
| Stage | references/stages/ |
The procedure — algorithm-independent |
| Algorithm | references/algorithms/ |
The data sheet — scripts, recipe, knobs, thresholds, known failures |
Read the stage file for what to do, and the algorithm sheet for the values to
plug in. When a stage file says "see the algorithm sheet", it means the section of
references/algorithms/<algorithm>.md with the matching heading.
Where the code lives
The launcher scripts in tools/launcher/common/{eagle3,specdec}/ are thin wrappers around
examples/speculative_decoding/ — main.py (training), collect_hidden_states/ (dumps),
scripts/{export_hf_checkpoint,ar_validate,quantize_drafter}.py. Read a wrapper for the
flags it passes, that directory for what they do. Its README.md is also the non-launcher
path: training invoked directly, without Slurm.
Stages
| Stage | Reference | Use when |
|---|---|---|
| 1. Configure | references/stages/configure.md |
Adding a model that has no pipeline YAML yet |
| 2. Review logs | references/stages/review-logs.md |
A run finished (or died) and you want a pass/fail summary |
| 3. Triage | references/stages/triage.md |
A task failed and you need root cause plus a fix |
| 4. Validate | references/stages/validate.md |
All tasks passed and you need to confirm the acceptance rate gate |
Review-logs and triage overlap by design: review-logs is the fast sweep across all tasks, triage is the deep dive into one failing task. Start with review-logs unless the user already knows which task broke.
Algorithms
All recipes live in modelopt_recipes/general/speculative_decoding/<algorithm>.yaml.
| Algorithm | Sheet | Family |
|---|---|---|
| EAGLE3 | references/algorithms/eagle3.md |
Autoregressive draft head |
| DFlash | references/algorithms/dflash.md |
Block diffusion |
| DSpark | references/algorithms/dspark.md |
DFlash backbone + Markov head + optional confidence head |
| Domino | references/algorithms/domino.md |
DFlash backbone + GRU causal correction head |
| DFlash2 | references/algorithms/dflash2.md |
DFlash backbone + sublayer convolutions + candidate selector |
| LiLiCorr | references/algorithms/lilicorr.md |
DFlash backbone + candidate-lattice reranker (covers lilicorr_conv.yaml) |
Everything below EAGLE3 is a DFlash variant, not a separate pipeline: same
recipe_type: speculative_dflash, same training script, same dflash.* config
namespace, selected by dflash_architecture_config.projector_type. Read
references/algorithms/dflash.md first, then the variant's sheet for the delta.
LiLiCorr ships two recipes — lilicorr.yaml and lilicorr_conv.yaml, the latter
adding DFlash2's sublayer convolutions — but one sheet covers both.
If the user's algorithm has no sheet yet, the stage procedures still apply — derive
the missing values from an existing launcher example for that algorithm
(tools/launcher/examples/*/*/hf_*_<algorithm>.yaml) and its recipe, then write the
sheet as you go. references/algorithms/README.md defines what a sheet must contain.
End-to-end: a new model
-
Confirm the algorithm and find the closest existing launcher example.
-
Configure — write the pipeline YAML (
references/stages/configure.md). -
Preview with
--dryrun, then submit:cd tools/launcher uv run launch.py --yaml examples/<Org>/<Model>/<config>.yaml --yes -
Register the job and set up monitoring per the modelopt-monitor skill.
-
Review logs when it finishes (
references/stages/review-logs.md). -
Triage anything that failed (
references/stages/triage.md), fix, re-run only the failed tasks onward viapipeline.task_N.skip=true. -
Validate once all tasks pass (
references/stages/validate.md).
Model-support gaps that need code changes land in modelopt/torch/speculative/ and
require a separate ModelOpt PR — the pipeline YAML alone cannot fix an unrecognized
architecture.
04 同仓库更多 Skills
modelopt-run-agentx
Run the AgentX agentic serving benchmark with the SemiAnalysis harness. Use for AgentX setup, dataset downloads, concurrency sweeps, or serving latency and throughput comparisons.
modelopt-release-cherry-pick
Audit merged bug-fix PRs and release NVBugs for missing cherry-pick labels, then cherry-pick labeled PRs into a release branch and open a PR. Use when asked to "cherry-pick PRs for release/X.Y.Z", "pick PRs to release branch", "verify cherry-pick labels", or "cherry-pick labeled PRs".
modelopt-quant-recipe-search
Use when the user asks to find, search for, or optimize the best quantization recipe for a model, including direct requests like "find the best quantization recipe and generate a PTQ checkpoint." Guides the multi-candidate loop: choose compute-vs-memory success metrics, select ModelOpt recipe baselines, design AutoQuant/manual recipe deltas, interpret sensitivity, and decide next candidates. Do NOT use for a single known PTQ recipe run (use modelopt-ptq), serving (use modelopt-deployment), creating/running evals (use modelopt-evaluation or modelopt-launching-evals), monitoring jobs (use modelopt-monitor), MLflow browsing (use modelopt-accessing-mlflow), or comparing completed baseline-vs-candidate scores only (use modelopt-compare-results).
modelopt-qad
Run explicitly requested ModelOpt Quantization-Aware Distillation (QAD) through Megatron Bridge (recommended) or Hugging Face Transformers to recover a measured BF16-to-PTQ accuracy gap. Use only when the user explicitly asks for QAD, including its topology, data preparation, launch, resume, checkpoint export, or recovery decisions.
modelopt-ptq
Use when the user asks to "quantize a model", "run PTQ", "post-training quantization", "NVFP4 quantization", "FP8 quantization", "INT8 quantization", "INT4 AWQ", "quantize LLM", "quantize MoE", "quantize VLM", or needs to produce a quantized HuggingFace checkpoint from a pretrained model using ModelOpt. Do NOT use for multi-candidate recipe exploration or optimization (use modelopt-quant-recipe-search).
modelopt-monitor
Monitor submitted jobs (PTQ, evaluation, deployment) on SLURM clusters. Use when the user asks "check job status", "is my job done", "monitor my evaluation", "what's the status of the PTQ", "check on job" followed by a SLURM job ID, or after any skill submits a long-running job. Also triggers on "nel status", "squeue", or any request to check progress of a previously submitted job.