[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6CP44T5R6WV38NVEEXAJ5RV":3,"skill-same:01M6CP44T5R6WV38NVEEXAJ5RV":83},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":22,"publishedAt":49,"linkOnly":50,"markdownContent":51,"markdownBody":52,"frontMatter":53,"refName":54,"commitSha":55,"skillDirectory":56,"forksCount":57,"repositoryHtmlUrl":58,"repositoryHomepageUrl":59,"ownerHtmlUrl":60,"ownerBlogUrl":61,"ownerDescription":6,"updatedAt":62,"install":63,"resources":76},"01M6CP44T5R6WV38NVEEXAJ5RV","vllm-bench",null,"Use vllm-bench to benchmark OpenAI-compatible or vLLM serving endpoints, especially multi-turn chat load tests. Use when the user asks for vllm bench, vllm-bench, multi-turn benchmark, chat serving benchmark, TTFT\u002FTPOT\u002Fthroughput measurement, concurrency sweep, load test, 压测, 多轮压测, or wants help installing, running --help, choosing flags, validating a local PegaInfer\u002FvLLM server, saving JSON results, or interpreting vllm-bench metrics.","pegainfer-project\u002Fpegainfer","pegainfer-project","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F292134277?v=4","01M6CP3N2567A4DACEYMN5W8S0","01M6CP3N66CWB0M39SGYE63S7D",739,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fpegainfer-project\u002Fpegainfer\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fvllm-bench\u002FSKILL.md",[17],{"categoryNo":18,"categoryName":19,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000010","后端与接口","SCENARIO","使用场景",[23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48],"vLLM","vllm bench","endpoints","running","when","for","load","vllm server","measurement","openai-compatible","throughput","tests","local","flags","test","use","user","results","sweep","json","concurrency","server","metrics","openai","help","benchmark","2026-11-10 10:12:30",false,"---\nname: vllm-bench\ndescription: \"Use vllm-bench to benchmark OpenAI-compatible or vLLM serving endpoints, especially multi-turn chat load tests. Use when the user asks for vllm bench, vllm-bench, multi-turn benchmark, chat serving benchmark, TTFT\u002FTPOT\u002Fthroughput measurement, concurrency sweep, load test, 压测, 多轮压测, or wants help installing, running --help, choosing flags, validating a local PegaInfer\u002FvLLM server, saving JSON results, or interpreting vllm-bench metrics.\"\n---\n\n# vllm-bench\n\n`vllm-bench` is a Rust benchmark client for online LLM serving. It does not run\nthe model. It drives an already running OpenAI-compatible or vLLM endpoint and\nreports TTFT, TPOT, ITL, E2EL, throughput, and multi-turn per-turn breakdowns.\n\nUse this workflow for multi-turn work:\n\n1. Install or locate `vllm-bench`.\n2. Run `vllm-bench --help` and use it as the authority for current flags.\n3. Confirm the server base URL and model id from `\u002Fv1\u002Fmodels`.\n4. Dry-run the dataset locally.\n5. Run a tiny smoke benchmark.\n6. Scale conversations, turns, and concurrency only after the smoke passes.\n7. Save JSON for any result that may be compared later.\n\n## Install\n\nInstall from the upstream repository:\n\n```bash\ncargo install --git https:\u002F\u002Fgithub.com\u002Fvllm-project\u002Fvllm-bench vllm-bench\n```\n\nThe trailing `vllm-bench` package name matters: the repo also contains another\nbinary, so omitting it can fail with \"multiple packages with binaries found\".\nCargo installs the binary to `~\u002F.cargo\u002Fbin\u002Fvllm-bench`.\n\nAlternatives from the README:\n\n```bash\ncurl -fsSL https:\u002F\u002Fgithub.com\u002Fvllm-project\u002Fvllm-bench\u002Freleases\u002Flatest\u002Fdownload\u002Fvllm-bench-$(uname -m)-linux-musl \\\n  -o vllm-bench && chmod +x vllm-bench\n\ngit clone https:\u002F\u002Fgithub.com\u002Fvllm-project\u002Fvllm-bench.git\ncd vllm-bench\n.\u002Finstall.sh\n```\n\nAfter install, run:\n\n```bash\nvllm-bench --version\nvllm-bench --help\n```\n\n## Choose Backend\n\nFor multi-turn chat, always use:\n\n```bash\n--backend openai-chat\n```\n\nCommon backends:\n\n| Backend | Endpoint | Use |\n| --- | --- | --- |\n| `vllm` \u002F `openai` | `\u002Fv1\u002Fcompletions` | text completions |\n| `openai-chat` | `\u002Fv1\u002Fchat\u002Fcompletions` | chat, multi-turn, multimodal |\n| `openai-embeddings` | `\u002Fv1\u002Fembeddings` | text embeddings |\n| `vllm-rerank` | `\u002Fv1\u002Frerank` | rerank |\n\nPrefer `http:\u002F\u002F127.0.0.1:\u003Cport>` over `localhost`; some servers bind IPv4 while\n`localhost` resolves to IPv6.\n\nConfirm the model id before benchmarking:\n\n```bash\ncurl -s http:\u002F\u002F127.0.0.1:8000\u002Fv1\u002Fmodels\n```\n\nUse the returned id as `--model`, or omit `--model` and let `vllm-bench`\nauto-fetch it.\n\n## Multi-Turn Flags\n\nCore flags:\n\n```bash\n--multi-turn\n--multi-turn-num-turns \u003CN>\n--num-prompts \u003CCONVERSATIONS>\n--multi-turn-concurrency \u003CCONCURRENT_CONVERSATIONS>\n```\n\nIn multi-turn mode, `--num-prompts` means conversation count, not request count.\nA run with `--num-prompts 50 --multi-turn-num-turns 5` can send up to 250 chat\nrequests.\n\nSynthetic random conversations:\n\n```bash\n--dataset-name random\n--random-input-len \u003CTOKENS_FOR_TURN_1>\n--per-turn-input-len \u003CTOKENS_FOR_TURNS_2_PLUS>\n--random-output-len \u003COUTPUT_TOKENS_PER_TURN>\n```\n\nIf `--per-turn-input-len` is omitted or `0`, later turns reuse\n`--random-input-len`. Use `--multi-turn-min-turns` and `--multi-turn-max-turns`\nwhen conversation lengths should vary.\n\nThink time and prefix sharing:\n\n```bash\n--multi-turn-delay-ms 500\n--multi-turn-prefix-global-ratio 0.2\n--multi-turn-prefix-conversation-ratio 0.5\n```\n\nPrefix sharing works only with `--dataset-name random`; the two ratios must sum\nto less than `1.0`.\n\nWhen either prefix-sharing flag is non-zero, `vllm-bench` switches to fixed-\nlength per-turn prompts with no accumulated chat history. Use this only for\nprefix-cache experiments; leave both ratios at `0` for realistic multi-turn\nconversation growth.\n\n## Dry Run\n\nDry-run before sending traffic. It loads the tokenizer and builds conversations\nwithout hitting the server:\n\n```bash\nvllm-bench \\\n  --backend openai-chat \\\n  --model Qwen\u002FQwen3-4B \\\n  --tokenizer \u003Cmodel-path> \\\n  --dataset-name random \\\n  --multi-turn --multi-turn-num-turns 2 \\\n  --random-input-len 16 --random-output-len 4 \\\n  --num-prompts 2 \\\n  --dry-run\n```\n\n`\u003Cmodel-path>` is a local tokenizer\u002Fmodel directory; for a public example,\nreplace it with the served model id or any local model directory.\n\n## Example: Generic vLLM Multi-Turn\n\nUse this after a vLLM OpenAI server is already running on port 8000:\n\n```bash\nvllm-bench \\\n  --backend openai-chat \\\n  --base-url http:\u002F\u002F127.0.0.1:8000 \\\n  --model \u003Cmodel-id-from-\u002Fv1\u002Fmodels> \\\n  --dataset-name random \\\n  --multi-turn \\\n  --multi-turn-num-turns 5 \\\n  --random-input-len 512 \\\n  --per-turn-input-len 256 \\\n  --random-output-len 128 \\\n  --num-prompts 50 \\\n  --multi-turn-concurrency 10 \\\n  --ready-check-timeout-sec 60 \\\n  --percentile-metrics ttft,tpot,itl,e2el \\\n  --metric-percentiles 50,90,99 \\\n  --save-result\n```\n\nRead the output as:\n\n- `Conversations completed\u002Ftotal`: end-to-end conversation success.\n- `Successful requests`: successful turns.\n- `Total input tokens`: includes growing chat history in normal multi-turn mode.\n- `per_turn_metrics` in JSON: TTFT\u002FTPOT\u002FITL\u002FE2EL by turn index.\n\nExpect later turns to have larger input token counts because each request sends\nthe prior user\u002Fassistant history plus the next user message.\n\n## Example: Local PegaInfer Qwen3-4B Smoke\n\nThis was validated locally with PegaInfer Qwen3-4B and `vllm-bench` built from\nthis repo. Start PegaInfer:\n\n```bash\ncd \u003Cpegainfer-repo>\ncargo run --release -p pegainfer-server -- \\\n  --model-path \u003Cmodel-path> \\\n  --served-model-name Qwen\u002FQwen3-4B \\\n  --port 18080 \\\n  --cuda-graph=false\n```\n\nConfirm the model:\n\n```bash\ncurl -s http:\u002F\u002F127.0.0.1:18080\u002Fv1\u002Fmodels\n```\n\nRun a tiny multi-turn benchmark:\n\n```bash\ncd \u003Cvllm-bench-repo>\ntarget\u002Frelease\u002Fvllm-bench \\\n  --backend openai-chat \\\n  --base-url http:\u002F\u002F127.0.0.1:18080 \\\n  --model Qwen\u002FQwen3-4B \\\n  --tokenizer \u003Cmodel-path> \\\n  --dataset-name random \\\n  --multi-turn \\\n  --multi-turn-min-turns 2 \\\n  --multi-turn-max-turns 3 \\\n  --random-input-len 32 \\\n  --per-turn-input-len 12 \\\n  --random-output-len 8 \\\n  --num-prompts 3 \\\n  --multi-turn-concurrency 2 \\\n  --ready-check-timeout-sec 30 \\\n  --extra-body '{\"min_tokens\":null}' \\\n  --percentile-metrics ttft,tpot,itl,e2el \\\n  --metric-percentiles 50,90 \\\n  --save-result \\\n  --result-dir \u002Ftmp \\\n  --result-filename vllm-bench-multi-turn-smoke.json\n```\n\nThe `--extra-body '{\"min_tokens\":null}'` is for PegaInfer compatibility. The\nrandom multi-turn path auto-adds `min_tokens` unless overridden; this pins\noutput length on vLLM but PegaInfer currently rejects that field. `--ignore-eos`\nalso skips `min_tokens` and worked in local smoke testing, but it asks the\nserver to ignore EOS and can grow contexts more aggressively.\n\nThe local smoke run completed 3\u002F3 conversations, 8\u002F8 turns, and wrote\n`\u002Ftmp\u002Fvllm-bench-multi-turn-smoke.json` with `per_turn_metrics`.\n\n## Scale Safely\n\nAfter the smoke passes:\n\n1. Increase `--num-prompts` first to improve measurement stability.\n2. Increase `--multi-turn-concurrency` to find the service knee.\n3. Increase `--random-input-len`, `--per-turn-input-len`, and\n   `--random-output-len` to match the target workload.\n4. Save every serious run with `--save-result --metadata KEY=VALUE`.\n\nFor deterministic latency A\u002FB runs, pass `--temperature 0` explicitly. Omit it\nonly when the benchmark is intentionally measuring server default generation\nbehavior.\n\nUse a sweep when searching for the concurrency knee:\n\n```bash\nvllm-bench \\\n  --backend openai-chat \\\n  --base-url http:\u002F\u002F127.0.0.1:8000 \\\n  --model \u003Cmodel-id> \\\n  --dataset-name random \\\n  --multi-turn --multi-turn-num-turns 5 \\\n  --random-input-len 512 --per-turn-input-len 256 --random-output-len 128 \\\n  --num-prompts 200 \\\n  --sweep-max-concurrency 1,2,4,8,16,32 \\\n  --sweep-summary-percentiles 90,99 \\\n  --save-result\n```\n\nFor A\u002FB comparisons, keep the workload flags identical and compare result JSON:\n\n```bash\nvllm-bench --compare baseline.json candidate.json\n```\n\n## Troubleshooting\n\n- `min_tokens is not supported by this engine`: pass\n  `--extra-body '{\"min_tokens\":null}'` or, if appropriate, `--ignore-eos`.\n- All-zero metrics with failed requests: inspect the first HTTP error printed\n  above the result block; it usually names the unsupported request field.\n- Connection fails on `localhost`: switch to `127.0.0.1`.\n- Model id mismatch: use `curl \u002Fv1\u002Fmodels` and set `--model` to that exact id.\n- Tokenizer load fails: set `--tokenizer` to a local model directory with\n  `tokenizer.json`, or rely on server-side tokenize\u002Fdetokenize fallback if the\n  server implements it.\n- Dataset construction is suspicious: run the same command with `--dry-run`.\n- Long serious runs need reproducibility: set `--seed`, `--result-filename`,\n  and `--metadata` entries for server commit, model, GPU, and key flags.\n\n## Metrics\n\nUse database-style thinking: requests are transactions, prompt tokens are read\nset size, output tokens are write volume, and concurrency is the number of\nin-flight conversations.\n\n- TTFT: queueing plus prefill latency until the first output token.\n- TPOT: decode time per output token after the first token.\n- ITL: per-token latency distribution.\n- E2EL: total turn latency.\n- Request throughput: turns per second.\n- Output throughput: generated tokens per second.\n- Multi-turn JSON: inspect `per_turn_metrics` to see how latency changes as\n  history grows.\n","# vllm-bench\n\n`vllm-bench` is a Rust benchmark client for online LLM serving. It does not run\nthe model. It drives an already running OpenAI-compatible or vLLM endpoint and\nreports TTFT, TPOT, ITL, E2EL, throughput, and multi-turn per-turn breakdowns.\n\nUse this workflow for multi-turn work:\n\n1. Install or locate `vllm-bench`.\n2. Run `vllm-bench --help` and use it as the authority for current flags.\n3. Confirm the server base URL and model id from `\u002Fv1\u002Fmodels`.\n4. Dry-run the dataset locally.\n5. Run a tiny smoke benchmark.\n6. Scale conversations, turns, and concurrency only after the smoke passes.\n7. Save JSON for any result that may be compared later.\n\n## Install\n\nInstall from the upstream repository:\n\n```bash\ncargo install --git https:\u002F\u002Fgithub.com\u002Fvllm-project\u002Fvllm-bench vllm-bench\n```\n\nThe trailing `vllm-bench` package name matters: the repo also contains another\nbinary, so omitting it can fail with \"multiple packages with binaries found\".\nCargo installs the binary to `~\u002F.cargo\u002Fbin\u002Fvllm-bench`.\n\nAlternatives from the README:\n\n```bash\ncurl -fsSL https:\u002F\u002Fgithub.com\u002Fvllm-project\u002Fvllm-bench\u002Freleases\u002Flatest\u002Fdownload\u002Fvllm-bench-$(uname -m)-linux-musl \\\n  -o vllm-bench && chmod +x vllm-bench\n\ngit clone https:\u002F\u002Fgithub.com\u002Fvllm-project\u002Fvllm-bench.git\ncd vllm-bench\n.\u002Finstall.sh\n```\n\nAfter install, run:\n\n```bash\nvllm-bench --version\nvllm-bench --help\n```\n\n## Choose Backend\n\nFor multi-turn chat, always use:\n\n```bash\n--backend openai-chat\n```\n\nCommon backends:\n\n| Backend | Endpoint | Use |\n| --- | --- | --- |\n| `vllm` \u002F `openai` | `\u002Fv1\u002Fcompletions` | text completions |\n| `openai-chat` | `\u002Fv1\u002Fchat\u002Fcompletions` | chat, multi-turn, multimodal |\n| `openai-embeddings` | `\u002Fv1\u002Fembeddings` | text embeddings |\n| `vllm-rerank` | `\u002Fv1\u002Frerank` | rerank |\n\nPrefer `http:\u002F\u002F127.0.0.1:\u003Cport>` over `localhost`; some servers bind IPv4 while\n`localhost` resolves to IPv6.\n\nConfirm the model id before benchmarking:\n\n```bash\ncurl -s http:\u002F\u002F127.0.0.1:8000\u002Fv1\u002Fmodels\n```\n\nUse the returned id as `--model`, or omit `--model` and let `vllm-bench`\nauto-fetch it.\n\n## Multi-Turn Flags\n\nCore flags:\n\n```bash\n--multi-turn\n--multi-turn-num-turns \u003CN>\n--num-prompts \u003CCONVERSATIONS>\n--multi-turn-concurrency \u003CCONCURRENT_CONVERSATIONS>\n```\n\nIn multi-turn mode, `--num-prompts` means conversation count, not request count.\nA run with `--num-prompts 50 --multi-turn-num-turns 5` can send up to 250 chat\nrequests.\n\nSynthetic random conversations:\n\n```bash\n--dataset-name random\n--random-input-len \u003CTOKENS_FOR_TURN_1>\n--per-turn-input-len \u003CTOKENS_FOR_TURNS_2_PLUS>\n--random-output-len \u003COUTPUT_TOKENS_PER_TURN>\n```\n\nIf `--per-turn-input-len` is omitted or `0`, later turns reuse\n`--random-input-len`. Use `--multi-turn-min-turns` and `--multi-turn-max-turns`\nwhen conversation lengths should vary.\n\nThink time and prefix sharing:\n\n```bash\n--multi-turn-delay-ms 500\n--multi-turn-prefix-global-ratio 0.2\n--multi-turn-prefix-conversation-ratio 0.5\n```\n\nPrefix sharing works only with `--dataset-name random`; the two ratios must sum\nto less than `1.0`.\n\nWhen either prefix-sharing flag is non-zero, `vllm-bench` switches to fixed-\nlength per-turn prompts with no accumulated chat history. Use this only for\nprefix-cache experiments; leave both ratios at `0` for realistic multi-turn\nconversation growth.\n\n## Dry Run\n\nDry-run before sending traffic. It loads the tokenizer and builds conversations\nwithout hitting the server:\n\n```bash\nvllm-bench \\\n  --backend openai-chat \\\n  --model Qwen\u002FQwen3-4B \\\n  --tokenizer \u003Cmodel-path> \\\n  --dataset-name random \\\n  --multi-turn --multi-turn-num-turns 2 \\\n  --random-input-len 16 --random-output-len 4 \\\n  --num-prompts 2 \\\n  --dry-run\n```\n\n`\u003Cmodel-path>` is a local tokenizer\u002Fmodel directory; for a public example,\nreplace it with the served model id or any local model directory.\n\n## Example: Generic vLLM Multi-Turn\n\nUse this after a vLLM OpenAI server is already running on port 8000:\n\n```bash\nvllm-bench \\\n  --backend openai-chat \\\n  --base-url http:\u002F\u002F127.0.0.1:8000 \\\n  --model \u003Cmodel-id-from-\u002Fv1\u002Fmodels> \\\n  --dataset-name random \\\n  --multi-turn \\\n  --multi-turn-num-turns 5 \\\n  --random-input-len 512 \\\n  --per-turn-input-len 256 \\\n  --random-output-len 128 \\\n  --num-prompts 50 \\\n  --multi-turn-concurrency 10 \\\n  --ready-check-timeout-sec 60 \\\n  --percentile-metrics ttft,tpot,itl,e2el \\\n  --metric-percentiles 50,90,99 \\\n  --save-result\n```\n\nRead the output as:\n\n- `Conversations completed\u002Ftotal`: end-to-end conversation success.\n- `Successful requests`: successful turns.\n- `Total input tokens`: includes growing chat history in normal multi-turn mode.\n- `per_turn_metrics` in JSON: TTFT\u002FTPOT\u002FITL\u002FE2EL by turn index.\n\nExpect later turns to have larger input token counts because each request sends\nthe prior user\u002Fassistant history plus the next user message.\n\n## Example: Local PegaInfer Qwen3-4B Smoke\n\nThis was validated locally with PegaInfer Qwen3-4B and `vllm-bench` built from\nthis repo. Start PegaInfer:\n\n```bash\ncd \u003Cpegainfer-repo>\ncargo run --release -p pegainfer-server -- \\\n  --model-path \u003Cmodel-path> \\\n  --served-model-name Qwen\u002FQwen3-4B \\\n  --port 18080 \\\n  --cuda-graph=false\n```\n\nConfirm the model:\n\n```bash\ncurl -s http:\u002F\u002F127.0.0.1:18080\u002Fv1\u002Fmodels\n```\n\nRun a tiny multi-turn benchmark:\n\n```bash\ncd \u003Cvllm-bench-repo>\ntarget\u002Frelease\u002Fvllm-bench \\\n  --backend openai-chat \\\n  --base-url http:\u002F\u002F127.0.0.1:18080 \\\n  --model Qwen\u002FQwen3-4B \\\n  --tokenizer \u003Cmodel-path> \\\n  --dataset-name random \\\n  --multi-turn \\\n  --multi-turn-min-turns 2 \\\n  --multi-turn-max-turns 3 \\\n  --random-input-len 32 \\\n  --per-turn-input-len 12 \\\n  --random-output-len 8 \\\n  --num-prompts 3 \\\n  --multi-turn-concurrency 2 \\\n  --ready-check-timeout-sec 30 \\\n  --extra-body '{\"min_tokens\":null}' \\\n  --percentile-metrics ttft,tpot,itl,e2el \\\n  --metric-percentiles 50,90 \\\n  --save-result \\\n  --result-dir \u002Ftmp \\\n  --result-filename vllm-bench-multi-turn-smoke.json\n```\n\nThe `--extra-body '{\"min_tokens\":null}'` is for PegaInfer compatibility. The\nrandom multi-turn path auto-adds `min_tokens` unless overridden; this pins\noutput length on vLLM but PegaInfer currently rejects that field. `--ignore-eos`\nalso skips `min_tokens` and worked in local smoke testing, but it asks the\nserver to ignore EOS and can grow contexts more aggressively.\n\nThe local smoke run completed 3\u002F3 conversations, 8\u002F8 turns, and wrote\n`\u002Ftmp\u002Fvllm-bench-multi-turn-smoke.json` with `per_turn_metrics`.\n\n## Scale Safely\n\nAfter the smoke passes:\n\n1. Increase `--num-prompts` first to improve measurement stability.\n2. Increase `--multi-turn-concurrency` to find the service knee.\n3. Increase `--random-input-len`, `--per-turn-input-len`, and\n   `--random-output-len` to match the target workload.\n4. Save every serious run with `--save-result --metadata KEY=VALUE`.\n\nFor deterministic latency A\u002FB runs, pass `--temperature 0` explicitly. Omit it\nonly when the benchmark is intentionally measuring server default generation\nbehavior.\n\nUse a sweep when searching for the concurrency knee:\n\n```bash\nvllm-bench \\\n  --backend openai-chat \\\n  --base-url http:\u002F\u002F127.0.0.1:8000 \\\n  --model \u003Cmodel-id> \\\n  --dataset-name random \\\n  --multi-turn --multi-turn-num-turns 5 \\\n  --random-input-len 512 --per-turn-input-len 256 --random-output-len 128 \\\n  --num-prompts 200 \\\n  --sweep-max-concurrency 1,2,4,8,16,32 \\\n  --sweep-summary-percentiles 90,99 \\\n  --save-result\n```\n\nFor A\u002FB comparisons, keep the workload flags identical and compare result JSON:\n\n```bash\nvllm-bench --compare baseline.json candidate.json\n```\n\n## Troubleshooting\n\n- `min_tokens is not supported by this engine`: pass\n  `--extra-body '{\"min_tokens\":null}'` or, if appropriate, `--ignore-eos`.\n- All-zero metrics with failed requests: inspect the first HTTP error printed\n  above the result block; it usually names the unsupported request field.\n- Connection fails on `localhost`: switch to `127.0.0.1`.\n- Model id mismatch: use `curl \u002Fv1\u002Fmodels` and set `--model` to that exact id.\n- Tokenizer load fails: set `--tokenizer` to a local model directory with\n  `tokenizer.json`, or rely on server-side tokenize\u002Fdetokenize fallback if the\n  server implements it.\n- Dataset construction is suspicious: run the same command with `--dry-run`.\n- Long serious runs need reproducibility: set `--seed`, `--result-filename`,\n  and `--metadata` entries for server commit, model, GPU, and key flags.\n\n## Metrics\n\nUse database-style thinking: requests are transactions, prompt tokens are read\nset size, output tokens are write volume, and concurrency is the number of\nin-flight conversations.\n\n- TTFT: queueing plus prefill latency until the first output token.\n- TPOT: decode time per output token after the first token.\n- ITL: per-token latency distribution.\n- E2EL: total turn latency.\n- Request throughput: turns per second.\n- Output throughput: generated tokens per second.\n- Multi-turn JSON: inspect `per_turn_metrics` to see how latency changes as\n  history grows.\n",{"name":5,"description":7},"main","2246b2ac1ed98ec92c408ff51ce7b33d4b255971",".claude\u002Fskills\u002Fvllm-bench",120,"https:\u002F\u002Fgithub.com\u002Fpegainfer-project\u002Fpegainfer","https:\u002F\u002Fpegainfer.org\u002F","https:\u002F\u002Fgithub.com\u002Fpegainfer-project","https:\u002F\u002Fopen-infer.org\u002F","2026-11-12 03:48:06",{"source":8,"skillName":5,"commands":64,"prompt":74,"zipAvailable":75},[65,68,71],{"tool":66,"command":67},"npx","npx skills add pegainfer-project\u002Fpegainfer --skill vllm-bench",{"tool":69,"command":70},"bunx","bunx skills add pegainfer-project\u002Fpegainfer --skill vllm-bench",{"tool":72,"command":73},"pnpm","pnpm dlx skills add pegainfer-project\u002Fpegainfer --skill vllm-bench","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fpegainfer-project\u002Fpegainfer\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fvllm-bench\u002FSKILL.md",true,[77],{"path":78,"resourceType":79,"resourceTypeDesc":80,"sizeBytes":81,"sourceHtmlUrl":82},"agents\u002Fopenai.yaml","REFERENCE","参考文档",204,"https:\u002F\u002Fgithub.com\u002Fpegainfer-project\u002Fpegainfer\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fvllm-bench\u002Fagents\u002Fopenai.yaml",[84,92,119,125,155],{"skillNo":85,"name":86,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":87,"categories":88,"keywords":89,"publishedAt":49},"01M6CP47ZZ9G8N4YW1AEW11QRQ","nsys-profiling","https:\u002F\u002Fgithub.com\u002Fpegainfer-project\u002Fpegainfer\u002Fblob\u002Fmain\u002F.codex\u002Fskills\u002Fnsys-profiling\u002FSKILL.md",[],[90,91],"nsys profiling","profiling",{"skillNo":93,"name":86,"nameZh":6,"description":94,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":95,"categories":96,"keywords":98,"publishedAt":49},"01M6CP41N22W0PPT3F3T032JQZ","Profile GPU workloads with NVIDIA Nsight Systems (nsys). Use this skill when the user says 'profile', 'nsys', 'capture a trace', 'GPU profiling', 'kernel breakdown', 'where is time spent', 'decode is slow', 'prefill is slow', 'TPOT regression', 'TTFT regression', or wants to understand where GPU time is going. Also trigger when the user pastes nsys output and asks for help interpreting it. This skill covers the full lifecycle: capturing traces with the right flags, analyzing kernel\u002FAPI summaries, detecting tail latency problems, and diagnosing performance regressions.","https:\u002F\u002Fgithub.com\u002Fpegainfer-project\u002Fpegainfer\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fnsys-profiling\u002FSKILL.md",[97],{"categoryNo":18,"categoryName":19,"taxonomyType":20,"taxonomyTypeDesc":21},[90,91,99,100,101,102,103,104,105,47,106,107,108,109,110,27,111,112,28,113,114,115,116,36,38,117,39,118],"trace","regression","api","lifecycle","analyzing","NVIDIA","gpu","performance","kernel","latency","prefill","breakdown","gpu profiling","summaries","detecting","systems","full","time","profile","capture",{"skillNo":120,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":121,"categories":122,"keywords":124,"publishedAt":49},"01M6CP3YKSBD68R742QN3915C9","https:\u002F\u002Fgithub.com\u002Fpegainfer-project\u002Fpegainfer\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fvllm-bench\u002FSKILL.md",[123],{"categoryNo":18,"categoryName":19,"taxonomyType":20,"taxonomyTypeDesc":21},[24,23,33,26,27,28,29,30,31,32,35,36,25,38,39,40,41,42,43,34,44,45,46,47,48,37],{"skillNo":126,"name":127,"nameZh":6,"description":128,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":129,"categories":130,"keywords":131,"publishedAt":49},"01M6CP3VG0B2HGD6AM6BTNGG7Y","project-doc","Track any non-trivial task through a doc placed in the right domain directory under docs\u002F (models\u002F\u003Cline>\u002F, subsystems\u002F\u003Carea>\u002F, playbooks\u002F, lessons\u002F, etc.). Covers the full lifecycle: read docs first, write a plan, wait for approval, log execution, and capture lessons learned. Use this skill whenever the user gives a concrete task — not just Q&A or casual discussion. Even if the task seems straightforward, if it involves doing something (not just answering something), open or reuse a domain doc. Bias toward using this skill. When in doubt, use it.","https:\u002F\u002Fgithub.com\u002Fpegainfer-project\u002Fpegainfer\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fproject-doc-bootstrap\u002FSKILL.md",[],[132,133,134,135,136,137,138,139,140,141,142,115,102,143,144,145,28,146,38,147,39,148,149,118,150,151,152,153,27,154],"project","project doc","doc","directory","execution","docs","models","q&a","wait","line","learned","domain","open","task","log","approval","write","read","just","wait for","first","plan","track",{"skillNo":156,"name":86,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":157,"categories":158,"keywords":159,"publishedAt":49},"01M6CP3QVJ2MHXRM1SBXVGQNTK","https:\u002F\u002Fgithub.com\u002Fpegainfer-project\u002Fpegainfer\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fnsys-profiling\u002FSKILL.md",[],[90,91]]