[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6QG0W1T24B4CFJC1WFMFB6J":3,"skill-same:01M6QG0W1T24B4CFJC1WFMFB6J":96},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":23,"publishedAt":44,"linkOnly":45,"markdownContent":46,"markdownBody":47,"frontMatter":48,"refName":55,"commitSha":56,"skillDirectory":57,"forksCount":14,"repositoryHtmlUrl":58,"repositoryHomepageUrl":59,"ownerHtmlUrl":60,"ownerBlogUrl":61,"ownerDescription":62,"updatedAt":63,"install":64,"resources":77},"01M6QG0W1T24B4CFJC1WFMFB6J","serving-llms-vllm",null,"Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching. Use when deploying production LLM APIs, optimizing inference latency\u002Fthroughput, or serving models with limited GPU memory. Supports OpenAI-compatible endpoints, quantization (GPTQ\u002FAWQ\u002FFP8), and tensor parallelism.","Tashima-Tarsh\u002FDisha6.6","Tashima-Tarsh","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F269906498?v=4","01M6PBH98X03XERH0HFT4PFB1P","NITISH KUMAR","01M6PBH9DA0M6KM3Z12P848F0K",0,"MIT","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fkal-serving-llms-vllm\u002FSKILL.md",[18],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000010","后端与接口","SCENARIO","使用场景",[24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43],"serving","vLLM","serving llms vllm","llms","ops","batching","openai-compatible","throughput","GPTQ","AWQ","latency","use","api","Transformers","Inference Serving","FP8","openai api","openai","ml","memory","2026-11-10 10:12:30",false,"---\nname: serving-llms-vllm\ndescription: Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching. Use when deploying production LLM APIs, optimizing inference latency\u002Fthroughput, or serving models with limited GPU memory. Supports OpenAI-compatible endpoints, quantization (GPTQ\u002FAWQ\u002FFP8), and tensor parallelism.\nlicense: MIT\nmetadata:\n  version: 1.0.0\n  category: ml-inference-and-ops\n  maintainer: Kalaris Labs\n  tags: vLLM, Inference Serving, PagedAttention, Continuous Batching, High Throughput, Production, OpenAI API, Quantization, Tensor Parallelism\n  dependencies: vllm, torch, transformers\n---\n\n# vLLM - High-Performance LLM Serving\n\n## Quick start\n\nvLLM achieves 24x higher throughput than standard transformers through PagedAttention (block-based KV cache) and continuous batching (mixing prefill\u002Fdecode requests).\n\n**Installation**:\n```bash\npip install vllm\n```\n\n**Basic offline inference**:\n```python\nfrom vllm import LLM, SamplingParams\n\nllm = LLM(model=\"meta-llama\u002FLlama-3-8B-Instruct\")\nsampling = SamplingParams(temperature=0.7, max_tokens=256)\n\noutputs = llm.generate([\"Explain quantum computing\"], sampling)\nprint(outputs[0].outputs[0].text)\n```\n\n**OpenAI-compatible server**:\n```bash\nvllm serve meta-llama\u002FLlama-3-8B-Instruct\n\n# Query with OpenAI SDK\npython -c \"\nfrom openai import OpenAI\nclient = OpenAI(base_url='http:\u002F\u002Flocalhost:8000\u002Fv1', api_key='EMPTY')\nprint(client.chat.completions.create(\n    model='meta-llama\u002FLlama-3-8B-Instruct',\n    messages=[{'role': 'user', 'content': 'Hello!'}]\n).choices[0].message.content)\n\"\n```\n\n## Common workflows\n\n### Workflow 1: Production API deployment\n\nCopy this checklist and track progress:\n\n```\nDeployment Progress:\n- [ ] Step 1: Configure server settings\n- [ ] Step 2: Test with limited traffic\n- [ ] Step 3: Enable monitoring\n- [ ] Step 4: Deploy to production\n- [ ] Step 5: Verify performance metrics\n```\n\n**Step 1: Configure server settings**\n\nChoose configuration based on your model size:\n\n```bash\n# For 7B-13B models on single GPU\nvllm serve meta-llama\u002FLlama-3-8B-Instruct \\\n  --gpu-memory-utilization 0.9 \\\n  --max-model-len 8192 \\\n  --port 8000\n\n# For 30B-70B models with tensor parallelism\nvllm serve meta-llama\u002FLlama-2-70b-hf \\\n  --tensor-parallel-size 4 \\\n  --gpu-memory-utilization 0.9 \\\n  --quantization awq \\\n  --port 8000\n\n# For production with caching and metrics\nvllm serve meta-llama\u002FLlama-3-8B-Instruct \\\n  --gpu-memory-utilization 0.9 \\\n  --enable-prefix-caching \\\n  --enable-metrics \\\n  --metrics-port 9090 \\\n  --port 8000 \\\n  --host 0.0.0.0\n```\n\n**Step 2: Test with limited traffic**\n\nRun load test before production:\n\n```bash\n# Install load testing tool\npip install locust\n\n# Create test_load.py with sample requests\n# Run: locust -f test_load.py --host http:\u002F\u002Flocalhost:8000\n```\n\nVerify TTFT (time to first token) \u003C 500ms and throughput > 100 req\u002Fsec.\n\n**Step 3: Enable monitoring**\n\nvLLM exposes Prometheus metrics on port 9090:\n\n```bash\ncurl http:\u002F\u002Flocalhost:9090\u002Fmetrics | grep vllm\n```\n\nKey metrics to monitor:\n- `vllm:time_to_first_token_seconds` - Latency\n- `vllm:num_requests_running` - Active requests\n- `vllm:gpu_cache_usage_perc` - KV cache utilization\n\n**Step 4: Deploy to production**\n\nUse Docker for consistent deployment:\n\n```bash\n# Run vLLM in Docker\ndocker run --gpus all -p 8000:8000 \\\n  vllm\u002Fvllm-openai:latest \\\n  --model meta-llama\u002FLlama-3-8B-Instruct \\\n  --gpu-memory-utilization 0.9 \\\n  --enable-prefix-caching\n```\n\n**Step 5: Verify performance metrics**\n\nCheck that deployment meets targets:\n- TTFT \u003C 500ms (for short prompts)\n- Throughput > target req\u002Fsec\n- GPU utilization > 80%\n- No OOM errors in logs\n\n### Workflow 2: Offline batch inference\n\nFor processing large datasets without server overhead.\n\nCopy this checklist:\n\n```\nBatch Processing:\n- [ ] Step 1: Prepare input data\n- [ ] Step 2: Configure LLM engine\n- [ ] Step 3: Run batch inference\n- [ ] Step 4: Process results\n```\n\n**Step 1: Prepare input data**\n\n```python\n# Load prompts from file\nprompts = []\nwith open(\"prompts.txt\") as f:\n    prompts = [line.strip() for line in f]\n\nprint(f\"Loaded {len(prompts)} prompts\")\n```\n\n**Step 2: Configure LLM engine**\n\n```python\nfrom vllm import LLM, SamplingParams\n\nllm = LLM(\n    model=\"meta-llama\u002FLlama-3-8B-Instruct\",\n    tensor_parallel_size=2,  # Use 2 GPUs\n    gpu_memory_utilization=0.9,\n    max_model_len=4096\n)\n\nsampling = SamplingParams(\n    temperature=0.7,\n    top_p=0.95,\n    max_tokens=512,\n    stop=[\"\u003C\u002Fs>\", \"\\n\\n\"]\n)\n```\n\n**Step 3: Run batch inference**\n\nvLLM automatically batches requests for efficiency:\n\n```python\n# Process all prompts in one call\noutputs = llm.generate(prompts, sampling)\n\n# vLLM handles batching internally\n# No need to manually chunk prompts\n```\n\n**Step 4: Process results**\n\n```python\n# Extract generated text\nresults = []\nfor output in outputs:\n    prompt = output.prompt\n    generated = output.outputs[0].text\n    results.append({\n        \"prompt\": prompt,\n        \"generated\": generated,\n        \"tokens\": len(output.outputs[0].token_ids)\n    })\n\n# Save to file\nimport json\nwith open(\"results.jsonl\", \"w\") as f:\n    for result in results:\n        f.write(json.dumps(result) + \"\\n\")\n\nprint(f\"Processed {len(results)} prompts\")\n```\n\n### Workflow 3: Quantized model serving\n\nFit large models in limited GPU memory.\n\n```\nQuantization Setup:\n- [ ] Step 1: Choose quantization method\n- [ ] Step 2: Find or create quantized model\n- [ ] Step 3: Launch with quantization flag\n- [ ] Step 4: Verify accuracy\n```\n\n**Step 1: Choose quantization method**\n\n- **AWQ**: Best for 70B models, minimal accuracy loss\n- **GPTQ**: Wide model support, good compression\n- **FP8**: Fastest on H100 GPUs\n\n**Step 2: Find or create quantized model**\n\nUse pre-quantized models from HuggingFace:\n\n```bash\n# Search for AWQ models\n# Example: TheBloke\u002FLlama-2-70B-AWQ\n```\n\n**Step 3: Launch with quantization flag**\n\n```bash\n# Using pre-quantized model\nvllm serve TheBloke\u002FLlama-2-70B-AWQ \\\n  --quantization awq \\\n  --tensor-parallel-size 1 \\\n  --gpu-memory-utilization 0.95\n\n# Results: 70B model in ~40GB VRAM\n```\n\n**Step 4: Verify accuracy**\n\nTest outputs match expected quality:\n\n```python\n# Compare quantized vs non-quantized responses\n# Verify task-specific performance unchanged\n```\n\n## When to use vs alternatives\n\n**Use vLLM when:**\n- Deploying production LLM APIs (100+ req\u002Fsec)\n- Serving OpenAI-compatible endpoints\n- Limited GPU memory but need large models\n- Multi-user applications (chatbots, assistants)\n- Need low latency with high throughput\n\n**Use alternatives instead:**\n- **llama.cpp**: CPU\u002Fedge inference, single-user\n- **HuggingFace transformers**: Research, prototyping, one-off generation\n- **TensorRT-LLM**: NVIDIA-only, need absolute maximum performance\n- **Text-Generation-Inference**: Already in HuggingFace ecosystem\n\n## Common issues\n\n**Issue: Out of memory during model loading**\n\nReduce memory usage:\n```bash\nvllm serve MODEL \\\n  --gpu-memory-utilization 0.7 \\\n  --max-model-len 4096\n```\n\nOr use quantization:\n```bash\nvllm serve MODEL --quantization awq\n```\n\n**Issue: Slow first token (TTFT > 1 second)**\n\nEnable prefix caching for repeated prompts:\n```bash\nvllm serve MODEL --enable-prefix-caching\n```\n\nFor long prompts, enable chunked prefill:\n```bash\nvllm serve MODEL --enable-chunked-prefill\n```\n\n**Issue: Model not found error**\n\nUse `--trust-remote-code` for custom models:\n```bash\nvllm serve MODEL --trust-remote-code\n```\n\n**Issue: Low throughput (\u003C50 req\u002Fsec)**\n\nIncrease concurrent sequences:\n```bash\nvllm serve MODEL --max-num-seqs 512\n```\n\nCheck GPU utilization with `nvidia-smi` - should be >80%.\n\n**Issue: Inference slower than expected**\n\nVerify tensor parallelism uses power of 2 GPUs:\n```bash\nvllm serve MODEL --tensor-parallel-size 4  # Not 3\n```\n\nEnable speculative decoding for faster generation:\n```bash\nvllm serve MODEL --speculative-model DRAFT_MODEL\n```\n\n## Advanced topics\n\n**Server deployment patterns**: See [references\u002Fserver-deployment.md](references\u002Fserver-deployment.md) for Docker, Kubernetes, and load balancing configurations.\n\n**Performance optimization**: See [references\u002Foptimization.md](references\u002Foptimization.md) for PagedAttention tuning, continuous batching details, and benchmark results.\n\n**Quantization guide**: See [references\u002Fquantization.md](references\u002Fquantization.md) for AWQ\u002FGPTQ\u002FFP8 setup, model preparation, and accuracy comparisons.\n\n**Troubleshooting**: See [references\u002Ftroubleshooting.md](references\u002Ftroubleshooting.md) for detailed error messages, debugging steps, and performance diagnostics.\n\n## Hardware requirements\n\n- **Small models (7B-13B)**: 1x A10 (24GB) or A100 (40GB)\n- **Medium models (30B-40B)**: 2x A100 (40GB) with tensor parallelism\n- **Large models (70B+)**: 4x A100 (40GB) or 2x A100 (80GB), use AWQ\u002FGPTQ\n\nSupported platforms: NVIDIA (primary), AMD ROCm, Intel GPUs, TPUs\n\n## Resources\n\n- Official docs: https:\u002F\u002Fdocs.vllm.ai\n- GitHub: https:\u002F\u002Fgithub.com\u002Fvllm-project\u002Fvllm\n- Paper: \"Efficient Memory Management for Large Language Model Serving with PagedAttention\" (SOSP 2023)\n- Community: https:\u002F\u002Fdiscuss.vllm.ai\n\n## Agent operating procedure\n\n1. **Check the environment.** Confirm hardware, framework and server versions, model format, and expected load.\n2. **Pin down the inputs.** Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.\n3. **Run a small version first.** Serve or log a single request or run end to end before scaling.\n4. **Execute the full task** using the instructions and references above.\n5. **Validate the result.** Measure latency, throughput and output correctness against a reference; check resource usage and costs.\n6. **Report.** State what was run (versions, commands, parameters), what was checked, and what is still uncertain.\n\n| If this happens | Do this |\n|---|---|\n| The server fails to start or OOMs | Check model size versus memory, quantization and parallelism settings. |\n| A function, flag or endpoint in these instructions is missing in the installed version | Check the installed version's own documentation (`help()`, `--help`, official docs), adapt, and tell the user. Never invent an API. |\n| A required input, identifier or parameter is ambiguous | Ask the user, or state the assumption explicitly before running. |\n\n**Integrity rules**\n\n- Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.\n- Do not expose services or credentials publicly; confirm cloud costs before provisioning.\n- Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.\n- Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.\n\n## Related skills\n\n- `tensorrt-llm`: Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency.\n- `awq-quantization`: Activation-aware weight quantization for 4-bit LLM compression with 3x speedup and minimal accuracy loss.\n- `gptq`: Quantizes LLMs to 4-bit (also 3-bit) with GPTQ using group-wise quantization (group size 128 by default), via AutoGPTQ and transformers.\n","# vLLM - High-Performance LLM Serving\n\n## Quick start\n\nvLLM achieves 24x higher throughput than standard transformers through PagedAttention (block-based KV cache) and continuous batching (mixing prefill\u002Fdecode requests).\n\n**Installation**:\n```bash\npip install vllm\n```\n\n**Basic offline inference**:\n```python\nfrom vllm import LLM, SamplingParams\n\nllm = LLM(model=\"meta-llama\u002FLlama-3-8B-Instruct\")\nsampling = SamplingParams(temperature=0.7, max_tokens=256)\n\noutputs = llm.generate([\"Explain quantum computing\"], sampling)\nprint(outputs[0].outputs[0].text)\n```\n\n**OpenAI-compatible server**:\n```bash\nvllm serve meta-llama\u002FLlama-3-8B-Instruct\n\n# Query with OpenAI SDK\npython -c \"\nfrom openai import OpenAI\nclient = OpenAI(base_url='http:\u002F\u002Flocalhost:8000\u002Fv1', api_key='EMPTY')\nprint(client.chat.completions.create(\n    model='meta-llama\u002FLlama-3-8B-Instruct',\n    messages=[{'role': 'user', 'content': 'Hello!'}]\n).choices[0].message.content)\n\"\n```\n\n## Common workflows\n\n### Workflow 1: Production API deployment\n\nCopy this checklist and track progress:\n\n```\nDeployment Progress:\n- [ ] Step 1: Configure server settings\n- [ ] Step 2: Test with limited traffic\n- [ ] Step 3: Enable monitoring\n- [ ] Step 4: Deploy to production\n- [ ] Step 5: Verify performance metrics\n```\n\n**Step 1: Configure server settings**\n\nChoose configuration based on your model size:\n\n```bash\n# For 7B-13B models on single GPU\nvllm serve meta-llama\u002FLlama-3-8B-Instruct \\\n  --gpu-memory-utilization 0.9 \\\n  --max-model-len 8192 \\\n  --port 8000\n\n# For 30B-70B models with tensor parallelism\nvllm serve meta-llama\u002FLlama-2-70b-hf \\\n  --tensor-parallel-size 4 \\\n  --gpu-memory-utilization 0.9 \\\n  --quantization awq \\\n  --port 8000\n\n# For production with caching and metrics\nvllm serve meta-llama\u002FLlama-3-8B-Instruct \\\n  --gpu-memory-utilization 0.9 \\\n  --enable-prefix-caching \\\n  --enable-metrics \\\n  --metrics-port 9090 \\\n  --port 8000 \\\n  --host 0.0.0.0\n```\n\n**Step 2: Test with limited traffic**\n\nRun load test before production:\n\n```bash\n# Install load testing tool\npip install locust\n\n# Create test_load.py with sample requests\n# Run: locust -f test_load.py --host http:\u002F\u002Flocalhost:8000\n```\n\nVerify TTFT (time to first token) \u003C 500ms and throughput > 100 req\u002Fsec.\n\n**Step 3: Enable monitoring**\n\nvLLM exposes Prometheus metrics on port 9090:\n\n```bash\ncurl http:\u002F\u002Flocalhost:9090\u002Fmetrics | grep vllm\n```\n\nKey metrics to monitor:\n- `vllm:time_to_first_token_seconds` - Latency\n- `vllm:num_requests_running` - Active requests\n- `vllm:gpu_cache_usage_perc` - KV cache utilization\n\n**Step 4: Deploy to production**\n\nUse Docker for consistent deployment:\n\n```bash\n# Run vLLM in Docker\ndocker run --gpus all -p 8000:8000 \\\n  vllm\u002Fvllm-openai:latest \\\n  --model meta-llama\u002FLlama-3-8B-Instruct \\\n  --gpu-memory-utilization 0.9 \\\n  --enable-prefix-caching\n```\n\n**Step 5: Verify performance metrics**\n\nCheck that deployment meets targets:\n- TTFT \u003C 500ms (for short prompts)\n- Throughput > target req\u002Fsec\n- GPU utilization > 80%\n- No OOM errors in logs\n\n### Workflow 2: Offline batch inference\n\nFor processing large datasets without server overhead.\n\nCopy this checklist:\n\n```\nBatch Processing:\n- [ ] Step 1: Prepare input data\n- [ ] Step 2: Configure LLM engine\n- [ ] Step 3: Run batch inference\n- [ ] Step 4: Process results\n```\n\n**Step 1: Prepare input data**\n\n```python\n# Load prompts from file\nprompts = []\nwith open(\"prompts.txt\") as f:\n    prompts = [line.strip() for line in f]\n\nprint(f\"Loaded {len(prompts)} prompts\")\n```\n\n**Step 2: Configure LLM engine**\n\n```python\nfrom vllm import LLM, SamplingParams\n\nllm = LLM(\n    model=\"meta-llama\u002FLlama-3-8B-Instruct\",\n    tensor_parallel_size=2,  # Use 2 GPUs\n    gpu_memory_utilization=0.9,\n    max_model_len=4096\n)\n\nsampling = SamplingParams(\n    temperature=0.7,\n    top_p=0.95,\n    max_tokens=512,\n    stop=[\"\u003C\u002Fs>\", \"\\n\\n\"]\n)\n```\n\n**Step 3: Run batch inference**\n\nvLLM automatically batches requests for efficiency:\n\n```python\n# Process all prompts in one call\noutputs = llm.generate(prompts, sampling)\n\n# vLLM handles batching internally\n# No need to manually chunk prompts\n```\n\n**Step 4: Process results**\n\n```python\n# Extract generated text\nresults = []\nfor output in outputs:\n    prompt = output.prompt\n    generated = output.outputs[0].text\n    results.append({\n        \"prompt\": prompt,\n        \"generated\": generated,\n        \"tokens\": len(output.outputs[0].token_ids)\n    })\n\n# Save to file\nimport json\nwith open(\"results.jsonl\", \"w\") as f:\n    for result in results:\n        f.write(json.dumps(result) + \"\\n\")\n\nprint(f\"Processed {len(results)} prompts\")\n```\n\n### Workflow 3: Quantized model serving\n\nFit large models in limited GPU memory.\n\n```\nQuantization Setup:\n- [ ] Step 1: Choose quantization method\n- [ ] Step 2: Find or create quantized model\n- [ ] Step 3: Launch with quantization flag\n- [ ] Step 4: Verify accuracy\n```\n\n**Step 1: Choose quantization method**\n\n- **AWQ**: Best for 70B models, minimal accuracy loss\n- **GPTQ**: Wide model support, good compression\n- **FP8**: Fastest on H100 GPUs\n\n**Step 2: Find or create quantized model**\n\nUse pre-quantized models from HuggingFace:\n\n```bash\n# Search for AWQ models\n# Example: TheBloke\u002FLlama-2-70B-AWQ\n```\n\n**Step 3: Launch with quantization flag**\n\n```bash\n# Using pre-quantized model\nvllm serve TheBloke\u002FLlama-2-70B-AWQ \\\n  --quantization awq \\\n  --tensor-parallel-size 1 \\\n  --gpu-memory-utilization 0.95\n\n# Results: 70B model in ~40GB VRAM\n```\n\n**Step 4: Verify accuracy**\n\nTest outputs match expected quality:\n\n```python\n# Compare quantized vs non-quantized responses\n# Verify task-specific performance unchanged\n```\n\n## When to use vs alternatives\n\n**Use vLLM when:**\n- Deploying production LLM APIs (100+ req\u002Fsec)\n- Serving OpenAI-compatible endpoints\n- Limited GPU memory but need large models\n- Multi-user applications (chatbots, assistants)\n- Need low latency with high throughput\n\n**Use alternatives instead:**\n- **llama.cpp**: CPU\u002Fedge inference, single-user\n- **HuggingFace transformers**: Research, prototyping, one-off generation\n- **TensorRT-LLM**: NVIDIA-only, need absolute maximum performance\n- **Text-Generation-Inference**: Already in HuggingFace ecosystem\n\n## Common issues\n\n**Issue: Out of memory during model loading**\n\nReduce memory usage:\n```bash\nvllm serve MODEL \\\n  --gpu-memory-utilization 0.7 \\\n  --max-model-len 4096\n```\n\nOr use quantization:\n```bash\nvllm serve MODEL --quantization awq\n```\n\n**Issue: Slow first token (TTFT > 1 second)**\n\nEnable prefix caching for repeated prompts:\n```bash\nvllm serve MODEL --enable-prefix-caching\n```\n\nFor long prompts, enable chunked prefill:\n```bash\nvllm serve MODEL --enable-chunked-prefill\n```\n\n**Issue: Model not found error**\n\nUse `--trust-remote-code` for custom models:\n```bash\nvllm serve MODEL --trust-remote-code\n```\n\n**Issue: Low throughput (\u003C50 req\u002Fsec)**\n\nIncrease concurrent sequences:\n```bash\nvllm serve MODEL --max-num-seqs 512\n```\n\nCheck GPU utilization with `nvidia-smi` - should be >80%.\n\n**Issue: Inference slower than expected**\n\nVerify tensor parallelism uses power of 2 GPUs:\n```bash\nvllm serve MODEL --tensor-parallel-size 4  # Not 3\n```\n\nEnable speculative decoding for faster generation:\n```bash\nvllm serve MODEL --speculative-model DRAFT_MODEL\n```\n\n## Advanced topics\n\n**Server deployment patterns**: See [references\u002Fserver-deployment.md](references\u002Fserver-deployment.md) for Docker, Kubernetes, and load balancing configurations.\n\n**Performance optimization**: See [references\u002Foptimization.md](references\u002Foptimization.md) for PagedAttention tuning, continuous batching details, and benchmark results.\n\n**Quantization guide**: See [references\u002Fquantization.md](references\u002Fquantization.md) for AWQ\u002FGPTQ\u002FFP8 setup, model preparation, and accuracy comparisons.\n\n**Troubleshooting**: See [references\u002Ftroubleshooting.md](references\u002Ftroubleshooting.md) for detailed error messages, debugging steps, and performance diagnostics.\n\n## Hardware requirements\n\n- **Small models (7B-13B)**: 1x A10 (24GB) or A100 (40GB)\n- **Medium models (30B-40B)**: 2x A100 (40GB) with tensor parallelism\n- **Large models (70B+)**: 4x A100 (40GB) or 2x A100 (80GB), use AWQ\u002FGPTQ\n\nSupported platforms: NVIDIA (primary), AMD ROCm, Intel GPUs, TPUs\n\n## Resources\n\n- Official docs: https:\u002F\u002Fdocs.vllm.ai\n- GitHub: https:\u002F\u002Fgithub.com\u002Fvllm-project\u002Fvllm\n- Paper: \"Efficient Memory Management for Large Language Model Serving with PagedAttention\" (SOSP 2023)\n- Community: https:\u002F\u002Fdiscuss.vllm.ai\n\n## Agent operating procedure\n\n1. **Check the environment.** Confirm hardware, framework and server versions, model format, and expected load.\n2. **Pin down the inputs.** Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.\n3. **Run a small version first.** Serve or log a single request or run end to end before scaling.\n4. **Execute the full task** using the instructions and references above.\n5. **Validate the result.** Measure latency, throughput and output correctness against a reference; check resource usage and costs.\n6. **Report.** State what was run (versions, commands, parameters), what was checked, and what is still uncertain.\n\n| If this happens | Do this |\n|---|---|\n| The server fails to start or OOMs | Check model size versus memory, quantization and parallelism settings. |\n| A function, flag or endpoint in these instructions is missing in the installed version | Check the installed version's own documentation (`help()`, `--help`, official docs), adapt, and tell the user. Never invent an API. |\n| A required input, identifier or parameter is ambiguous | Ask the user, or state the assumption explicitly before running. |\n\n**Integrity rules**\n\n- Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.\n- Do not expose services or credentials publicly; confirm cloud costs before provisioning.\n- Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.\n- Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.\n\n## Related skills\n\n- `tensorrt-llm`: Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency.\n- `awq-quantization`: Activation-aware weight quantization for 4-bit LLM compression with 3x speedup and minimal accuracy loss.\n- `gptq`: Quantizes LLMs to 4-bit (also 3-bit) with GPTQ using group-wise quantization (group size 128 by default), via AutoGPTQ and transformers.\n",{"name":5,"license":15,"metadata":49,"description":7},{"tags":50,"version":51,"category":52,"maintainer":53,"dependencies":54},"vLLM, Inference Serving, PagedAttention, Continuous Batching, High Throughput, Production, OpenAI API, Quantization, Tensor Parallelism","1.0.0","ml-inference-and-ops","Kalaris Labs","vllm, torch, transformers","main","0da4485785ee599388900be50db1635489d53056",".agents\u002Fskills\u002Fkal-serving-llms-vllm","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6","https:\u002F\u002Fdisha6.6.thenitishkr.in\u002F","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh","www.thenitishkr","thenitishkr | AI • Cyber-security • Research\r\n","2026-11-08 03:33:25",{"source":8,"skillName":5,"commands":65,"prompt":75,"zipAvailable":76},[66,69,72],{"tool":67,"command":68},"npx","npx skills add Tashima-Tarsh\u002FDisha6.6 --skill serving-llms-vllm",{"tool":70,"command":71},"bunx","bunx skills add Tashima-Tarsh\u002FDisha6.6 --skill serving-llms-vllm",{"tool":73,"command":74},"pnpm","pnpm dlx skills add Tashima-Tarsh\u002FDisha6.6 --skill serving-llms-vllm","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fkal-serving-llms-vllm\u002FSKILL.md",true,[78,84,88,92],{"path":79,"resourceType":80,"resourceTypeDesc":81,"sizeBytes":82,"sourceHtmlUrl":83},"references\u002Foptimization.md","REFERENCE","参考文档",5769,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fkal-serving-llms-vllm\u002Freferences\u002Foptimization.md",{"path":85,"resourceType":80,"resourceTypeDesc":81,"sizeBytes":86,"sourceHtmlUrl":87},"references\u002Fquantization.md",6746,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fkal-serving-llms-vllm\u002Freferences\u002Fquantization.md",{"path":89,"resourceType":80,"resourceTypeDesc":81,"sizeBytes":90,"sourceHtmlUrl":91},"references\u002Fserver-deployment.md",5252,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fkal-serving-llms-vllm\u002Freferences\u002Fserver-deployment.md",{"path":93,"resourceType":80,"resourceTypeDesc":81,"sizeBytes":94,"sourceHtmlUrl":95},"references\u002Ftroubleshooting.md",9037,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fkal-serving-llms-vllm\u002Freferences\u002Ftroubleshooting.md",[97,127,151,178,203,227],{"skillNo":98,"name":99,"nameZh":6,"description":100,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":101,"categories":102,"keywords":106,"publishedAt":44},"01M6QRD07D4TGNDK6VJ4EHF8BV","youtube-full","Use when the user needs YouTube transcripts, video search, channel browsing, playlist extraction, or content monitoring. Trigger phrases: 'get the transcript for', 'search YouTube for', 'what are the latest videos on', 'list this playlist', 'monitor this channel', or any request involving a YouTube URL, video ID, or @handle. Do NOT use for downloading video or audio files, YouTube engagement data (likes, comments), or private\u002Fage-restricted videos.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fyoutube-full\u002FSKILL.md",[103],{"categoryNo":104,"categoryName":105,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000016","图像与多媒体",[107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126],"youtube full","youtube","full","transcript","search","monitoring","data","Video","Audio","list","files","do","get","age","trigger","URL","for","browsing","request","youtube transcripts",{"skillNo":128,"name":129,"nameZh":6,"description":130,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":131,"categories":132,"keywords":133,"publishedAt":44},"01M6QRCJ3112VYPA97YG7KGMC2","x-twitter-growth","X\u002FTwitter growth engine for building audience, crafting viral content, and analyzing engagement. Use when the user wants to grow on X\u002FTwitter, write tweets or threads, analyze their X profile, research competitors on X, plan a posting strategy, or optimize engagement. Complements social-content (generic multi-platform) with X-specific depth: algorithm mechanics, thread engineering, reply strategy, profile optimization, and competitive intelligence via web search.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fx-twitter-growth\u002FSKILL.md",[],[134,135,136,137,111,138,139,140,141,142,143,144,145,146,35,147,148,149,150,123],"twitter","x twitter growth","growth","research","content","engineering","plan","web search","competitors","analyze","engine","write","strategy","platform","grow","posting","competitive",{"skillNo":152,"name":153,"nameZh":6,"description":154,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":155,"categories":156,"keywords":160,"publishedAt":44},"01M6QRBN3EHQNJY42RERP7A149","write-a-skill","Create new agent skills with proper structure, progressive disclosure, and bundled resources. Use when user wants to create, write, build, or author a new skill.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fwrite-a-skill\u002FSKILL.md",[157],{"categoryNo":158,"categoryName":159,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000023","Skill与智能体开发",[161,145,162,163,164,165,166,167,35,168,169,55,170,171,172,173,174,175,176,177],"write a skill","github","create","https","productivity","matt pocock","direct","example","build","new skill","structure","new agent","resources","with","tree","progressive disclosure","author",{"skillNo":179,"name":180,"nameZh":6,"description":181,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":182,"categories":183,"keywords":184,"publishedAt":44},"01M6QRB6G8P33PNGAF9Y69EJ9T","wrap-up","Close out a launched Claude Managed Agent — recap every primitive the founder now owns, regenerate the single-file overview page, and suggest the next 1-2 upgrades. Use when the user says \"wrap up\", \"close this out\", \"what do I own now\", \"give me the summary\", \"recap the agent\", or when the orchestrator routes phase=wrap-up. primitives_inventory.py tables everything owned (agent, environment, session, memory, outcome, deployment); overview_page.py regenerates a self-contained .\u002Fmy-agent\u002Fagent-overview.html; upgrade_suggester.py ranks the next moves from recorded deferrals plus standing hardening steps. Companion to run-without-you; the last stop before phase=done.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fwrap-up\u002FSKILL.md",[],[185,186,187,188,189,190,191,192,193,35,194,118,195,196,197,198,199,200,201,202],"wrap up","up","wrap","gemini","deployment","run","file","hardening","closeout","stop","upgrade","founder","close","companion","you","tables","directions","phase",{"skillNo":204,"name":205,"nameZh":6,"description":206,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":207,"categories":208,"keywords":210,"publishedAt":44},"01M6QRA5AYC1RBFZ39GTWGS3N8","workflow-builder","Design and write deterministic multi-agent workflow scripts (.js files in .claude\u002Fworkflows\u002F) for Claude Code's Workflow tool. Use when a user wants to build, create, author, scaffold, or run a custom Claude Code workflow, orchestrate sub-agents (fan-out, pipeline, loop, judge-panel), or automate a repeatable multi-step task across fresh-context agents.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fworkflow-builder\u002FSKILL.md",[209],{"categoryNo":158,"categoryName":159,"taxonomyType":21,"taxonomyTypeDesc":22},[211,212,213,214,215,163,190,216,217,218,219,220,145,221,117,222,223,224,225,226],"workflow","workflow builder","builder","design","workflows","code","orchestrate","pipeline","loop","multi-agent","context","across","fresh","scripts","judge","claude code workflow",{"skillNo":228,"name":229,"nameZh":6,"description":230,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":231,"categories":232,"keywords":233,"publishedAt":44},"01M6QR7QMCZESKQA0ERW1E657M","weekly-review","Use when someone wants to run a weekly review, close open loops, audit stalled projects and commitments, get their system back to trusted, restart a lapsed review habit, or says \"\u002Fcs:weekly-review\". Walks David Allen's three-phase loop — GET CLEAR, GET CURRENT, GET CREATIVE — with deterministic scripts that inventory open loops, gate the checklist with named gaps, and score commitment health 0-100.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fweekly-review\u002FSKILL.md",[],[234,235,236,237,238,239,240,241,242,219,243,244,245,246,247,248,119,249,250,202],"review","weekly review","weekly","deterministic scripts","reflect","feeds","inventory","open","team","projects","project-management","scanner","conversation","back","project","sprint","cadence"]