[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6KQ2CQNSSM76QDB2MT9NQCN":3,"skill-same:01M6KQ2CQNSSM76QDB2MT9NQCN":85},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":51,"commitSha":52,"skillDirectory":53,"forksCount":13,"repositoryHtmlUrl":54,"repositoryHomepageUrl":6,"ownerHtmlUrl":55,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":56,"install":57,"resources":70},"01M6KQ2CQNSSM76QDB2MT9NQCN","sglang",null,"Fast structured generation and serving for LLMs with RadixAttention prefix caching. Use for JSON\u002Fregex outputs, constrained decoding, agentic workflows with tool calls, or when you need 5× faster inference than vLLM with prefix sharing. Powers 300,000+ GPUs at xAI, AMD, NVIDIA, and LinkedIn.","endman100\u002Fskill-router-nexus","endman100","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F31997019?v=4","01M6KPDX17BYDWW4RRB0TNGGMX","01M6KPDX64Y2QBEWVFG5W5PWPF",0,"MIT","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Inference-and-Optimization\u002Fsglang\u002FSKILL.md",[],[18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"SGLang","Constrained Decoding","when","at","you","regex","caching","xai","Transformers","Production Scale","Fast Inference","JSON Output","agentic","Prefix Caching","RadixAttention","Structured Generation","Inference Serving","NVIDIA","agentic workflows","workflows","llms","2026-11-10 10:12:30",false,"---\nname: sglang\ndescription: Fast structured generation and serving for LLMs with RadixAttention prefix caching. Use for JSON\u002Fregex outputs, constrained decoding, agentic workflows with tool calls, or when you need 5× faster inference than vLLM with prefix sharing. Powers 300,000+ GPUs at xAI, AMD, NVIDIA, and LinkedIn.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Inference Serving, SGLang, Structured Generation, RadixAttention, Prefix Caching, Constrained Decoding, Agents, JSON Output, Fast Inference, Production Scale]\ndependencies: [sglang, torch, transformers]\n---\n\n# SGLang\n\nHigh-performance serving framework for LLMs and VLMs with RadixAttention for automatic prefix caching.\n\n## When to use SGLang\n\n**Use SGLang when:**\n- Need structured outputs (JSON, regex, grammar)\n- Building agents with repeated prefixes (system prompts, tools)\n- Agentic workflows with function calling\n- Multi-turn conversations with shared context\n- Need faster JSON decoding (3× vs standard)\n\n**Use vLLM instead when:**\n- Simple text generation without structure\n- Don't need prefix caching\n- Want mature, widely-tested production system\n\n**Use TensorRT-LLM instead when:**\n- Maximum single-request latency (no batching needed)\n- NVIDIA-only deployment\n- Need FP8\u002FINT4 quantization on H100\n\n## Quick start\n\n### Installation\n\n```bash\n# pip install (recommended)\npip install \"sglang[all]\"\n\n# With FlashInfer (faster, CUDA 11.8\u002F12.1)\npip install sglang[all] flashinfer -i https:\u002F\u002Fflashinfer.ai\u002Fwhl\u002Fcu121\u002Ftorch2.4\u002F\n\n# From source\ngit clone https:\u002F\u002Fgithub.com\u002Fsgl-project\u002Fsglang.git\ncd sglang\npip install -e \"python[all]\"\n```\n\n### Launch server\n\n```bash\n# Basic server (Llama 3-8B)\npython -m sglang.launch_server \\\n    --model-path meta-llama\u002FMeta-Llama-3-8B-Instruct \\\n    --port 30000\n\n# With RadixAttention (automatic prefix caching)\npython -m sglang.launch_server \\\n    --model-path meta-llama\u002FMeta-Llama-3-8B-Instruct \\\n    --port 30000 \\\n    --enable-radix-cache  # Default: enabled\n\n# Multi-GPU (tensor parallelism)\npython -m sglang.launch_server \\\n    --model-path meta-llama\u002FMeta-Llama-3-70B-Instruct \\\n    --tp 4 \\\n    --port 30000\n```\n\n### Basic inference\n\n```python\nimport sglang as sgl\n\n# Set backend\nsgl.set_default_backend(sgl.OpenAI(\"http:\u002F\u002Flocalhost:30000\u002Fv1\"))\n\n# Simple generation\n@sgl.function\ndef simple_gen(s, question):\n    s += \"Q: \" + question + \"\\n\"\n    s += \"A:\" + sgl.gen(\"answer\", max_tokens=100)\n\n# Run\nstate = simple_gen.run(question=\"What is the capital of France?\")\nprint(state[\"answer\"])\n# Output: \"The capital of France is Paris.\"\n```\n\n### Structured JSON output\n\n```python\nimport sglang as sgl\n\n@sgl.function\ndef extract_person(s, text):\n    s += f\"Extract person information from: {text}\\n\"\n    s += \"Output JSON:\\n\"\n\n    # Constrained JSON generation\n    s += sgl.gen(\n        \"json_output\",\n        max_tokens=200,\n        regex=r'\\{\"name\": \"[^\"]+\", \"age\": \\d+, \"occupation\": \"[^\"]+\"\\}'\n    )\n\n# Run\nstate = extract_person.run(\n    text=\"John Smith is a 35-year-old software engineer.\"\n)\nprint(state[\"json_output\"])\n# Output: {\"name\": \"John Smith\", \"age\": 35, \"occupation\": \"software engineer\"}\n```\n\n## RadixAttention (Key Innovation)\n\n**What it does**: Automatically caches and reuses common prefixes across requests.\n\n**Performance**:\n- **5× faster** for agentic workloads with shared system prompts\n- **10× faster** for few-shot prompting with repeated examples\n- **Zero configuration** - works automatically\n\n**How it works**:\n1. Builds radix tree of all processed tokens\n2. Automatically detects shared prefixes\n3. Reuses KV cache for matching prefixes\n4. Only computes new tokens\n\n**Example** (Agent with system prompt):\n\n```\nRequest 1: [SYSTEM_PROMPT] + \"What's the weather?\"\n→ Computes full prompt (1000 tokens)\n\nRequest 2: [SAME_SYSTEM_PROMPT] + \"Book a flight\"\n→ Reuses system prompt KV cache (998 tokens)\n→ Only computes 2 new tokens\n→ 5× faster!\n```\n\n## Structured generation patterns\n\n### JSON with schema\n\n```python\n@sgl.function\ndef structured_extraction(s, article):\n    s += f\"Article: {article}\\n\\n\"\n    s += \"Extract key information as JSON:\\n\"\n\n    # JSON schema constraint\n    schema = {\n        \"type\": \"object\",\n        \"properties\": {\n            \"title\": {\"type\": \"string\"},\n            \"author\": {\"type\": \"string\"},\n            \"summary\": {\"type\": \"string\"},\n            \"sentiment\": {\"type\": \"string\", \"enum\": [\"positive\", \"negative\", \"neutral\"]}\n        },\n        \"required\": [\"title\", \"author\", \"summary\", \"sentiment\"]\n    }\n\n    s += sgl.gen(\"info\", max_tokens=300, json_schema=schema)\n\nstate = structured_extraction.run(article=\"...\")\nprint(state[\"info\"])\n# Output: Valid JSON matching schema\n```\n\n### Regex-constrained generation\n\n```python\n@sgl.function\ndef extract_email(s, text):\n    s += f\"Extract email from: {text}\\n\"\n    s += \"Email: \"\n\n    # Email regex pattern\n    s += sgl.gen(\n        \"email\",\n        max_tokens=50,\n        regex=r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}'\n    )\n\nstate = extract_email.run(text=\"Contact john.doe@example.com for details\")\nprint(state[\"email\"])\n# Output: \"john.doe@example.com\"\n```\n\n### Grammar-based generation\n\n```python\n@sgl.function\ndef generate_code(s, description):\n    s += f\"Generate Python code for: {description}\\n\"\n    s += \"```python\\n\"\n\n    # EBNF grammar for Python\n    python_grammar = \"\"\"\n    ?start: function_def\n    function_def: \"def\" NAME \"(\" [parameters] \"):\" suite\n    parameters: parameter (\",\" parameter)*\n    parameter: NAME\n    suite: simple_stmt | NEWLINE INDENT stmt+ DEDENT\n    \"\"\"\n\n    s += sgl.gen(\"code\", max_tokens=200, grammar=python_grammar)\n    s += \"\\n```\"\n```\n\n## Agent workflows with function calling\n\n```python\nimport sglang as sgl\n\n# Define tools\ntools = [\n    {\n        \"name\": \"get_weather\",\n        \"description\": \"Get weather for a location\",\n        \"parameters\": {\n            \"type\": \"object\",\n            \"properties\": {\n                \"location\": {\"type\": \"string\"}\n            }\n        }\n    },\n    {\n        \"name\": \"book_flight\",\n        \"description\": \"Book a flight\",\n        \"parameters\": {\n            \"type\": \"object\",\n            \"properties\": {\n                \"from\": {\"type\": \"string\"},\n                \"to\": {\"type\": \"string\"},\n                \"date\": {\"type\": \"string\"}\n            }\n        }\n    }\n]\n\n@sgl.function\ndef agent_workflow(s, user_query, tools):\n    # System prompt (cached with RadixAttention)\n    s += \"You are a helpful assistant with access to tools.\\n\"\n    s += f\"Available tools: {tools}\\n\\n\"\n\n    # User query\n    s += f\"User: {user_query}\\n\"\n    s += \"Assistant: \"\n\n    # Generate with function calling\n    s += sgl.gen(\n        \"response\",\n        max_tokens=200,\n        tools=tools,  # SGLang handles tool call format\n        stop=[\"User:\", \"\\n\\n\"]\n    )\n\n# Multiple queries reuse system prompt\nstate1 = agent_workflow.run(\n    user_query=\"What's the weather in NYC?\",\n    tools=tools\n)\n# First call: Computes full system prompt\n\nstate2 = agent_workflow.run(\n    user_query=\"Book a flight to LA\",\n    tools=tools\n)\n# Second call: Reuses system prompt (5× faster)\n```\n\n## Performance benchmarks\n\n### RadixAttention speedup\n\n**Few-shot prompting** (10 examples in prompt):\n- vLLM: 2.5 sec\u002Frequest\n- SGLang: **0.25 sec\u002Frequest** (10× faster)\n- Throughput: 4× higher\n\n**Agent workflows** (1000-token system prompt):\n- vLLM: 1.8 sec\u002Frequest\n- SGLang: **0.35 sec\u002Frequest** (5× faster)\n\n**JSON decoding**:\n- Standard: 45 tok\u002Fs\n- SGLang: **135 tok\u002Fs** (3× faster)\n\n### Throughput (Llama 3-8B, A100)\n\n| Workload | vLLM | SGLang | Speedup |\n|----------|------|--------|---------|\n| Simple generation | 2500 tok\u002Fs | 2800 tok\u002Fs | 1.12× |\n| Few-shot (10 examples) | 500 tok\u002Fs | 5000 tok\u002Fs | 10× |\n| Agent (tool calls) | 800 tok\u002Fs | 4000 tok\u002Fs | 5× |\n| JSON output | 600 tok\u002Fs | 2400 tok\u002Fs | 4× |\n\n## Multi-turn conversations\n\n```python\n@sgl.function\ndef multi_turn_chat(s, history, new_message):\n    # System prompt (always cached)\n    s += \"You are a helpful AI assistant.\\n\\n\"\n\n    # Conversation history (cached as it grows)\n    for msg in history:\n        s += f\"{msg['role']}: {msg['content']}\\n\"\n\n    # New user message (only new part)\n    s += f\"User: {new_message}\\n\"\n    s += \"Assistant: \"\n    s += sgl.gen(\"response\", max_tokens=200)\n\n# Turn 1\nhistory = []\nstate = multi_turn_chat.run(history=history, new_message=\"Hi there!\")\nhistory.append({\"role\": \"User\", \"content\": \"Hi there!\"})\nhistory.append({\"role\": \"Assistant\", \"content\": state[\"response\"]})\n\n# Turn 2 (reuses Turn 1 KV cache)\nstate = multi_turn_chat.run(history=history, new_message=\"What's 2+2?\")\n# Only computes new message (much faster!)\n\n# Turn 3 (reuses Turn 1 + Turn 2 KV cache)\nstate = multi_turn_chat.run(history=history, new_message=\"Tell me a joke\")\n# Progressively faster as history grows\n```\n\n## Advanced features\n\n### Speculative decoding\n\n```bash\n# Launch with draft model (2-3× faster)\npython -m sglang.launch_server \\\n    --model-path meta-llama\u002FMeta-Llama-3-70B-Instruct \\\n    --speculative-model meta-llama\u002FMeta-Llama-3-8B-Instruct \\\n    --speculative-num-steps 5\n```\n\n### Multi-modal (vision models)\n\n```python\n@sgl.function\ndef describe_image(s, image_path):\n    s += sgl.image(image_path)\n    s += \"Describe this image in detail: \"\n    s += sgl.gen(\"description\", max_tokens=200)\n\nstate = describe_image.run(image_path=\"photo.jpg\")\nprint(state[\"description\"])\n```\n\n### Batching and parallel requests\n\n```python\n# Automatic batching (continuous batching)\nstates = sgl.run_batch(\n    [\n        simple_gen.bind(question=\"What is AI?\"),\n        simple_gen.bind(question=\"What is ML?\"),\n        simple_gen.bind(question=\"What is DL?\"),\n    ]\n)\n\n# All 3 processed in single batch (efficient)\n```\n\n## OpenAI-compatible API\n\n```bash\n# Start server with OpenAI API\npython -m sglang.launch_server \\\n    --model-path meta-llama\u002FMeta-Llama-3-8B-Instruct \\\n    --port 30000\n\n# Use with OpenAI client\ncurl http:\u002F\u002Flocalhost:30000\u002Fv1\u002Fchat\u002Fcompletions \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"model\": \"default\",\n    \"messages\": [\n      {\"role\": \"system\", \"content\": \"You are helpful\"},\n      {\"role\": \"user\", \"content\": \"Hello\"}\n    ],\n    \"temperature\": 0.7,\n    \"max_tokens\": 100\n  }'\n\n# Works with OpenAI Python SDK\nfrom openai import OpenAI\nclient = OpenAI(base_url=\"http:\u002F\u002Flocalhost:30000\u002Fv1\", api_key=\"EMPTY\")\n\nresponse = client.chat.completions.create(\n    model=\"default\",\n    messages=[{\"role\": \"user\", \"content\": \"Hello\"}]\n)\n```\n\n## Supported models\n\n**Text models**:\n- Llama 2, Llama 3, Llama 3.1, Llama 3.2\n- Mistral, Mixtral\n- Qwen, Qwen2, QwQ\n- DeepSeek-V2, DeepSeek-V3\n- Gemma, Phi-3\n\n**Vision models**:\n- LLaVA, LLaVA-OneVision\n- Phi-3-Vision\n- Qwen2-VL\n\n**100+ models** from HuggingFace\n\n## Hardware support\n\n**NVIDIA**: A100, H100, L4, T4 (CUDA 11.8+)\n**AMD**: MI300, MI250 (ROCm 6.0+)\n**Intel**: Xeon with GPU (coming soon)\n**Apple**: M1\u002FM2\u002FM3 via MPS (experimental)\n\n## References\n\n- **[Structured Generation Guide](references\u002Fstructured-generation.md)** - JSON schemas, regex, grammars, validation\n- **[RadixAttention Deep Dive](references\u002Fradix-attention.md)** - How it works, optimization, benchmarks\n- **[Production Deployment](references\u002Fdeployment.md)** - Multi-GPU, monitoring, autoscaling\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002Fsgl-project\u002Fsglang\n- **Docs**: https:\u002F\u002Fsgl-project.github.io\u002F\n- **Paper**: RadixAttention (arXiv:2312.07104)\n- **Discord**: https:\u002F\u002Fdiscord.gg\u002Fsglang\n\n\n","# SGLang\n\nHigh-performance serving framework for LLMs and VLMs with RadixAttention for automatic prefix caching.\n\n## When to use SGLang\n\n**Use SGLang when:**\n- Need structured outputs (JSON, regex, grammar)\n- Building agents with repeated prefixes (system prompts, tools)\n- Agentic workflows with function calling\n- Multi-turn conversations with shared context\n- Need faster JSON decoding (3× vs standard)\n\n**Use vLLM instead when:**\n- Simple text generation without structure\n- Don't need prefix caching\n- Want mature, widely-tested production system\n\n**Use TensorRT-LLM instead when:**\n- Maximum single-request latency (no batching needed)\n- NVIDIA-only deployment\n- Need FP8\u002FINT4 quantization on H100\n\n## Quick start\n\n### Installation\n\n```bash\n# pip install (recommended)\npip install \"sglang[all]\"\n\n# With FlashInfer (faster, CUDA 11.8\u002F12.1)\npip install sglang[all] flashinfer -i https:\u002F\u002Fflashinfer.ai\u002Fwhl\u002Fcu121\u002Ftorch2.4\u002F\n\n# From source\ngit clone https:\u002F\u002Fgithub.com\u002Fsgl-project\u002Fsglang.git\ncd sglang\npip install -e \"python[all]\"\n```\n\n### Launch server\n\n```bash\n# Basic server (Llama 3-8B)\npython -m sglang.launch_server \\\n    --model-path meta-llama\u002FMeta-Llama-3-8B-Instruct \\\n    --port 30000\n\n# With RadixAttention (automatic prefix caching)\npython -m sglang.launch_server \\\n    --model-path meta-llama\u002FMeta-Llama-3-8B-Instruct \\\n    --port 30000 \\\n    --enable-radix-cache  # Default: enabled\n\n# Multi-GPU (tensor parallelism)\npython -m sglang.launch_server \\\n    --model-path meta-llama\u002FMeta-Llama-3-70B-Instruct \\\n    --tp 4 \\\n    --port 30000\n```\n\n### Basic inference\n\n```python\nimport sglang as sgl\n\n# Set backend\nsgl.set_default_backend(sgl.OpenAI(\"http:\u002F\u002Flocalhost:30000\u002Fv1\"))\n\n# Simple generation\n@sgl.function\ndef simple_gen(s, question):\n    s += \"Q: \" + question + \"\\n\"\n    s += \"A:\" + sgl.gen(\"answer\", max_tokens=100)\n\n# Run\nstate = simple_gen.run(question=\"What is the capital of France?\")\nprint(state[\"answer\"])\n# Output: \"The capital of France is Paris.\"\n```\n\n### Structured JSON output\n\n```python\nimport sglang as sgl\n\n@sgl.function\ndef extract_person(s, text):\n    s += f\"Extract person information from: {text}\\n\"\n    s += \"Output JSON:\\n\"\n\n    # Constrained JSON generation\n    s += sgl.gen(\n        \"json_output\",\n        max_tokens=200,\n        regex=r'\\{\"name\": \"[^\"]+\", \"age\": \\d+, \"occupation\": \"[^\"]+\"\\}'\n    )\n\n# Run\nstate = extract_person.run(\n    text=\"John Smith is a 35-year-old software engineer.\"\n)\nprint(state[\"json_output\"])\n# Output: {\"name\": \"John Smith\", \"age\": 35, \"occupation\": \"software engineer\"}\n```\n\n## RadixAttention (Key Innovation)\n\n**What it does**: Automatically caches and reuses common prefixes across requests.\n\n**Performance**:\n- **5× faster** for agentic workloads with shared system prompts\n- **10× faster** for few-shot prompting with repeated examples\n- **Zero configuration** - works automatically\n\n**How it works**:\n1. Builds radix tree of all processed tokens\n2. Automatically detects shared prefixes\n3. Reuses KV cache for matching prefixes\n4. Only computes new tokens\n\n**Example** (Agent with system prompt):\n\n```\nRequest 1: [SYSTEM_PROMPT] + \"What's the weather?\"\n→ Computes full prompt (1000 tokens)\n\nRequest 2: [SAME_SYSTEM_PROMPT] + \"Book a flight\"\n→ Reuses system prompt KV cache (998 tokens)\n→ Only computes 2 new tokens\n→ 5× faster!\n```\n\n## Structured generation patterns\n\n### JSON with schema\n\n```python\n@sgl.function\ndef structured_extraction(s, article):\n    s += f\"Article: {article}\\n\\n\"\n    s += \"Extract key information as JSON:\\n\"\n\n    # JSON schema constraint\n    schema = {\n        \"type\": \"object\",\n        \"properties\": {\n            \"title\": {\"type\": \"string\"},\n            \"author\": {\"type\": \"string\"},\n            \"summary\": {\"type\": \"string\"},\n            \"sentiment\": {\"type\": \"string\", \"enum\": [\"positive\", \"negative\", \"neutral\"]}\n        },\n        \"required\": [\"title\", \"author\", \"summary\", \"sentiment\"]\n    }\n\n    s += sgl.gen(\"info\", max_tokens=300, json_schema=schema)\n\nstate = structured_extraction.run(article=\"...\")\nprint(state[\"info\"])\n# Output: Valid JSON matching schema\n```\n\n### Regex-constrained generation\n\n```python\n@sgl.function\ndef extract_email(s, text):\n    s += f\"Extract email from: {text}\\n\"\n    s += \"Email: \"\n\n    # Email regex pattern\n    s += sgl.gen(\n        \"email\",\n        max_tokens=50,\n        regex=r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}'\n    )\n\nstate = extract_email.run(text=\"Contact john.doe@example.com for details\")\nprint(state[\"email\"])\n# Output: \"john.doe@example.com\"\n```\n\n### Grammar-based generation\n\n```python\n@sgl.function\ndef generate_code(s, description):\n    s += f\"Generate Python code for: {description}\\n\"\n    s += \"```python\\n\"\n\n    # EBNF grammar for Python\n    python_grammar = \"\"\"\n    ?start: function_def\n    function_def: \"def\" NAME \"(\" [parameters] \"):\" suite\n    parameters: parameter (\",\" parameter)*\n    parameter: NAME\n    suite: simple_stmt | NEWLINE INDENT stmt+ DEDENT\n    \"\"\"\n\n    s += sgl.gen(\"code\", max_tokens=200, grammar=python_grammar)\n    s += \"\\n```\"\n```\n\n## Agent workflows with function calling\n\n```python\nimport sglang as sgl\n\n# Define tools\ntools = [\n    {\n        \"name\": \"get_weather\",\n        \"description\": \"Get weather for a location\",\n        \"parameters\": {\n            \"type\": \"object\",\n            \"properties\": {\n                \"location\": {\"type\": \"string\"}\n            }\n        }\n    },\n    {\n        \"name\": \"book_flight\",\n        \"description\": \"Book a flight\",\n        \"parameters\": {\n            \"type\": \"object\",\n            \"properties\": {\n                \"from\": {\"type\": \"string\"},\n                \"to\": {\"type\": \"string\"},\n                \"date\": {\"type\": \"string\"}\n            }\n        }\n    }\n]\n\n@sgl.function\ndef agent_workflow(s, user_query, tools):\n    # System prompt (cached with RadixAttention)\n    s += \"You are a helpful assistant with access to tools.\\n\"\n    s += f\"Available tools: {tools}\\n\\n\"\n\n    # User query\n    s += f\"User: {user_query}\\n\"\n    s += \"Assistant: \"\n\n    # Generate with function calling\n    s += sgl.gen(\n        \"response\",\n        max_tokens=200,\n        tools=tools,  # SGLang handles tool call format\n        stop=[\"User:\", \"\\n\\n\"]\n    )\n\n# Multiple queries reuse system prompt\nstate1 = agent_workflow.run(\n    user_query=\"What's the weather in NYC?\",\n    tools=tools\n)\n# First call: Computes full system prompt\n\nstate2 = agent_workflow.run(\n    user_query=\"Book a flight to LA\",\n    tools=tools\n)\n# Second call: Reuses system prompt (5× faster)\n```\n\n## Performance benchmarks\n\n### RadixAttention speedup\n\n**Few-shot prompting** (10 examples in prompt):\n- vLLM: 2.5 sec\u002Frequest\n- SGLang: **0.25 sec\u002Frequest** (10× faster)\n- Throughput: 4× higher\n\n**Agent workflows** (1000-token system prompt):\n- vLLM: 1.8 sec\u002Frequest\n- SGLang: **0.35 sec\u002Frequest** (5× faster)\n\n**JSON decoding**:\n- Standard: 45 tok\u002Fs\n- SGLang: **135 tok\u002Fs** (3× faster)\n\n### Throughput (Llama 3-8B, A100)\n\n| Workload | vLLM | SGLang | Speedup |\n|----------|------|--------|---------|\n| Simple generation | 2500 tok\u002Fs | 2800 tok\u002Fs | 1.12× |\n| Few-shot (10 examples) | 500 tok\u002Fs | 5000 tok\u002Fs | 10× |\n| Agent (tool calls) | 800 tok\u002Fs | 4000 tok\u002Fs | 5× |\n| JSON output | 600 tok\u002Fs | 2400 tok\u002Fs | 4× |\n\n## Multi-turn conversations\n\n```python\n@sgl.function\ndef multi_turn_chat(s, history, new_message):\n    # System prompt (always cached)\n    s += \"You are a helpful AI assistant.\\n\\n\"\n\n    # Conversation history (cached as it grows)\n    for msg in history:\n        s += f\"{msg['role']}: {msg['content']}\\n\"\n\n    # New user message (only new part)\n    s += f\"User: {new_message}\\n\"\n    s += \"Assistant: \"\n    s += sgl.gen(\"response\", max_tokens=200)\n\n# Turn 1\nhistory = []\nstate = multi_turn_chat.run(history=history, new_message=\"Hi there!\")\nhistory.append({\"role\": \"User\", \"content\": \"Hi there!\"})\nhistory.append({\"role\": \"Assistant\", \"content\": state[\"response\"]})\n\n# Turn 2 (reuses Turn 1 KV cache)\nstate = multi_turn_chat.run(history=history, new_message=\"What's 2+2?\")\n# Only computes new message (much faster!)\n\n# Turn 3 (reuses Turn 1 + Turn 2 KV cache)\nstate = multi_turn_chat.run(history=history, new_message=\"Tell me a joke\")\n# Progressively faster as history grows\n```\n\n## Advanced features\n\n### Speculative decoding\n\n```bash\n# Launch with draft model (2-3× faster)\npython -m sglang.launch_server \\\n    --model-path meta-llama\u002FMeta-Llama-3-70B-Instruct \\\n    --speculative-model meta-llama\u002FMeta-Llama-3-8B-Instruct \\\n    --speculative-num-steps 5\n```\n\n### Multi-modal (vision models)\n\n```python\n@sgl.function\ndef describe_image(s, image_path):\n    s += sgl.image(image_path)\n    s += \"Describe this image in detail: \"\n    s += sgl.gen(\"description\", max_tokens=200)\n\nstate = describe_image.run(image_path=\"photo.jpg\")\nprint(state[\"description\"])\n```\n\n### Batching and parallel requests\n\n```python\n# Automatic batching (continuous batching)\nstates = sgl.run_batch(\n    [\n        simple_gen.bind(question=\"What is AI?\"),\n        simple_gen.bind(question=\"What is ML?\"),\n        simple_gen.bind(question=\"What is DL?\"),\n    ]\n)\n\n# All 3 processed in single batch (efficient)\n```\n\n## OpenAI-compatible API\n\n```bash\n# Start server with OpenAI API\npython -m sglang.launch_server \\\n    --model-path meta-llama\u002FMeta-Llama-3-8B-Instruct \\\n    --port 30000\n\n# Use with OpenAI client\ncurl http:\u002F\u002Flocalhost:30000\u002Fv1\u002Fchat\u002Fcompletions \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"model\": \"default\",\n    \"messages\": [\n      {\"role\": \"system\", \"content\": \"You are helpful\"},\n      {\"role\": \"user\", \"content\": \"Hello\"}\n    ],\n    \"temperature\": 0.7,\n    \"max_tokens\": 100\n  }'\n\n# Works with OpenAI Python SDK\nfrom openai import OpenAI\nclient = OpenAI(base_url=\"http:\u002F\u002Flocalhost:30000\u002Fv1\", api_key=\"EMPTY\")\n\nresponse = client.chat.completions.create(\n    model=\"default\",\n    messages=[{\"role\": \"user\", \"content\": \"Hello\"}]\n)\n```\n\n## Supported models\n\n**Text models**:\n- Llama 2, Llama 3, Llama 3.1, Llama 3.2\n- Mistral, Mixtral\n- Qwen, Qwen2, QwQ\n- DeepSeek-V2, DeepSeek-V3\n- Gemma, Phi-3\n\n**Vision models**:\n- LLaVA, LLaVA-OneVision\n- Phi-3-Vision\n- Qwen2-VL\n\n**100+ models** from HuggingFace\n\n## Hardware support\n\n**NVIDIA**: A100, H100, L4, T4 (CUDA 11.8+)\n**AMD**: MI300, MI250 (ROCm 6.0+)\n**Intel**: Xeon with GPU (coming soon)\n**Apple**: M1\u002FM2\u002FM3 via MPS (experimental)\n\n## References\n\n- **[Structured Generation Guide](references\u002Fstructured-generation.md)** - JSON schemas, regex, grammars, validation\n- **[RadixAttention Deep Dive](references\u002Fradix-attention.md)** - How it works, optimization, benchmarks\n- **[Production Deployment](references\u002Fdeployment.md)** - Multi-GPU, monitoring, autoscaling\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002Fsgl-project\u002Fsglang\n- **Docs**: https:\u002F\u002Fsgl-project.github.io\u002F\n- **Paper**: RadixAttention (arXiv:2312.07104)\n- **Discord**: https:\u002F\u002Fdiscord.gg\u002Fsglang\n\n\n",{"name":5,"tags":44,"author":46,"license":14,"version":47,"description":7,"dependencies":48},[34,18,33,32,31,19,45,29,28,27],"Agents","Orchestra Research","1.0.0",[5,49,50],"torch","transformers","main","f19b254e8975d0bfddb394686706106930d7964d","AI-Inference-and-Optimization\u002Fsglang","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus","https:\u002F\u002Fgithub.com\u002Fendman100","2026-11-13 20:29:52",{"source":8,"skillName":5,"commands":58,"prompt":68,"zipAvailable":69},[59,62,65],{"tool":60,"command":61},"npx","npx skills add endman100\u002Fskill-router-nexus --skill sglang",{"tool":63,"command":64},"bunx","bunx skills add endman100\u002Fskill-router-nexus --skill sglang",{"tool":66,"command":67},"pnpm","pnpm dlx skills add endman100\u002Fskill-router-nexus --skill sglang","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Inference-and-Optimization\u002Fsglang\u002FSKILL.md",true,[71,77,81],{"path":72,"resourceType":73,"resourceTypeDesc":74,"sizeBytes":75,"sourceHtmlUrl":76},"references\u002Fdeployment.md","REFERENCE","参考文档",10020,"https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Inference-and-Optimization\u002Fsglang\u002Freferences\u002Fdeployment.md",{"path":78,"resourceType":73,"resourceTypeDesc":74,"sizeBytes":79,"sourceHtmlUrl":80},"references\u002Fradix-attention.md",11005,"https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Inference-and-Optimization\u002Fsglang\u002Freferences\u002Fradix-attention.md",{"path":82,"resourceType":73,"resourceTypeDesc":74,"sizeBytes":83,"sourceHtmlUrl":84},"references\u002Fstructured-generation.md",13468,"https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Inference-and-Optimization\u002Fsglang\u002Freferences\u002Fstructured-generation.md",[86,110,147,169,178,202],{"skillNo":87,"name":88,"nameZh":6,"description":89,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":90,"sourceHtmlUrl":91,"categories":92,"keywords":98,"publishedAt":39},"01M6MPCT5BXVER8HQZTW3QE9AS","create-master","基于佛教经典文献，生成特定高僧大德的 AI 教学角色","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FMetaphysics-and-Spirituality\u002FMaster-skill\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FMetaphysics-and-Spirituality\u002FMaster-skill\u002FSKILL.md",[93],{"categoryNo":94,"categoryName":95,"taxonomyType":96,"taxonomyTypeDesc":97},"01JBSEEDCAT000000000000039","教师与教育","PROFESSION","适用职业",[99,100,101,102,103,104,105,106,107,108,109],"create master","master","create","生成","true","webfetch","edit","write","read","grep","Bash",{"skillNo":111,"name":112,"nameZh":6,"description":113,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":114,"sourceHtmlUrl":115,"categories":116,"keywords":128,"publishedAt":39},"01M6M43AKE95Y64E1JRXEF5QQ3","writing-dna-skill","从至少 20 篇完整文章中蒸馏可复用的写作 DNA，分析语言、文章结构、选题逻辑、素材策略、认知框架和视觉风格，并生成 Writing-DNA.md；按该 DNA 写作时会读取全部蒸馏产物和 5 篇相关原文。用于中英文作者、账号、品牌或出版物的风格分析与一致性写作。 Distill reusable Writing DNA from at least 20 complete articles for Chinese or English authors, publications, brands, and accounts; use for language, structure, topic logic, source strategy, cognitive-frame, and visual-style analysis.","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fwriting-dna-skill\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fwriting-dna-skill\u002FSKILL.md",[117,122,125],{"categoryNo":118,"categoryName":119,"taxonomyType":120,"taxonomyTypeDesc":121},"01JBSEEDCAT000000000000018","写作与文案","SCENARIO","使用场景",{"categoryNo":123,"categoryName":124,"taxonomyType":120,"taxonomyTypeDesc":121},"01JBSEEDCAT000000000000023","Skill与智能体开发",{"categoryNo":126,"categoryName":127,"taxonomyType":96,"taxonomyTypeDesc":97},"01JBSEEDCAT000000000000036","内容创作者",[129,130,131,132,133,134,135,136,137,138,139,140,141,102,142,143,144,145,21,146],"writing dna skill","DNA","writing","strategy","english","logic","visual","structure","chinese","reusable","use","style","distill","analysis","frame","source","for","complete",{"skillNo":148,"name":149,"nameZh":6,"description":150,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":151,"sourceHtmlUrl":152,"categories":153,"keywords":156,"publishedAt":39},"01M6M428TTCGQ4ME3KY99VN037","stop-slop","Remove AI writing patterns from prose. Use when drafting, editing, or reviewing text to eliminate predictable AI tells.","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fstop-slop\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fstop-slop\u002FSKILL.md",[154,155],{"categoryNo":118,"categoryName":119,"taxonomyType":120,"taxonomyTypeDesc":121},{"categoryNo":126,"categoryName":127,"taxonomyType":96,"taxonomyTypeDesc":97},[157,158,159,160,20,161,162,163,164,139,165,166,167,145,168,131],"slop","stop slop","stop","drafting","drafts","https","patterns","prose","remove","editing","text","content",{"skillNo":170,"name":171,"nameZh":6,"description":172,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":173,"sourceHtmlUrl":174,"categories":175,"keywords":176,"publishedAt":39},"01M6M41X0B7EBX42TK435DMGGR","shuorenhua","检查和清理中英文文本里的 AI 套路，适用于“去 AI 味”“说人话”“自然一点”“别像模板”“先标问题”这类改写和审稿需求；按场景控制力度，同时保留事实、术语、语域和责任主体。","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fshuorenhua\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fshuorenhua\u002FSKILL.md",[],[171,177],"文本",{"skillNo":179,"name":180,"nameZh":6,"description":181,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":182,"sourceHtmlUrl":183,"categories":184,"keywords":185,"publishedAt":39},"01M6M3ZD12WBEW5KXKDM4N6DFG","naming","Name products, SaaS, brands, open source projects, bots, and apps. Use when the user needs to name something, find a brand name, or pick a product name. Metaphor-driven process that produces memorable, meaningful names and avoids AI slop.","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fnaming-main\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fnaming-main\u002FSKILL.md",[],[180,186,187,188,189,190,139,191,107,144,192,193,194,195,109,196,197,198,199,20,200,201,157],"repo","websearch","product name","apps","bots","view","brand","pick","gh","projects","open","saas","open source","product","describe","ai slop",{"skillNo":203,"name":204,"nameZh":6,"description":205,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":206,"sourceHtmlUrl":207,"categories":208,"keywords":210,"publishedAt":39},"01M6M3YVXAFAAWJ7F6JMPFREYE","humanizer-zh","去除文本中的 AI 生成痕迹。适用于编辑或审阅文本，使其听起来更自然、更像人类书写。\n基于维基百科的\"AI 写作特征\"综合指南。检测并修复以下模式：夸大的象征意义、\n宣传性语言、以 -ing 结尾的肤浅分析、模糊的归因、破折号过度使用、三段式法则、\nAI 词汇、否定式排比、过多的连接性短语。","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fhumanizer-zh\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fhumanizer-zh\u002FSKILL.md",[209],{"categoryNo":118,"categoryName":119,"taxonomyType":120,"taxonomyTypeDesc":121},[211,212,213,107,106,105,131,102,177,214,157,159],"humanizer","zh","humanizer zh","翻译"]