[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6QG2Y41EV6HP2Z88VVF8GE6":3,"skill-same:01M6QG2Y41EV6HP2Z88VVF8GE6":83},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":50,"commitSha":51,"skillDirectory":52,"forksCount":14,"repositoryHtmlUrl":53,"repositoryHomepageUrl":54,"ownerHtmlUrl":55,"ownerBlogUrl":56,"ownerDescription":57,"updatedAt":58,"install":59,"resources":72},"01M6QG2Y41EV6HP2Z88VVF8GE6","speculative-decoding",null,"Accelerate LLM inference using speculative decoding, Medusa multiple heads, and lookahead decoding techniques. Use when optimizing inference speed (1.5-3.6× speedup), reducing latency for real-time applications, or deploying models with limited compute. Covers draft models, tree-based attention, Jacobi iteration, parallel token generation, and production deployment strategies.","Tashima-Tarsh\u002FDisha6.6","Tashima-Tarsh","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F269906498?v=4","01M6PBH98X03XERH0HFT4PFB1P","NITISH KUMAR","01M6PBH9DA0M6KM3Z12P848F0K",0,"MIT","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fkal-speculative-decoding\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"speculative decoding","iteration","tree","Latency Reduction","Parallel Generation","Tree Attention","Draft Models","Lookahead Decoding","medusa","Emerging Techniques","Accelerate","multimodal","token","attention","strategies","time","deployment strategies","parallel","Transformers","Fast Inference","2026-11-10 10:12:30",false,"---\nname: speculative-decoding\ndescription: Accelerate LLM inference using speculative decoding, Medusa multiple heads, and lookahead decoding techniques. Use when optimizing inference speed (1.5-3.6× speedup), reducing latency for real-time applications, or deploying models with limited compute. Covers draft models, tree-based attention, Jacobi iteration, parallel token generation, and production deployment strategies.\nlicense: MIT\nmetadata:\n  version: 1.0.0\n  category: multimodal-and-emerging\n  maintainer: Kalaris Labs\n  tags: Emerging Techniques, Speculative Decoding, Medusa, Lookahead Decoding, Fast Inference, Draft Models, Tree Attention, Parallel Generation, Latency Reduction, Inference Optimization\n  dependencies: transformers, torch\n---\n\n# Speculative Decoding: Accelerating LLM Inference\n\n## When to Use This Skill\n\nUse Speculative Decoding when you need to:\n- **Speed up inference** by 1.5-3.6× without quality loss\n- **Reduce latency** for real-time applications (chatbots, code generation)\n- **Optimize throughput** for high-volume serving\n- **Deploy efficiently** on limited hardware\n- **Generate faster** without changing model architecture\n\n**Key Techniques**: Draft model speculative decoding, Medusa (multiple heads), Lookahead Decoding (Jacobi iteration)\n\n**Papers**: Medusa (arXiv 2401.10774), Lookahead Decoding (ICML 2024), Speculative Decoding Survey (ACL 2024)\n\n## Installation\n\n```bash\n# Standard speculative decoding (transformers)\npip install transformers accelerate\n\n# Medusa (multiple decoding heads)\ngit clone https:\u002F\u002Fgithub.com\u002FFasterDecoding\u002FMedusa\ncd Medusa\npip install -e .\n\n# Lookahead Decoding\ngit clone https:\u002F\u002Fgithub.com\u002Fhao-ai-lab\u002FLookaheadDecoding\ncd LookaheadDecoding\npip install -e .\n\n# Optional: vLLM with speculative decoding\npip install vllm\n```\n\n## Quick Start\n\n### Basic Speculative Decoding (Draft Model)\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load target model (large, slow)\ntarget_model = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-70b-hf\",\n    device_map=\"auto\",\n    torch_dtype=torch.float16\n)\n\n# Load draft model (small, fast)\ndraft_model = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    device_map=\"auto\",\n    torch_dtype=torch.float16\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-2-70b-hf\")\n\n# Generate with speculative decoding\nprompt = \"Explain quantum computing in simple terms:\"\ninputs = tokenizer(prompt, return_tensors=\"pt\").to(\"cuda\")\n\n# Transformers 4.36+ supports assisted generation\noutputs = target_model.generate(\n    **inputs,\n    assistant_model=draft_model,  # Enable speculative decoding\n    max_new_tokens=256,\n    do_sample=True,\n    temperature=0.7,\n)\n\nresponse = tokenizer.decode(outputs[0], skip_special_tokens=True)\nprint(response)\n```\n\n### Medusa (Multiple Decoding Heads)\n\n```python\nfrom medusa.model.medusa_model import MedusaModel\n\n# Load Medusa-enhanced model\nmodel = MedusaModel.from_pretrained(\n    \"FasterDecoding\u002Fmedusa-vicuna-7b-v1.3\",  # Pre-trained with Medusa heads\n    torch_dtype=torch.float16,\n    device_map=\"auto\"\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"FasterDecoding\u002Fmedusa-vicuna-7b-v1.3\")\n\n# Generate with Medusa (2-3× speedup)\nprompt = \"Write a Python function to calculate fibonacci numbers:\"\ninputs = tokenizer(prompt, return_tensors=\"pt\").to(\"cuda\")\n\noutputs = model.medusa_generate(\n    **inputs,\n    max_new_tokens=256,\n    temperature=0.7,\n    posterior_threshold=0.09,  # Acceptance threshold\n    posterior_alpha=0.3,       # Tree construction parameter\n)\n\nresponse = tokenizer.decode(outputs[0], skip_special_tokens=True)\n```\n\n### Lookahead Decoding (Jacobi Iteration)\n\n```python\nfrom lookahead.lookahead_decoding import LookaheadDecoding\n\n# Load model\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    torch_dtype=torch.float16,\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-2-7b-hf\")\n\n# Initialize lookahead decoding\nlookahead = LookaheadDecoding(\n    model=model,\n    tokenizer=tokenizer,\n    window_size=15,    # Lookahead window (W)\n    ngram_size=5,      # N-gram size (N)\n    guess_size=5       # Number of parallel guesses\n)\n\n# Generate (1.5-2.3× speedup)\nprompt = \"Implement quicksort in Python:\"\noutput = lookahead.generate(prompt, max_new_tokens=256)\nprint(output)\n```\n\n## Core Concepts\n\n### 1. Speculative Decoding (Draft Model)\n\n**Idea**: Use small draft model to generate candidates, large target model to verify in parallel.\n\n**Algorithm**:\n1. Draft model generates K tokens speculatively\n2. Target model evaluates all K tokens in parallel (single forward pass)\n3. Accept tokens where draft and target agree\n4. Reject first disagreement, continue from there\n\n```python\ndef speculative_decode(target_model, draft_model, prompt, K=4):\n    \"\"\"Speculative decoding algorithm.\"\"\"\n    # 1. Generate K draft tokens\n    draft_tokens = draft_model.generate(prompt, max_new_tokens=K)\n\n    # 2. Target model evaluates all K tokens in one forward pass\n    target_logits = target_model(draft_tokens)  # Parallel!\n\n    # 3. Accept\u002Freject based on probability match\n    accepted = []\n    for i in range(K):\n        p_draft = softmax(draft_model.logits[i])\n        p_target = softmax(target_logits[i])\n\n        # Acceptance probability\n        if random.random() \u003C min(1, p_target[draft_tokens[i]] \u002F p_draft[draft_tokens[i]]):\n            accepted.append(draft_tokens[i])\n        else:\n            break  # Reject, resample from target\n\n    return accepted\n```\n\n**Performance**:\n- Speedup: 1.5-2× with good draft model\n- Zero quality loss (mathematically equivalent to target model)\n- Best when draft model is 5-10× smaller than target\n\n### 2. Medusa (Multiple Decoding Heads)\n\n**Source**: arXiv 2401.10774 (2024)\n\n**Innovation**: Add multiple prediction heads to existing model, predict future tokens without separate draft model.\n\n**Architecture**:\n```\nInput → Base LLM (frozen) → Hidden State\n                                ├→ Head 1 (predicts token t+1)\n                                ├→ Head 2 (predicts token t+2)\n                                ├→ Head 3 (predicts token t+3)\n                                └→ Head 4 (predicts token t+4)\n```\n\n**Training**:\n- **Medusa-1**: Freeze base LLM, train only heads\n  - 2.2× speedup, lossless\n- **Medusa-2**: Fine-tune base LLM + heads together\n  - 2.3-3.6× speedup, better quality\n\n**Tree-based Attention**:\n```python\n# Medusa constructs tree of candidates\n# Example: Predict 2 steps ahead with top-2 per step\n\n#         Root\n#        \u002F    \\\n#      T1a    T1b  (Step 1: 2 candidates)\n#     \u002F  \\    \u002F \\\n#  T2a  T2b T2c T2d  (Step 2: 4 candidates total)\n\n# Single forward pass evaluates entire tree!\n```\n\n**Advantages**:\n- No separate draft model needed\n- Minimal training (only heads)\n- Compatible with any LLM\n\n### 3. Lookahead Decoding (Jacobi Iteration)\n\n**Source**: ICML 2024\n\n**Core idea**: Reformulate autoregressive decoding as solving system of equations, solve in parallel using Jacobi iteration.\n\n**Mathematical formulation**:\n```\nTraditional:  y_t = f(x, y_1, ..., y_{t-1})  (sequential)\nJacobi:       y_t^{(k+1)} = f(x, y_1^{(k)}, ..., y_{t-1}^{(k)})  (parallel)\n```\n\n**Two branches**:\n\n1. **Lookahead Branch**: Generate n-grams in parallel\n   - Window size W: How many steps to look ahead\n   - N-gram size N: How many past tokens to use\n\n2. **Verification Branch**: Verify promising n-grams\n   - Match n-grams with generated tokens\n   - Accept if first token matches\n\n```python\nclass LookaheadDecoding:\n    def __init__(self, model, window_size=15, ngram_size=5):\n        self.model = model\n        self.W = window_size  # Lookahead window\n        self.N = ngram_size   # N-gram size\n\n    def generate_step(self, tokens):\n        # Lookahead branch: Generate W × N candidates\n        candidates = {}\n        for w in range(1, self.W + 1):\n            for n in range(1, self.N + 1):\n                # Generate n-gram starting at position w\n                ngram = self.generate_ngram(tokens, start=w, length=n)\n                candidates[(w, n)] = ngram\n\n        # Verification branch: Find matching n-grams\n        verified = []\n        for ngram in candidates.values():\n            if ngram[0] == tokens[-1]:  # First token matches last input\n                if self.verify(tokens, ngram):\n                    verified.append(ngram)\n\n        # Accept longest verified n-gram\n        return max(verified, key=len) if verified else [self.model.generate_next(tokens)]\n```\n\n**Performance**:\n- Speedup: 1.5-2.3× (up to 3.6× for code generation)\n- No draft model or training needed\n- Works out-of-the-box with any model\n\n## Method Comparison\n\n| Method | Speedup | Training Needed | Draft Model | Quality Loss |\n|--------|---------|-----------------|-------------|--------------|\n| **Draft Model Speculative** | 1.5-2× | No | Yes (external) | None |\n| **Medusa** | 2-3.6× | Minimal (heads only) | No (built-in heads) | None |\n| **Lookahead** | 1.5-2.3× | None | No | None |\n| **Naive Batching** | 1.2-1.5× | No | No | None |\n\n## Advanced Patterns\n\n### Training Medusa Heads\n\n```python\nfrom medusa.model.medusa_model import MedusaModel\nfrom medusa.model.kv_cache import initialize_past_key_values\nimport torch.nn as nn\n\n# 1. Load base model\nbase_model = AutoModelForCausalLM.from_pretrained(\n    \"lmsys\u002Fvicuna-7b-v1.3\",\n    torch_dtype=torch.float16\n)\n\n# 2. Add Medusa heads\nnum_heads = 4\nmedusa_heads = nn.ModuleList([\n    nn.Linear(base_model.config.hidden_size, base_model.config.vocab_size, bias=False)\n    for _ in range(num_heads)\n])\n\n# 3. Training loop (freeze base model for Medusa-1)\nfor param in base_model.parameters():\n    param.requires_grad = False  # Freeze base\n\noptimizer = torch.optim.Adam(medusa_heads.parameters(), lr=1e-3)\n\nfor batch in dataloader:\n    # Forward pass\n    hidden_states = base_model(**batch, output_hidden_states=True).hidden_states[-1]\n\n    # Predict future tokens with each head\n    loss = 0\n    for i, head in enumerate(medusa_heads):\n        logits = head(hidden_states)\n        # Target: tokens shifted by (i+1) positions\n        target = batch['input_ids'][:, i+1:]\n        loss += F.cross_entropy(logits[:, :-i-1], target)\n\n    # Backward\n    optimizer.zero_grad()\n    loss.backward()\n    optimizer.step()\n```\n\n### Hybrid: Speculative + Medusa\n\n```python\n# Use Medusa as draft model for speculative decoding\ndraft_medusa = MedusaModel.from_pretrained(\"medusa-vicuna-7b\")\ntarget_model = AutoModelForCausalLM.from_pretrained(\"vicuna-33b\")\n\n# Draft generates multiple candidates with Medusa\ndraft_tokens = draft_medusa.medusa_generate(prompt, max_new_tokens=5)\n\n# Target verifies in single forward pass\noutputs = target_model.generate(\n    prompt,\n    assistant_model=draft_medusa,  # Use Medusa as draft\n    max_new_tokens=256\n)\n\n# Combines benefits: Medusa speed + large model quality\n```\n\n### Optimal Draft Model Selection\n\n```python\ndef select_draft_model(target_model_size, target):\n    \"\"\"Select optimal draft model for speculative decoding.\"\"\"\n    # Rule: Draft should be 5-10× smaller\n    if target_model_size == \"70B\":\n        return \"7B\"  # 10× smaller\n    elif target_model_size == \"33B\":\n        return \"7B\"  # 5× smaller\n    elif target_model_size == \"13B\":\n        return \"1B\"  # 13× smaller\n    else:\n        return None  # Target too small, use Medusa\u002FLookahead instead\n\n# Example\ndraft = select_draft_model(\"70B\", target_model)\n# Returns \"7B\" → Use Llama-2-7b as draft for Llama-2-70b\n```\n\n## Best Practices\n\n### 1. Choose the Right Method\n\n```python\n# New deployment → Medusa (best overall speedup, no draft model)\nif deploying_new_model:\n    use_method = \"Medusa\"\n\n# Existing deployment with small model available → Draft speculative\nelif have_small_version_of_model:\n    use_method = \"Draft Model Speculative\"\n\n# Want zero training\u002Fsetup → Lookahead\nelif want_plug_and_play:\n    use_method = \"Lookahead Decoding\"\n```\n\n### 2. Hyperparameter Tuning\n\n**Draft Model Speculative**:\n```python\n# K = number of speculative tokens\nK = 4  # Good default\nK = 2  # Conservative (higher acceptance)\nK = 8  # Aggressive (lower acceptance, but more when accepted)\n\n# Rule: Larger K → more speedup IF draft model is good\n```\n\n**Medusa**:\n```python\n# Posterior threshold (acceptance confidence)\nposterior_threshold = 0.09  # Standard (from paper)\nposterior_threshold = 0.05  # More conservative (slower, higher quality)\nposterior_threshold = 0.15  # More aggressive (faster, may degrade quality)\n\n# Tree depth (how many steps ahead)\nmedusa_choices = [[0], [0, 0], [0, 1], [0, 0, 0]]  # Depth 3 (standard)\n```\n\n**Lookahead**:\n```python\n# Window size W (lookahead distance)\n# N-gram size N (context for generation)\n\n# 7B model (more resources)\nW, N = 15, 5\n\n# 13B model (moderate)\nW, N = 10, 5\n\n# 33B+ model (limited resources)\nW, N = 7, 5\n```\n\n### 3. Production Deployment\n\n```python\n# vLLM with speculative decoding\nfrom vllm import LLM, SamplingParams\n\n# Initialize with draft model\nllm = LLM(\n    model=\"meta-llama\u002FLlama-2-70b-hf\",\n    speculative_model=\"meta-llama\u002FLlama-2-7b-hf\",  # Draft model\n    num_speculative_tokens=5,\n    use_v2_block_manager=True,\n)\n\n# Generate\nprompts = [\"Tell me about AI:\", \"Explain quantum physics:\"]\nsampling_params = SamplingParams(temperature=0.7, max_tokens=256)\n\noutputs = llm.generate(prompts, sampling_params)\nfor output in outputs:\n    print(output.outputs[0].text)\n```\n\n## Resources\n\n- **Medusa Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2401.10774\n- **Medusa GitHub**: https:\u002F\u002Fgithub.com\u002FFasterDecoding\u002FMedusa\n- **Lookahead Decoding (ICML 2024)**: https:\u002F\u002Flmsys.org\u002Fblog\u002F2023-11-21-lookahead-decoding\u002F\n- **Lookahead GitHub**: https:\u002F\u002Fgithub.com\u002Fhao-ai-lab\u002FLookaheadDecoding\n- **Speculative Decoding Survey (ACL 2024)**: https:\u002F\u002Faclanthology.org\u002F2024.findings-acl.456.pdf\n- **Comprehensive Survey**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2401.07851\n\n## See Also\n\n- `references\u002Fdraft_model.md` - Draft model selection and training\n- `references\u002Fmedusa.md` - Medusa architecture and training\n- `references\u002Flookahead.md` - Lookahead decoding implementation details\n","# Speculative Decoding: Accelerating LLM Inference\n\n## When to Use This Skill\n\nUse Speculative Decoding when you need to:\n- **Speed up inference** by 1.5-3.6× without quality loss\n- **Reduce latency** for real-time applications (chatbots, code generation)\n- **Optimize throughput** for high-volume serving\n- **Deploy efficiently** on limited hardware\n- **Generate faster** without changing model architecture\n\n**Key Techniques**: Draft model speculative decoding, Medusa (multiple heads), Lookahead Decoding (Jacobi iteration)\n\n**Papers**: Medusa (arXiv 2401.10774), Lookahead Decoding (ICML 2024), Speculative Decoding Survey (ACL 2024)\n\n## Installation\n\n```bash\n# Standard speculative decoding (transformers)\npip install transformers accelerate\n\n# Medusa (multiple decoding heads)\ngit clone https:\u002F\u002Fgithub.com\u002FFasterDecoding\u002FMedusa\ncd Medusa\npip install -e .\n\n# Lookahead Decoding\ngit clone https:\u002F\u002Fgithub.com\u002Fhao-ai-lab\u002FLookaheadDecoding\ncd LookaheadDecoding\npip install -e .\n\n# Optional: vLLM with speculative decoding\npip install vllm\n```\n\n## Quick Start\n\n### Basic Speculative Decoding (Draft Model)\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load target model (large, slow)\ntarget_model = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-70b-hf\",\n    device_map=\"auto\",\n    torch_dtype=torch.float16\n)\n\n# Load draft model (small, fast)\ndraft_model = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    device_map=\"auto\",\n    torch_dtype=torch.float16\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-2-70b-hf\")\n\n# Generate with speculative decoding\nprompt = \"Explain quantum computing in simple terms:\"\ninputs = tokenizer(prompt, return_tensors=\"pt\").to(\"cuda\")\n\n# Transformers 4.36+ supports assisted generation\noutputs = target_model.generate(\n    **inputs,\n    assistant_model=draft_model,  # Enable speculative decoding\n    max_new_tokens=256,\n    do_sample=True,\n    temperature=0.7,\n)\n\nresponse = tokenizer.decode(outputs[0], skip_special_tokens=True)\nprint(response)\n```\n\n### Medusa (Multiple Decoding Heads)\n\n```python\nfrom medusa.model.medusa_model import MedusaModel\n\n# Load Medusa-enhanced model\nmodel = MedusaModel.from_pretrained(\n    \"FasterDecoding\u002Fmedusa-vicuna-7b-v1.3\",  # Pre-trained with Medusa heads\n    torch_dtype=torch.float16,\n    device_map=\"auto\"\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"FasterDecoding\u002Fmedusa-vicuna-7b-v1.3\")\n\n# Generate with Medusa (2-3× speedup)\nprompt = \"Write a Python function to calculate fibonacci numbers:\"\ninputs = tokenizer(prompt, return_tensors=\"pt\").to(\"cuda\")\n\noutputs = model.medusa_generate(\n    **inputs,\n    max_new_tokens=256,\n    temperature=0.7,\n    posterior_threshold=0.09,  # Acceptance threshold\n    posterior_alpha=0.3,       # Tree construction parameter\n)\n\nresponse = tokenizer.decode(outputs[0], skip_special_tokens=True)\n```\n\n### Lookahead Decoding (Jacobi Iteration)\n\n```python\nfrom lookahead.lookahead_decoding import LookaheadDecoding\n\n# Load model\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    torch_dtype=torch.float16,\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-2-7b-hf\")\n\n# Initialize lookahead decoding\nlookahead = LookaheadDecoding(\n    model=model,\n    tokenizer=tokenizer,\n    window_size=15,    # Lookahead window (W)\n    ngram_size=5,      # N-gram size (N)\n    guess_size=5       # Number of parallel guesses\n)\n\n# Generate (1.5-2.3× speedup)\nprompt = \"Implement quicksort in Python:\"\noutput = lookahead.generate(prompt, max_new_tokens=256)\nprint(output)\n```\n\n## Core Concepts\n\n### 1. Speculative Decoding (Draft Model)\n\n**Idea**: Use small draft model to generate candidates, large target model to verify in parallel.\n\n**Algorithm**:\n1. Draft model generates K tokens speculatively\n2. Target model evaluates all K tokens in parallel (single forward pass)\n3. Accept tokens where draft and target agree\n4. Reject first disagreement, continue from there\n\n```python\ndef speculative_decode(target_model, draft_model, prompt, K=4):\n    \"\"\"Speculative decoding algorithm.\"\"\"\n    # 1. Generate K draft tokens\n    draft_tokens = draft_model.generate(prompt, max_new_tokens=K)\n\n    # 2. Target model evaluates all K tokens in one forward pass\n    target_logits = target_model(draft_tokens)  # Parallel!\n\n    # 3. Accept\u002Freject based on probability match\n    accepted = []\n    for i in range(K):\n        p_draft = softmax(draft_model.logits[i])\n        p_target = softmax(target_logits[i])\n\n        # Acceptance probability\n        if random.random() \u003C min(1, p_target[draft_tokens[i]] \u002F p_draft[draft_tokens[i]]):\n            accepted.append(draft_tokens[i])\n        else:\n            break  # Reject, resample from target\n\n    return accepted\n```\n\n**Performance**:\n- Speedup: 1.5-2× with good draft model\n- Zero quality loss (mathematically equivalent to target model)\n- Best when draft model is 5-10× smaller than target\n\n### 2. Medusa (Multiple Decoding Heads)\n\n**Source**: arXiv 2401.10774 (2024)\n\n**Innovation**: Add multiple prediction heads to existing model, predict future tokens without separate draft model.\n\n**Architecture**:\n```\nInput → Base LLM (frozen) → Hidden State\n                                ├→ Head 1 (predicts token t+1)\n                                ├→ Head 2 (predicts token t+2)\n                                ├→ Head 3 (predicts token t+3)\n                                └→ Head 4 (predicts token t+4)\n```\n\n**Training**:\n- **Medusa-1**: Freeze base LLM, train only heads\n  - 2.2× speedup, lossless\n- **Medusa-2**: Fine-tune base LLM + heads together\n  - 2.3-3.6× speedup, better quality\n\n**Tree-based Attention**:\n```python\n# Medusa constructs tree of candidates\n# Example: Predict 2 steps ahead with top-2 per step\n\n#         Root\n#        \u002F    \\\n#      T1a    T1b  (Step 1: 2 candidates)\n#     \u002F  \\    \u002F \\\n#  T2a  T2b T2c T2d  (Step 2: 4 candidates total)\n\n# Single forward pass evaluates entire tree!\n```\n\n**Advantages**:\n- No separate draft model needed\n- Minimal training (only heads)\n- Compatible with any LLM\n\n### 3. Lookahead Decoding (Jacobi Iteration)\n\n**Source**: ICML 2024\n\n**Core idea**: Reformulate autoregressive decoding as solving system of equations, solve in parallel using Jacobi iteration.\n\n**Mathematical formulation**:\n```\nTraditional:  y_t = f(x, y_1, ..., y_{t-1})  (sequential)\nJacobi:       y_t^{(k+1)} = f(x, y_1^{(k)}, ..., y_{t-1}^{(k)})  (parallel)\n```\n\n**Two branches**:\n\n1. **Lookahead Branch**: Generate n-grams in parallel\n   - Window size W: How many steps to look ahead\n   - N-gram size N: How many past tokens to use\n\n2. **Verification Branch**: Verify promising n-grams\n   - Match n-grams with generated tokens\n   - Accept if first token matches\n\n```python\nclass LookaheadDecoding:\n    def __init__(self, model, window_size=15, ngram_size=5):\n        self.model = model\n        self.W = window_size  # Lookahead window\n        self.N = ngram_size   # N-gram size\n\n    def generate_step(self, tokens):\n        # Lookahead branch: Generate W × N candidates\n        candidates = {}\n        for w in range(1, self.W + 1):\n            for n in range(1, self.N + 1):\n                # Generate n-gram starting at position w\n                ngram = self.generate_ngram(tokens, start=w, length=n)\n                candidates[(w, n)] = ngram\n\n        # Verification branch: Find matching n-grams\n        verified = []\n        for ngram in candidates.values():\n            if ngram[0] == tokens[-1]:  # First token matches last input\n                if self.verify(tokens, ngram):\n                    verified.append(ngram)\n\n        # Accept longest verified n-gram\n        return max(verified, key=len) if verified else [self.model.generate_next(tokens)]\n```\n\n**Performance**:\n- Speedup: 1.5-2.3× (up to 3.6× for code generation)\n- No draft model or training needed\n- Works out-of-the-box with any model\n\n## Method Comparison\n\n| Method | Speedup | Training Needed | Draft Model | Quality Loss |\n|--------|---------|-----------------|-------------|--------------|\n| **Draft Model Speculative** | 1.5-2× | No | Yes (external) | None |\n| **Medusa** | 2-3.6× | Minimal (heads only) | No (built-in heads) | None |\n| **Lookahead** | 1.5-2.3× | None | No | None |\n| **Naive Batching** | 1.2-1.5× | No | No | None |\n\n## Advanced Patterns\n\n### Training Medusa Heads\n\n```python\nfrom medusa.model.medusa_model import MedusaModel\nfrom medusa.model.kv_cache import initialize_past_key_values\nimport torch.nn as nn\n\n# 1. Load base model\nbase_model = AutoModelForCausalLM.from_pretrained(\n    \"lmsys\u002Fvicuna-7b-v1.3\",\n    torch_dtype=torch.float16\n)\n\n# 2. Add Medusa heads\nnum_heads = 4\nmedusa_heads = nn.ModuleList([\n    nn.Linear(base_model.config.hidden_size, base_model.config.vocab_size, bias=False)\n    for _ in range(num_heads)\n])\n\n# 3. Training loop (freeze base model for Medusa-1)\nfor param in base_model.parameters():\n    param.requires_grad = False  # Freeze base\n\noptimizer = torch.optim.Adam(medusa_heads.parameters(), lr=1e-3)\n\nfor batch in dataloader:\n    # Forward pass\n    hidden_states = base_model(**batch, output_hidden_states=True).hidden_states[-1]\n\n    # Predict future tokens with each head\n    loss = 0\n    for i, head in enumerate(medusa_heads):\n        logits = head(hidden_states)\n        # Target: tokens shifted by (i+1) positions\n        target = batch['input_ids'][:, i+1:]\n        loss += F.cross_entropy(logits[:, :-i-1], target)\n\n    # Backward\n    optimizer.zero_grad()\n    loss.backward()\n    optimizer.step()\n```\n\n### Hybrid: Speculative + Medusa\n\n```python\n# Use Medusa as draft model for speculative decoding\ndraft_medusa = MedusaModel.from_pretrained(\"medusa-vicuna-7b\")\ntarget_model = AutoModelForCausalLM.from_pretrained(\"vicuna-33b\")\n\n# Draft generates multiple candidates with Medusa\ndraft_tokens = draft_medusa.medusa_generate(prompt, max_new_tokens=5)\n\n# Target verifies in single forward pass\noutputs = target_model.generate(\n    prompt,\n    assistant_model=draft_medusa,  # Use Medusa as draft\n    max_new_tokens=256\n)\n\n# Combines benefits: Medusa speed + large model quality\n```\n\n### Optimal Draft Model Selection\n\n```python\ndef select_draft_model(target_model_size, target):\n    \"\"\"Select optimal draft model for speculative decoding.\"\"\"\n    # Rule: Draft should be 5-10× smaller\n    if target_model_size == \"70B\":\n        return \"7B\"  # 10× smaller\n    elif target_model_size == \"33B\":\n        return \"7B\"  # 5× smaller\n    elif target_model_size == \"13B\":\n        return \"1B\"  # 13× smaller\n    else:\n        return None  # Target too small, use Medusa\u002FLookahead instead\n\n# Example\ndraft = select_draft_model(\"70B\", target_model)\n# Returns \"7B\" → Use Llama-2-7b as draft for Llama-2-70b\n```\n\n## Best Practices\n\n### 1. Choose the Right Method\n\n```python\n# New deployment → Medusa (best overall speedup, no draft model)\nif deploying_new_model:\n    use_method = \"Medusa\"\n\n# Existing deployment with small model available → Draft speculative\nelif have_small_version_of_model:\n    use_method = \"Draft Model Speculative\"\n\n# Want zero training\u002Fsetup → Lookahead\nelif want_plug_and_play:\n    use_method = \"Lookahead Decoding\"\n```\n\n### 2. Hyperparameter Tuning\n\n**Draft Model Speculative**:\n```python\n# K = number of speculative tokens\nK = 4  # Good default\nK = 2  # Conservative (higher acceptance)\nK = 8  # Aggressive (lower acceptance, but more when accepted)\n\n# Rule: Larger K → more speedup IF draft model is good\n```\n\n**Medusa**:\n```python\n# Posterior threshold (acceptance confidence)\nposterior_threshold = 0.09  # Standard (from paper)\nposterior_threshold = 0.05  # More conservative (slower, higher quality)\nposterior_threshold = 0.15  # More aggressive (faster, may degrade quality)\n\n# Tree depth (how many steps ahead)\nmedusa_choices = [[0], [0, 0], [0, 1], [0, 0, 0]]  # Depth 3 (standard)\n```\n\n**Lookahead**:\n```python\n# Window size W (lookahead distance)\n# N-gram size N (context for generation)\n\n# 7B model (more resources)\nW, N = 15, 5\n\n# 13B model (moderate)\nW, N = 10, 5\n\n# 33B+ model (limited resources)\nW, N = 7, 5\n```\n\n### 3. Production Deployment\n\n```python\n# vLLM with speculative decoding\nfrom vllm import LLM, SamplingParams\n\n# Initialize with draft model\nllm = LLM(\n    model=\"meta-llama\u002FLlama-2-70b-hf\",\n    speculative_model=\"meta-llama\u002FLlama-2-7b-hf\",  # Draft model\n    num_speculative_tokens=5,\n    use_v2_block_manager=True,\n)\n\n# Generate\nprompts = [\"Tell me about AI:\", \"Explain quantum physics:\"]\nsampling_params = SamplingParams(temperature=0.7, max_tokens=256)\n\noutputs = llm.generate(prompts, sampling_params)\nfor output in outputs:\n    print(output.outputs[0].text)\n```\n\n## Resources\n\n- **Medusa Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2401.10774\n- **Medusa GitHub**: https:\u002F\u002Fgithub.com\u002FFasterDecoding\u002FMedusa\n- **Lookahead Decoding (ICML 2024)**: https:\u002F\u002Flmsys.org\u002Fblog\u002F2023-11-21-lookahead-decoding\u002F\n- **Lookahead GitHub**: https:\u002F\u002Fgithub.com\u002Fhao-ai-lab\u002FLookaheadDecoding\n- **Speculative Decoding Survey (ACL 2024)**: https:\u002F\u002Faclanthology.org\u002F2024.findings-acl.456.pdf\n- **Comprehensive Survey**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2401.07851\n\n## See Also\n\n- `references\u002Fdraft_model.md` - Draft model selection and training\n- `references\u002Fmedusa.md` - Medusa architecture and training\n- `references\u002Flookahead.md` - Lookahead decoding implementation details\n",{"name":5,"license":15,"metadata":44,"description":7},{"tags":45,"version":46,"category":47,"maintainer":48,"dependencies":49},"Emerging Techniques, Speculative Decoding, Medusa, Lookahead Decoding, Fast Inference, Draft Models, Tree Attention, Parallel Generation, Latency Reduction, Inference Optimization","1.0.0","multimodal-and-emerging","Kalaris Labs","transformers, torch","main","0da4485785ee599388900be50db1635489d53056",".agents\u002Fskills\u002Fkal-speculative-decoding","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6","https:\u002F\u002Fdisha6.6.thenitishkr.in\u002F","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh","www.thenitishkr","thenitishkr | AI • Cyber-security • Research\r\n","2026-11-08 03:33:25",{"source":8,"skillName":5,"commands":60,"prompt":70,"zipAvailable":71},[61,64,67],{"tool":62,"command":63},"npx","npx skills add Tashima-Tarsh\u002FDisha6.6 --skill speculative-decoding",{"tool":65,"command":66},"bunx","bunx skills add Tashima-Tarsh\u002FDisha6.6 --skill speculative-decoding",{"tool":68,"command":69},"pnpm","pnpm dlx skills add Tashima-Tarsh\u002FDisha6.6 --skill speculative-decoding","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fkal-speculative-decoding\u002FSKILL.md",true,[73,79],{"path":74,"resourceType":75,"resourceTypeDesc":76,"sizeBytes":77,"sourceHtmlUrl":78},"references\u002Flookahead.md","REFERENCE","参考文档",8890,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fkal-speculative-decoding\u002Freferences\u002Flookahead.md",{"path":80,"resourceType":75,"resourceTypeDesc":76,"sizeBytes":81,"sourceHtmlUrl":82},"references\u002Fmedusa.md",9851,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fkal-speculative-decoding\u002Freferences\u002Fmedusa.md",[84,116,141,167,192,216],{"skillNo":85,"name":86,"nameZh":6,"description":87,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":88,"categories":89,"keywords":95,"publishedAt":39},"01M6QRD07D4TGNDK6VJ4EHF8BV","youtube-full","Use when the user needs YouTube transcripts, video search, channel browsing, playlist extraction, or content monitoring. Trigger phrases: 'get the transcript for', 'search YouTube for', 'what are the latest videos on', 'list this playlist', 'monitor this channel', or any request involving a YouTube URL, video ID, or @handle. Do NOT use for downloading video or audio files, YouTube engagement data (likes, comments), or private\u002Fage-restricted videos.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fyoutube-full\u002FSKILL.md",[90],{"categoryNo":91,"categoryName":92,"taxonomyType":93,"taxonomyTypeDesc":94},"01JBSEEDCAT000000000000016","图像与多媒体","SCENARIO","使用场景",[96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115],"youtube full","youtube","full","transcript","search","monitoring","data","Video","Audio","list","files","do","get","age","trigger","URL","for","browsing","request","youtube transcripts",{"skillNo":117,"name":118,"nameZh":6,"description":119,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":120,"categories":121,"keywords":122,"publishedAt":39},"01M6QRCJ3112VYPA97YG7KGMC2","x-twitter-growth","X\u002FTwitter growth engine for building audience, crafting viral content, and analyzing engagement. Use when the user wants to grow on X\u002FTwitter, write tweets or threads, analyze their X profile, research competitors on X, plan a posting strategy, or optimize engagement. Complements social-content (generic multi-platform) with X-specific depth: algorithm mechanics, thread engineering, reply strategy, profile optimization, and competitive intelligence via web search.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fx-twitter-growth\u002FSKILL.md",[],[123,124,125,126,100,127,128,129,130,131,132,133,134,135,136,137,138,139,140,112],"twitter","x twitter growth","growth","research","content","engineering","plan","web search","competitors","analyze","engine","write","strategy","use","platform","grow","posting","competitive",{"skillNo":142,"name":143,"nameZh":6,"description":144,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":145,"categories":146,"keywords":150,"publishedAt":39},"01M6QRBN3EHQNJY42RERP7A149","write-a-skill","Create new agent skills with proper structure, progressive disclosure, and bundled resources. Use when user wants to create, write, build, or author a new skill.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fwrite-a-skill\u002FSKILL.md",[147],{"categoryNo":148,"categoryName":149,"taxonomyType":93,"taxonomyTypeDesc":94},"01JBSEEDCAT000000000000023","Skill与智能体开发",[151,134,152,153,154,155,156,157,136,158,159,50,160,161,162,163,164,21,165,166],"write a skill","github","create","https","productivity","matt pocock","direct","example","build","new skill","structure","new agent","resources","with","progressive disclosure","author",{"skillNo":168,"name":169,"nameZh":6,"description":170,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":171,"categories":172,"keywords":173,"publishedAt":39},"01M6QRB6G8P33PNGAF9Y69EJ9T","wrap-up","Close out a launched Claude Managed Agent — recap every primitive the founder now owns, regenerate the single-file overview page, and suggest the next 1-2 upgrades. Use when the user says \"wrap up\", \"close this out\", \"what do I own now\", \"give me the summary\", \"recap the agent\", or when the orchestrator routes phase=wrap-up. primitives_inventory.py tables everything owned (agent, environment, session, memory, outcome, deployment); overview_page.py regenerates a self-contained .\u002Fmy-agent\u002Fagent-overview.html; upgrade_suggester.py ranks the next moves from recorded deferrals plus standing hardening steps. Companion to run-without-you; the last stop before phase=done.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fwrap-up\u002FSKILL.md",[],[174,175,176,177,178,179,180,181,182,136,183,107,184,185,186,187,188,189,190,191],"wrap up","up","wrap","gemini","deployment","run","file","hardening","closeout","stop","upgrade","founder","close","companion","you","tables","directions","phase",{"skillNo":193,"name":194,"nameZh":6,"description":195,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":196,"categories":197,"keywords":199,"publishedAt":39},"01M6QRA5AYC1RBFZ39GTWGS3N8","workflow-builder","Design and write deterministic multi-agent workflow scripts (.js files in .claude\u002Fworkflows\u002F) for Claude Code's Workflow tool. Use when a user wants to build, create, author, scaffold, or run a custom Claude Code workflow, orchestrate sub-agents (fan-out, pipeline, loop, judge-panel), or automate a repeatable multi-step task across fresh-context agents.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fworkflow-builder\u002FSKILL.md",[198],{"categoryNo":148,"categoryName":149,"taxonomyType":93,"taxonomyTypeDesc":94},[200,201,202,203,204,153,179,205,206,207,208,209,134,210,106,211,212,213,214,215],"workflow","workflow builder","builder","design","workflows","code","orchestrate","pipeline","loop","multi-agent","context","across","fresh","scripts","judge","claude code workflow",{"skillNo":217,"name":218,"nameZh":6,"description":219,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":220,"categories":221,"keywords":222,"publishedAt":39},"01M6QR7QMCZESKQA0ERW1E657M","weekly-review","Use when someone wants to run a weekly review, close open loops, audit stalled projects and commitments, get their system back to trusted, restart a lapsed review habit, or says \"\u002Fcs:weekly-review\". Walks David Allen's three-phase loop — GET CLEAR, GET CURRENT, GET CREATIVE — with deterministic scripts that inventory open loops, gate the checklist with named gaps, and score commitment health 0-100.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fweekly-review\u002FSKILL.md",[],[223,224,225,226,227,228,229,230,231,208,232,233,234,235,236,237,108,238,239,191],"review","weekly review","weekly","deterministic scripts","reflect","feeds","inventory","open","team","projects","project-management","scanner","conversation","back","project","sprint","cadence"]