[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6KQHR21Q55VAPKC467XKVEB":3,"skill-same:01M6KQHR21Q55VAPKC467XKVEB":83},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":22,"publishedAt":44,"linkOnly":45,"markdownContent":46,"markdownBody":47,"frontMatter":48,"refName":57,"commitSha":58,"skillDirectory":59,"forksCount":13,"repositoryHtmlUrl":60,"repositoryHomepageUrl":6,"ownerHtmlUrl":61,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":62,"install":63,"resources":76},"01M6KQHR21Q55VAPKC467XKVEB","knowledge-distillation",null,"Compress large language models using knowledge distillation from teacher to student models. Use when deploying smaller models with retained performance, transferring GPT-4 capabilities to open-source models, or reducing inference costs. Covers temperature scaling, soft targets, reverse KLD, logit distillation, and MiniLLM training strategies.","endman100\u002Fskill-router-nexus","endman100","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F31997019?v=4","01M6KPDX17BYDWW4RRB0TNGGMX","01M6KPDX64Y2QBEWVFG5W5PWPF",0,"MIT","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Training\u002Fknowledge-distillation\u002Fknowledge-distillation\u002FSKILL.md",[17],{"categoryNo":18,"categoryName":19,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000021","资料检索与研究","SCENARIO","使用场景",[23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43],"knowledge","distillation","Knowledge Distillation","when","performance","teacher","student","Model Transfer","Logit Distillation","Temperature Scaling","Soft Targets","Reverse KLD","MiniLLM","Teacher-Student","Emerging Techniques","scaling","strategies","model","open","capabilities","training","2026-11-10 10:12:30",false,"---\nname: knowledge-distillation\ndescription: Compress large language models using knowledge distillation from teacher to student models. Use when deploying smaller models with retained performance, transferring GPT-4 capabilities to open-source models, or reducing inference costs. Covers temperature scaling, soft targets, reverse KLD, logit distillation, and MiniLLM training strategies.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Emerging Techniques, Knowledge Distillation, Model Compression, Teacher-Student, MiniLLM, Reverse KLD, Soft Targets, Temperature Scaling, Logit Distillation, Model Transfer]\ndependencies: [transformers, torch, datasets]\n---\n\n# Knowledge Distillation: Compressing LLMs\n\n## When to Use This Skill\n\nUse Knowledge Distillation when you need to:\n- **Compress models** from 70B → 7B while retaining 90%+ performance\n- **Transfer capabilities** from proprietary models (GPT-4) to open-source (LLaMA, Mistral)\n- **Reduce inference costs** by deploying smaller student models\n- **Create specialized models** by distilling domain-specific knowledge\n- **Improve small models** using synthetic data from large teachers\n\n**Key Techniques**: Temperature scaling, soft targets, reverse KLD (MiniLLM), logit distillation, response distillation\n\n**Papers**: Hinton et al. 2015 (arXiv 1503.02531), MiniLLM (arXiv 2306.08543), KD Survey (arXiv 2402.13116)\n\n## Installation\n\n```bash\n# Standard transformers\npip install transformers datasets accelerate\n\n# For training\npip install torch deepspeed wandb\n\n# Optional: MiniLLM implementation\ngit clone https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FLMOps\ncd LMOps\u002Fminillm\npip install -e .\n```\n\n## Quick Start\n\n### Basic Knowledge Distillation\n\n```python\nimport torch\nimport torch.nn.functional as F\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, Trainer, TrainingArguments\n\n# 1. Load teacher (large) and student (small) models\nteacher = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-70b-hf\",  # Large teacher\n    torch_dtype=torch.float16,\n    device_map=\"auto\"\n)\n\nstudent = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",  # Small student\n    torch_dtype=torch.float16,\n    device_map=\"cuda:0\"\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-2-70b-hf\")\n\n# 2. Define distillation loss\ndef distillation_loss(student_logits, teacher_logits, labels, temperature=2.0, alpha=0.5):\n    \"\"\"\n    Combine hard loss (cross-entropy) with soft loss (KL divergence).\n\n    Args:\n        temperature: Softens probability distributions (higher = softer)\n        alpha: Weight for distillation loss (1-alpha for hard loss)\n    \"\"\"\n    # Hard loss: Standard cross-entropy with true labels\n    hard_loss = F.cross_entropy(student_logits.view(-1, student_logits.size(-1)), labels.view(-1))\n\n    # Soft loss: KL divergence between student and teacher\n    soft_targets = F.softmax(teacher_logits \u002F temperature, dim=-1)\n    soft_student = F.log_softmax(student_logits \u002F temperature, dim=-1)\n    soft_loss = F.kl_div(soft_student, soft_targets, reduction='batchmean') * (temperature ** 2)\n\n    # Combined loss\n    return alpha * soft_loss + (1 - alpha) * hard_loss\n\n# 3. Training loop\nfor batch in dataloader:\n    # Teacher forward (no grad)\n    with torch.no_grad():\n        teacher_outputs = teacher(**batch)\n        teacher_logits = teacher_outputs.logits\n\n    # Student forward\n    student_outputs = student(**batch)\n    student_logits = student_outputs.logits\n\n    # Compute distillation loss\n    loss = distillation_loss(\n        student_logits,\n        teacher_logits,\n        batch['labels'],\n        temperature=2.0,\n        alpha=0.7  # 70% soft, 30% hard\n    )\n\n    # Backward and optimize\n    loss.backward()\n    optimizer.step()\n    optimizer.zero_grad()\n```\n\n### MiniLLM (Reverse KLD)\n\n**Source**: arXiv 2306.08543 (2024)\n\n**Innovation**: Use reverse KLD instead of forward KLD for better generative model distillation.\n\n```python\ndef reverse_kl_loss(student_logits, teacher_logits, temperature=1.0):\n    \"\"\"\n    Reverse KL divergence: KL(Teacher || Student)\n    Better for generative models than forward KL.\n    \"\"\"\n    # Teacher distribution (target)\n    p_teacher = F.softmax(teacher_logits \u002F temperature, dim=-1)\n\n    # Student distribution (model)\n    log_p_student = F.log_softmax(student_logits \u002F temperature, dim=-1)\n\n    # Reverse KL: Sum over teacher, student learns to cover teacher's modes\n    reverse_kl = -(p_teacher * log_p_student).sum(dim=-1).mean()\n\n    return reverse_kl * (temperature ** 2)\n\n# Training with MiniLLM\nfor batch in dataloader:\n    with torch.no_grad():\n        teacher_logits = teacher(**batch).logits\n\n    student_logits = student(**batch).logits\n\n    # Reverse KLD (better for generation)\n    loss = reverse_kl_loss(student_logits, teacher_logits, temperature=1.0)\n\n    loss.backward()\n    optimizer.step()\n```\n\n**Why reverse KL?**\n- **Forward KL** (standard): Student learns to match teacher's *mean*\n- **Reverse KL** (MiniLLM): Student learns to *cover* all teacher's modes\n- Better for diverse text generation\n\n### Response Distillation\n\n```python\n# Generate synthetic data from teacher, train student to imitate\n\n# 1. Generate synthetic responses from teacher\nprompts = [\"Explain AI:\", \"What is ML?\", \"Define NLP:\"]\n\nteacher_responses = []\nfor prompt in prompts:\n    inputs = tokenizer(prompt, return_tensors='pt').to(teacher.device)\n    outputs = teacher.generate(**inputs, max_new_tokens=256, do_sample=True, temperature=0.7)\n    response = tokenizer.decode(outputs[0], skip_special_tokens=True)\n    teacher_responses.append(response)\n\n# 2. Train student on teacher's responses (standard fine-tuning)\ntrain_dataset = [\n    {\"text\": f\"{prompt}\\n{response}\"}\n    for prompt, response in zip(prompts, teacher_responses)\n]\n\n# 3. Fine-tune student\ntrainer = Trainer(\n    model=student,\n    args=TrainingArguments(output_dir=\".\u002Fstudent\", num_train_epochs=3, learning_rate=2e-5),\n    train_dataset=train_dataset,\n)\ntrainer.train()\n```\n\n## Core Concepts\n\n### 1. Temperature Scaling\n\n**Purpose**: Soften probability distributions to expose teacher's uncertainty.\n\n```python\n# Low temperature (T=1): Sharp distribution\nlogits = [3.0, 2.0, 1.0]\nprobs_T1 = softmax(logits \u002F 1.0)  # [0.67, 0.24, 0.09]\n\n# High temperature (T=4): Soft distribution\nprobs_T4 = softmax(logits \u002F 4.0)  # [0.42, 0.34, 0.24]\n\n# Higher T reveals more information about relative rankings\n```\n\n**Rule**: Use T=2-5 for distillation (2 is common default).\n\n### 2. Loss Function Components\n\n```python\n# Total loss = alpha * soft_loss + (1 - alpha) * hard_loss\n\n# Soft loss: Learn from teacher's knowledge\nsoft_loss = KL(student || teacher)\n\n# Hard loss: Learn from ground truth labels\nhard_loss = CrossEntropy(student_output, true_labels)\n\n# Typical values:\nalpha = 0.5  # Balanced\nalpha = 0.7  # More emphasis on teacher\nalpha = 0.3  # More emphasis on labels\n```\n\n### 3. Forward vs Reverse KLD\n\n```python\n# Forward KL: KL(Student || Teacher)\n# - Student matches teacher's average behavior\n# - Mode-seeking: Student focuses on teacher's highest probability modes\n# - Good for classification\n\n# Reverse KL: KL(Teacher || Student)\n# - Student covers all of teacher's behaviors\n# - Mode-covering: Student learns diverse behaviors\n# - Good for generation (MiniLLM)\n```\n\n## Training Strategies\n\n### Strategy 1: Logit Distillation\n\n```python\n# Train student to match teacher's logits directly\n\ndef logit_distillation_trainer(student, teacher, dataloader, temperature=2.0):\n    optimizer = torch.optim.AdamW(student.parameters(), lr=2e-5)\n\n    for epoch in range(3):\n        for batch in dataloader:\n            # Get logits\n            with torch.no_grad():\n                teacher_logits = teacher(**batch).logits\n\n            student_logits = student(**batch).logits\n\n            # MSE on logits (alternative to KLD)\n            loss = F.mse_loss(student_logits, teacher_logits)\n\n            # Or use KLD\n            # loss = F.kl_div(\n            #     F.log_softmax(student_logits\u002Ftemperature, dim=-1),\n            #     F.softmax(teacher_logits\u002Ftemperature, dim=-1),\n            #     reduction='batchmean'\n            # ) * (temperature ** 2)\n\n            loss.backward()\n            optimizer.step()\n            optimizer.zero_grad()\n\n    return student\n```\n\n### Strategy 2: Two-Stage Distillation\n\n```python\n# Stage 1: Distill from teacher\nstudent = distill(teacher, student, epochs=5)\n\n# Stage 2: Fine-tune on task-specific data\nstudent = fine_tune(student, task_data, epochs=3)\n\n# Results in better task performance than single-stage\n```\n\n### Strategy 3: Multi-Teacher Distillation\n\n```python\n# Learn from multiple expert teachers\n\ndef multi_teacher_distillation(student, teachers, batch):\n    \"\"\"Distill from ensemble of teachers.\"\"\"\n    teacher_logits_list = []\n\n    # Get logits from all teachers\n    with torch.no_grad():\n        for teacher in teachers:\n            logits = teacher(**batch).logits\n            teacher_logits_list.append(logits)\n\n    # Average teacher predictions\n    avg_teacher_logits = torch.stack(teacher_logits_list).mean(dim=0)\n\n    # Student learns from ensemble\n    student_logits = student(**batch).logits\n    loss = F.kl_div(\n        F.log_softmax(student_logits, dim=-1),\n        F.softmax(avg_teacher_logits, dim=-1),\n        reduction='batchmean'\n    )\n\n    return loss\n```\n\n## Production Deployment\n\n### Complete Training Script\n\n```python\nfrom transformers import Trainer, TrainingArguments, DataCollatorForLanguageModeling\n\ndef train_distilled_model(\n    teacher_name=\"meta-llama\u002FLlama-2-70b-hf\",\n    student_name=\"meta-llama\u002FLlama-2-7b-hf\",\n    output_dir=\".\u002Fdistilled-llama-7b\",\n    temperature=2.0,\n    alpha=0.7,\n):\n    # Load models\n    teacher = AutoModelForCausalLM.from_pretrained(teacher_name, torch_dtype=torch.float16, device_map=\"auto\")\n    student = AutoModelForCausalLM.from_pretrained(student_name, torch_dtype=torch.float16)\n    tokenizer = AutoTokenizer.from_pretrained(teacher_name)\n\n    # Custom trainer with distillation\n    class DistillationTrainer(Trainer):\n        def compute_loss(self, model, inputs, return_outputs=False):\n            # Student forward\n            outputs_student = model(**inputs)\n            student_logits = outputs_student.logits\n\n            # Teacher forward (no grad)\n            with torch.no_grad():\n                outputs_teacher = teacher(**inputs)\n                teacher_logits = outputs_teacher.logits\n\n            # Distillation loss\n            soft_targets = F.softmax(teacher_logits \u002F temperature, dim=-1)\n            soft_student = F.log_softmax(student_logits \u002F temperature, dim=-1)\n            soft_loss = F.kl_div(soft_student, soft_targets, reduction='batchmean') * (temperature ** 2)\n\n            # Hard loss\n            hard_loss = outputs_student.loss\n\n            # Combined\n            loss = alpha * soft_loss + (1 - alpha) * hard_loss\n\n            return (loss, outputs_student) if return_outputs else loss\n\n    # Training arguments\n    training_args = TrainingArguments(\n        output_dir=output_dir,\n        num_train_epochs=3,\n        per_device_train_batch_size=4,\n        gradient_accumulation_steps=8,\n        learning_rate=2e-5,\n        warmup_steps=500,\n        logging_steps=100,\n        save_steps=1000,\n        bf16=True,\n        gradient_checkpointing=True,\n    )\n\n    # Train\n    trainer = DistillationTrainer(\n        model=student,\n        args=training_args,\n        train_dataset=train_dataset,\n        data_collator=DataCollatorForLanguageModeling(tokenizer, mlm=False),\n    )\n\n    trainer.train()\n    student.save_pretrained(output_dir)\n    tokenizer.save_pretrained(output_dir)\n\n# Usage\ntrain_distilled_model(\n    teacher_name=\"meta-llama\u002FLlama-2-70b-hf\",\n    student_name=\"meta-llama\u002FLlama-2-7b-hf\",\n    temperature=2.0,\n    alpha=0.7\n)\n```\n\n## Best Practices\n\n### 1. Hyperparameter Selection\n\n```python\n# Temperature\nT = 1.0  # Sharp (less knowledge transfer)\nT = 2.0  # Standard (good balance)\nT = 5.0  # Soft (more knowledge transfer)\n\n# Alpha (weight)\nalpha = 0.5  # Balanced\nalpha = 0.7  # Emphasize teacher knowledge\nalpha = 0.9  # Strong distillation\n\n# Rule: Higher T + higher alpha = stronger distillation\n```\n\n### 2. Model Size Ratio\n\n```python\n# Good ratios (teacher\u002Fstudent)\n70B \u002F 7B = 10×    # Excellent\n13B \u002F 1B = 13×    # Good\n7B \u002F 1B = 7×      # Acceptable\n\n# Avoid too large gap\n70B \u002F 1B = 70×    # Too large, ineffective\n```\n\n### 3. Data Quality\n\n```python\n# Best: Use teacher-generated data + real data\ntrain_data = {\n    \"teacher_generated\": 70%,  # Diverse, high-quality\n    \"real_data\": 30%            # Ground truth\n}\n\n# Avoid: Only real data (doesn't utilize teacher fully)\n```\n\n## Evaluation\n\n```python\nfrom transformers import pipeline\n\n# Compare student vs teacher\nteacher_pipe = pipeline(\"text-generation\", model=teacher)\nstudent_pipe = pipeline(\"text-generation\", model=student)\n\nprompts = [\"Explain quantum computing:\", \"What is AI?\"]\n\nfor prompt in prompts:\n    teacher_out = teacher_pipe(prompt, max_new_tokens=100)\n    student_out = student_pipe(prompt, max_new_tokens=100)\n\n    print(f\"Prompt: {prompt}\")\n    print(f\"Teacher: {teacher_out[0]['generated_text']}\")\n    print(f\"Student: {student_out[0]['generated_text']}\")\n    print(f\"Match quality: {calculate_similarity(teacher_out, student_out):.2f}\")\n```\n\n## Resources\n\n- **Hinton et al. 2015 (Foundational)**: https:\u002F\u002Farxiv.org\u002Fabs\u002F1503.02531\n- **MiniLLM (Reverse KLD)**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2306.08543\n- **KD Survey for LLMs (2024)**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2402.13116\n- **MiniLLM GitHub**: https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FLMOps\u002Ftree\u002Fmain\u002Fminillm\n\n\n","# Knowledge Distillation: Compressing LLMs\n\n## When to Use This Skill\n\nUse Knowledge Distillation when you need to:\n- **Compress models** from 70B → 7B while retaining 90%+ performance\n- **Transfer capabilities** from proprietary models (GPT-4) to open-source (LLaMA, Mistral)\n- **Reduce inference costs** by deploying smaller student models\n- **Create specialized models** by distilling domain-specific knowledge\n- **Improve small models** using synthetic data from large teachers\n\n**Key Techniques**: Temperature scaling, soft targets, reverse KLD (MiniLLM), logit distillation, response distillation\n\n**Papers**: Hinton et al. 2015 (arXiv 1503.02531), MiniLLM (arXiv 2306.08543), KD Survey (arXiv 2402.13116)\n\n## Installation\n\n```bash\n# Standard transformers\npip install transformers datasets accelerate\n\n# For training\npip install torch deepspeed wandb\n\n# Optional: MiniLLM implementation\ngit clone https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FLMOps\ncd LMOps\u002Fminillm\npip install -e .\n```\n\n## Quick Start\n\n### Basic Knowledge Distillation\n\n```python\nimport torch\nimport torch.nn.functional as F\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, Trainer, TrainingArguments\n\n# 1. Load teacher (large) and student (small) models\nteacher = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-70b-hf\",  # Large teacher\n    torch_dtype=torch.float16,\n    device_map=\"auto\"\n)\n\nstudent = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",  # Small student\n    torch_dtype=torch.float16,\n    device_map=\"cuda:0\"\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-2-70b-hf\")\n\n# 2. Define distillation loss\ndef distillation_loss(student_logits, teacher_logits, labels, temperature=2.0, alpha=0.5):\n    \"\"\"\n    Combine hard loss (cross-entropy) with soft loss (KL divergence).\n\n    Args:\n        temperature: Softens probability distributions (higher = softer)\n        alpha: Weight for distillation loss (1-alpha for hard loss)\n    \"\"\"\n    # Hard loss: Standard cross-entropy with true labels\n    hard_loss = F.cross_entropy(student_logits.view(-1, student_logits.size(-1)), labels.view(-1))\n\n    # Soft loss: KL divergence between student and teacher\n    soft_targets = F.softmax(teacher_logits \u002F temperature, dim=-1)\n    soft_student = F.log_softmax(student_logits \u002F temperature, dim=-1)\n    soft_loss = F.kl_div(soft_student, soft_targets, reduction='batchmean') * (temperature ** 2)\n\n    # Combined loss\n    return alpha * soft_loss + (1 - alpha) * hard_loss\n\n# 3. Training loop\nfor batch in dataloader:\n    # Teacher forward (no grad)\n    with torch.no_grad():\n        teacher_outputs = teacher(**batch)\n        teacher_logits = teacher_outputs.logits\n\n    # Student forward\n    student_outputs = student(**batch)\n    student_logits = student_outputs.logits\n\n    # Compute distillation loss\n    loss = distillation_loss(\n        student_logits,\n        teacher_logits,\n        batch['labels'],\n        temperature=2.0,\n        alpha=0.7  # 70% soft, 30% hard\n    )\n\n    # Backward and optimize\n    loss.backward()\n    optimizer.step()\n    optimizer.zero_grad()\n```\n\n### MiniLLM (Reverse KLD)\n\n**Source**: arXiv 2306.08543 (2024)\n\n**Innovation**: Use reverse KLD instead of forward KLD for better generative model distillation.\n\n```python\ndef reverse_kl_loss(student_logits, teacher_logits, temperature=1.0):\n    \"\"\"\n    Reverse KL divergence: KL(Teacher || Student)\n    Better for generative models than forward KL.\n    \"\"\"\n    # Teacher distribution (target)\n    p_teacher = F.softmax(teacher_logits \u002F temperature, dim=-1)\n\n    # Student distribution (model)\n    log_p_student = F.log_softmax(student_logits \u002F temperature, dim=-1)\n\n    # Reverse KL: Sum over teacher, student learns to cover teacher's modes\n    reverse_kl = -(p_teacher * log_p_student).sum(dim=-1).mean()\n\n    return reverse_kl * (temperature ** 2)\n\n# Training with MiniLLM\nfor batch in dataloader:\n    with torch.no_grad():\n        teacher_logits = teacher(**batch).logits\n\n    student_logits = student(**batch).logits\n\n    # Reverse KLD (better for generation)\n    loss = reverse_kl_loss(student_logits, teacher_logits, temperature=1.0)\n\n    loss.backward()\n    optimizer.step()\n```\n\n**Why reverse KL?**\n- **Forward KL** (standard): Student learns to match teacher's *mean*\n- **Reverse KL** (MiniLLM): Student learns to *cover* all teacher's modes\n- Better for diverse text generation\n\n### Response Distillation\n\n```python\n# Generate synthetic data from teacher, train student to imitate\n\n# 1. Generate synthetic responses from teacher\nprompts = [\"Explain AI:\", \"What is ML?\", \"Define NLP:\"]\n\nteacher_responses = []\nfor prompt in prompts:\n    inputs = tokenizer(prompt, return_tensors='pt').to(teacher.device)\n    outputs = teacher.generate(**inputs, max_new_tokens=256, do_sample=True, temperature=0.7)\n    response = tokenizer.decode(outputs[0], skip_special_tokens=True)\n    teacher_responses.append(response)\n\n# 2. Train student on teacher's responses (standard fine-tuning)\ntrain_dataset = [\n    {\"text\": f\"{prompt}\\n{response}\"}\n    for prompt, response in zip(prompts, teacher_responses)\n]\n\n# 3. Fine-tune student\ntrainer = Trainer(\n    model=student,\n    args=TrainingArguments(output_dir=\".\u002Fstudent\", num_train_epochs=3, learning_rate=2e-5),\n    train_dataset=train_dataset,\n)\ntrainer.train()\n```\n\n## Core Concepts\n\n### 1. Temperature Scaling\n\n**Purpose**: Soften probability distributions to expose teacher's uncertainty.\n\n```python\n# Low temperature (T=1): Sharp distribution\nlogits = [3.0, 2.0, 1.0]\nprobs_T1 = softmax(logits \u002F 1.0)  # [0.67, 0.24, 0.09]\n\n# High temperature (T=4): Soft distribution\nprobs_T4 = softmax(logits \u002F 4.0)  # [0.42, 0.34, 0.24]\n\n# Higher T reveals more information about relative rankings\n```\n\n**Rule**: Use T=2-5 for distillation (2 is common default).\n\n### 2. Loss Function Components\n\n```python\n# Total loss = alpha * soft_loss + (1 - alpha) * hard_loss\n\n# Soft loss: Learn from teacher's knowledge\nsoft_loss = KL(student || teacher)\n\n# Hard loss: Learn from ground truth labels\nhard_loss = CrossEntropy(student_output, true_labels)\n\n# Typical values:\nalpha = 0.5  # Balanced\nalpha = 0.7  # More emphasis on teacher\nalpha = 0.3  # More emphasis on labels\n```\n\n### 3. Forward vs Reverse KLD\n\n```python\n# Forward KL: KL(Student || Teacher)\n# - Student matches teacher's average behavior\n# - Mode-seeking: Student focuses on teacher's highest probability modes\n# - Good for classification\n\n# Reverse KL: KL(Teacher || Student)\n# - Student covers all of teacher's behaviors\n# - Mode-covering: Student learns diverse behaviors\n# - Good for generation (MiniLLM)\n```\n\n## Training Strategies\n\n### Strategy 1: Logit Distillation\n\n```python\n# Train student to match teacher's logits directly\n\ndef logit_distillation_trainer(student, teacher, dataloader, temperature=2.0):\n    optimizer = torch.optim.AdamW(student.parameters(), lr=2e-5)\n\n    for epoch in range(3):\n        for batch in dataloader:\n            # Get logits\n            with torch.no_grad():\n                teacher_logits = teacher(**batch).logits\n\n            student_logits = student(**batch).logits\n\n            # MSE on logits (alternative to KLD)\n            loss = F.mse_loss(student_logits, teacher_logits)\n\n            # Or use KLD\n            # loss = F.kl_div(\n            #     F.log_softmax(student_logits\u002Ftemperature, dim=-1),\n            #     F.softmax(teacher_logits\u002Ftemperature, dim=-1),\n            #     reduction='batchmean'\n            # ) * (temperature ** 2)\n\n            loss.backward()\n            optimizer.step()\n            optimizer.zero_grad()\n\n    return student\n```\n\n### Strategy 2: Two-Stage Distillation\n\n```python\n# Stage 1: Distill from teacher\nstudent = distill(teacher, student, epochs=5)\n\n# Stage 2: Fine-tune on task-specific data\nstudent = fine_tune(student, task_data, epochs=3)\n\n# Results in better task performance than single-stage\n```\n\n### Strategy 3: Multi-Teacher Distillation\n\n```python\n# Learn from multiple expert teachers\n\ndef multi_teacher_distillation(student, teachers, batch):\n    \"\"\"Distill from ensemble of teachers.\"\"\"\n    teacher_logits_list = []\n\n    # Get logits from all teachers\n    with torch.no_grad():\n        for teacher in teachers:\n            logits = teacher(**batch).logits\n            teacher_logits_list.append(logits)\n\n    # Average teacher predictions\n    avg_teacher_logits = torch.stack(teacher_logits_list).mean(dim=0)\n\n    # Student learns from ensemble\n    student_logits = student(**batch).logits\n    loss = F.kl_div(\n        F.log_softmax(student_logits, dim=-1),\n        F.softmax(avg_teacher_logits, dim=-1),\n        reduction='batchmean'\n    )\n\n    return loss\n```\n\n## Production Deployment\n\n### Complete Training Script\n\n```python\nfrom transformers import Trainer, TrainingArguments, DataCollatorForLanguageModeling\n\ndef train_distilled_model(\n    teacher_name=\"meta-llama\u002FLlama-2-70b-hf\",\n    student_name=\"meta-llama\u002FLlama-2-7b-hf\",\n    output_dir=\".\u002Fdistilled-llama-7b\",\n    temperature=2.0,\n    alpha=0.7,\n):\n    # Load models\n    teacher = AutoModelForCausalLM.from_pretrained(teacher_name, torch_dtype=torch.float16, device_map=\"auto\")\n    student = AutoModelForCausalLM.from_pretrained(student_name, torch_dtype=torch.float16)\n    tokenizer = AutoTokenizer.from_pretrained(teacher_name)\n\n    # Custom trainer with distillation\n    class DistillationTrainer(Trainer):\n        def compute_loss(self, model, inputs, return_outputs=False):\n            # Student forward\n            outputs_student = model(**inputs)\n            student_logits = outputs_student.logits\n\n            # Teacher forward (no grad)\n            with torch.no_grad():\n                outputs_teacher = teacher(**inputs)\n                teacher_logits = outputs_teacher.logits\n\n            # Distillation loss\n            soft_targets = F.softmax(teacher_logits \u002F temperature, dim=-1)\n            soft_student = F.log_softmax(student_logits \u002F temperature, dim=-1)\n            soft_loss = F.kl_div(soft_student, soft_targets, reduction='batchmean') * (temperature ** 2)\n\n            # Hard loss\n            hard_loss = outputs_student.loss\n\n            # Combined\n            loss = alpha * soft_loss + (1 - alpha) * hard_loss\n\n            return (loss, outputs_student) if return_outputs else loss\n\n    # Training arguments\n    training_args = TrainingArguments(\n        output_dir=output_dir,\n        num_train_epochs=3,\n        per_device_train_batch_size=4,\n        gradient_accumulation_steps=8,\n        learning_rate=2e-5,\n        warmup_steps=500,\n        logging_steps=100,\n        save_steps=1000,\n        bf16=True,\n        gradient_checkpointing=True,\n    )\n\n    # Train\n    trainer = DistillationTrainer(\n        model=student,\n        args=training_args,\n        train_dataset=train_dataset,\n        data_collator=DataCollatorForLanguageModeling(tokenizer, mlm=False),\n    )\n\n    trainer.train()\n    student.save_pretrained(output_dir)\n    tokenizer.save_pretrained(output_dir)\n\n# Usage\ntrain_distilled_model(\n    teacher_name=\"meta-llama\u002FLlama-2-70b-hf\",\n    student_name=\"meta-llama\u002FLlama-2-7b-hf\",\n    temperature=2.0,\n    alpha=0.7\n)\n```\n\n## Best Practices\n\n### 1. Hyperparameter Selection\n\n```python\n# Temperature\nT = 1.0  # Sharp (less knowledge transfer)\nT = 2.0  # Standard (good balance)\nT = 5.0  # Soft (more knowledge transfer)\n\n# Alpha (weight)\nalpha = 0.5  # Balanced\nalpha = 0.7  # Emphasize teacher knowledge\nalpha = 0.9  # Strong distillation\n\n# Rule: Higher T + higher alpha = stronger distillation\n```\n\n### 2. Model Size Ratio\n\n```python\n# Good ratios (teacher\u002Fstudent)\n70B \u002F 7B = 10×    # Excellent\n13B \u002F 1B = 13×    # Good\n7B \u002F 1B = 7×      # Acceptable\n\n# Avoid too large gap\n70B \u002F 1B = 70×    # Too large, ineffective\n```\n\n### 3. Data Quality\n\n```python\n# Best: Use teacher-generated data + real data\ntrain_data = {\n    \"teacher_generated\": 70%,  # Diverse, high-quality\n    \"real_data\": 30%            # Ground truth\n}\n\n# Avoid: Only real data (doesn't utilize teacher fully)\n```\n\n## Evaluation\n\n```python\nfrom transformers import pipeline\n\n# Compare student vs teacher\nteacher_pipe = pipeline(\"text-generation\", model=teacher)\nstudent_pipe = pipeline(\"text-generation\", model=student)\n\nprompts = [\"Explain quantum computing:\", \"What is AI?\"]\n\nfor prompt in prompts:\n    teacher_out = teacher_pipe(prompt, max_new_tokens=100)\n    student_out = student_pipe(prompt, max_new_tokens=100)\n\n    print(f\"Prompt: {prompt}\")\n    print(f\"Teacher: {teacher_out[0]['generated_text']}\")\n    print(f\"Student: {student_out[0]['generated_text']}\")\n    print(f\"Match quality: {calculate_similarity(teacher_out, student_out):.2f}\")\n```\n\n## Resources\n\n- **Hinton et al. 2015 (Foundational)**: https:\u002F\u002Farxiv.org\u002Fabs\u002F1503.02531\n- **MiniLLM (Reverse KLD)**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2306.08543\n- **KD Survey for LLMs (2024)**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2402.13116\n- **MiniLLM GitHub**: https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FLMOps\u002Ftree\u002Fmain\u002Fminillm\n\n\n",{"name":5,"tags":49,"author":51,"license":14,"version":52,"description":7,"dependencies":53},[37,25,50,36,35,34,33,32,31,30],"Model Compression","Orchestra Research","1.0.0",[54,55,56],"transformers","torch","datasets","main","f19b254e8975d0bfddb394686706106930d7964d","AI-Training\u002Fknowledge-distillation\u002Fknowledge-distillation","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus","https:\u002F\u002Fgithub.com\u002Fendman100","2026-11-13 20:29:52",{"source":8,"skillName":5,"commands":64,"prompt":74,"zipAvailable":75},[65,68,71],{"tool":66,"command":67},"npx","npx skills add endman100\u002Fskill-router-nexus --skill knowledge-distillation",{"tool":69,"command":70},"bunx","bunx skills add endman100\u002Fskill-router-nexus --skill knowledge-distillation",{"tool":72,"command":73},"pnpm","pnpm dlx skills add endman100\u002Fskill-router-nexus --skill knowledge-distillation","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Training\u002Fknowledge-distillation\u002Fknowledge-distillation\u002FSKILL.md",true,[77],{"path":78,"resourceType":79,"resourceTypeDesc":80,"sizeBytes":81,"sourceHtmlUrl":82},"references\u002Fminillm.md","REFERENCE","参考文档",9117,"https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Training\u002Fknowledge-distillation\u002Fknowledge-distillation\u002Freferences\u002Fminillm.md",[84,108,144,166,175,198],{"skillNo":85,"name":86,"nameZh":6,"description":87,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":88,"sourceHtmlUrl":89,"categories":90,"keywords":96,"publishedAt":44},"01M6MPCT5BXVER8HQZTW3QE9AS","create-master","基于佛教经典文献，生成特定高僧大德的 AI 教学角色","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FMetaphysics-and-Spirituality\u002FMaster-skill\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FMetaphysics-and-Spirituality\u002FMaster-skill\u002FSKILL.md",[91],{"categoryNo":92,"categoryName":93,"taxonomyType":94,"taxonomyTypeDesc":95},"01JBSEEDCAT000000000000039","教师与教育","PROFESSION","适用职业",[97,98,99,100,101,102,103,104,105,106,107],"create master","master","create","生成","true","webfetch","edit","write","read","grep","Bash",{"skillNo":109,"name":110,"nameZh":6,"description":111,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":112,"sourceHtmlUrl":113,"categories":114,"keywords":124,"publishedAt":44},"01M6M43AKE95Y64E1JRXEF5QQ3","writing-dna-skill","从至少 20 篇完整文章中蒸馏可复用的写作 DNA，分析语言、文章结构、选题逻辑、素材策略、认知框架和视觉风格，并生成 Writing-DNA.md；按该 DNA 写作时会读取全部蒸馏产物和 5 篇相关原文。用于中英文作者、账号、品牌或出版物的风格分析与一致性写作。 Distill reusable Writing DNA from at least 20 complete articles for Chinese or English authors, publications, brands, and accounts; use for language, structure, topic logic, source strategy, cognitive-frame, and visual-style analysis.","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fwriting-dna-skill\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fwriting-dna-skill\u002FSKILL.md",[115,118,121],{"categoryNo":116,"categoryName":117,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000018","写作与文案",{"categoryNo":119,"categoryName":120,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000023","Skill与智能体开发",{"categoryNo":122,"categoryName":123,"taxonomyType":94,"taxonomyTypeDesc":95},"01JBSEEDCAT000000000000036","内容创作者",[125,126,127,128,129,130,131,132,133,134,135,136,137,100,138,139,140,141,142,143],"writing dna skill","DNA","writing","strategy","english","logic","visual","structure","chinese","reusable","use","style","distill","analysis","frame","source","for","at","complete",{"skillNo":145,"name":146,"nameZh":6,"description":147,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":148,"sourceHtmlUrl":149,"categories":150,"keywords":153,"publishedAt":44},"01M6M428TTCGQ4ME3KY99VN037","stop-slop","Remove AI writing patterns from prose. Use when drafting, editing, or reviewing text to eliminate predictable AI tells.","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fstop-slop\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fstop-slop\u002FSKILL.md",[151,152],{"categoryNo":116,"categoryName":117,"taxonomyType":20,"taxonomyTypeDesc":21},{"categoryNo":122,"categoryName":123,"taxonomyType":94,"taxonomyTypeDesc":95},[154,155,156,157,26,158,159,160,161,135,162,163,164,141,165,127],"slop","stop slop","stop","drafting","drafts","https","patterns","prose","remove","editing","text","content",{"skillNo":167,"name":168,"nameZh":6,"description":169,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":170,"sourceHtmlUrl":171,"categories":172,"keywords":173,"publishedAt":44},"01M6M41X0B7EBX42TK435DMGGR","shuorenhua","检查和清理中英文文本里的 AI 套路，适用于“去 AI 味”“说人话”“自然一点”“别像模板”“先标问题”这类改写和审稿需求；按场景控制力度，同时保留事实、术语、语域和责任主体。","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fshuorenhua\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fshuorenhua\u002FSKILL.md",[],[168,174],"文本",{"skillNo":176,"name":177,"nameZh":6,"description":178,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":179,"sourceHtmlUrl":180,"categories":181,"keywords":182,"publishedAt":44},"01M6M3ZD12WBEW5KXKDM4N6DFG","naming","Name products, SaaS, brands, open source projects, bots, and apps. Use when the user needs to name something, find a brand name, or pick a product name. Metaphor-driven process that produces memorable, meaningful names and avoids AI slop.","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fnaming-main\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fnaming-main\u002FSKILL.md",[],[177,183,184,185,186,187,135,188,105,140,189,190,191,192,107,41,193,194,195,26,196,197,154],"repo","websearch","product name","apps","bots","view","brand","pick","gh","projects","saas","open source","product","describe","ai slop",{"skillNo":199,"name":200,"nameZh":6,"description":201,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":202,"sourceHtmlUrl":203,"categories":204,"keywords":206,"publishedAt":44},"01M6M3YVXAFAAWJ7F6JMPFREYE","humanizer-zh","去除文本中的 AI 生成痕迹。适用于编辑或审阅文本，使其听起来更自然、更像人类书写。\n基于维基百科的\"AI 写作特征\"综合指南。检测并修复以下模式：夸大的象征意义、\n宣传性语言、以 -ing 结尾的肤浅分析、模糊的归因、破折号过度使用、三段式法则、\nAI 词汇、否定式排比、过多的连接性短语。","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fhumanizer-zh\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fhumanizer-zh\u002FSKILL.md",[205],{"categoryNo":116,"categoryName":117,"taxonomyType":20,"taxonomyTypeDesc":21},[207,208,209,105,104,103,127,100,174,210,154,156],"humanizer","zh","humanizer zh","翻译"]