[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6XG4VSQ2WZQ2J54V6B05B0X":3,"skill-same:01M6XG4VSQ2WZQ2J54V6B05B0X":97},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":40,"linkOnly":41,"markdownContent":42,"markdownBody":43,"frontMatter":44,"refName":63,"commitSha":64,"skillDirectory":65,"forksCount":14,"repositoryHtmlUrl":66,"repositoryHomepageUrl":6,"ownerHtmlUrl":67,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":68,"install":69,"resources":82},"01M6XG4VSQ2WZQ2J54V6B05B0X","long-context",null,"Extend context windows of transformer models using RoPE, YaRN, ALiBi, and position interpolation techniques. Use when processing long documents (32k-128k+ tokens), extending pre-trained models beyond original context limits, or implementing efficient positional encodings. Covers rotary embeddings, attention biases, interpolation methods, and extrapolation strategies for LLMs.","fadrienne\u002FMyGitHub","fadrienne","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F197874571?v=4","01M6X64V8VWCRSMFZ4EPS2BWXG","Francesca Adrienne","01M6X64VD5MW4F9PHDWQKS7ZAF",0,"MIT","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fopenscience\u002Fbackend\u002Fcli\u002Fskills\u002Fllm-tools\u002Flong-context\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39],"context","long","long context","synthetic","extending","when","limits","interpolation","extend","implementing","Emerging Techniques","processing","llms","strategies","flash","Tools","Transformers","embeddings","yarn","documents","windows","2026-11-10 10:12:30",false,"---\nname: long-context\ndescription: Extend context windows of transformer models using RoPE, YaRN, ALiBi, and position interpolation techniques. Use when processing long documents (32k-128k+ tokens), extending pre-trained models beyond original context limits, or implementing efficient positional encodings. Covers rotary embeddings, attention biases, interpolation methods, and extrapolation strategies for LLMs.\ncategory: llm-tools\nversion: 1.0.0\nauthor: Synthetic Sciences\nlicense: MIT\ntags: [Emerging Techniques, Long Context, RoPE, YaRN, ALiBi, Position Interpolation, Extended Context, Rotary Embeddings, Attention Bias, Context Extension, Positional Encoding]\ndependencies: [transformers, torch, flash-attn]\n---\n\n# Long Context: Extending Transformer Context Windows\n\n## When to Use This Skill\n\nUse Long Context techniques when you need to:\n- **Process long documents** (32k, 64k, 128k+ tokens) with transformer models\n- **Extend context windows** of pre-trained models (LLaMA, Mistral, etc.)\n- **Implement efficient positional encodings** (RoPE, ALiBi)\n- **Train models** with length extrapolation capabilities\n- **Deploy models** that handle variable-length inputs efficiently\n- **Fine-tune** existing models for longer contexts with minimal compute\n\n**Key Techniques**: RoPE (Rotary Position Embeddings), YaRN, ALiBi (Attention with Linear Biases), Position Interpolation\n\n**Papers**: RoFormer (arXiv 2104.09864), YaRN (arXiv 2309.00071), ALiBi (arXiv 2108.12409), Position Interpolation (arXiv 2306.15595)\n\n## Installation\n\n```bash\n# HuggingFace Transformers (includes RoPE, YaRN support)\npip install transformers torch\n\n# For custom implementations\npip install einops  # Tensor operations\npip install rotary-embedding-torch  # Standalone RoPE\n\n# Optional: FlashAttention for efficiency\npip install flash-attn --no-build-isolation\n```\n\n## Quick Start\n\n### RoPE (Rotary Position Embeddings)\n\n```python\nimport torch\nimport torch.nn as nn\n\nclass RotaryEmbedding(nn.Module):\n    \"\"\"Rotary Position Embeddings (RoPE).\"\"\"\n\n    def __init__(self, dim, max_seq_len=8192, base=10000):\n        super().__init__()\n        # Compute inverse frequencies\n        inv_freq = 1.0 \u002F (base ** (torch.arange(0, dim, 2).float() \u002F dim))\n        self.register_buffer(\"inv_freq\", inv_freq)\n        self.max_seq_len = max_seq_len\n\n    def forward(self, seq_len, device):\n        # Position indices\n        t = torch.arange(seq_len, device=device).type_as(self.inv_freq)\n\n        # Compute frequencies\n        freqs = torch.outer(t, self.inv_freq)  # (seq_len, dim\u002F2)\n\n        # Compute sin and cos\n        emb = torch.cat((freqs, freqs), dim=-1)  # (seq_len, dim)\n        return emb.cos(), emb.sin()\n\ndef rotate_half(x):\n    \"\"\"Rotate half the hidden dimensions.\"\"\"\n    x1, x2 = x.chunk(2, dim=-1)\n    return torch.cat((-x2, x1), dim=-1)\n\ndef apply_rotary_pos_emb(q, k, cos, sin):\n    \"\"\"Apply rotary embeddings to queries and keys.\"\"\"\n    # q, k shape: (batch, heads, seq_len, dim)\n    q_embed = (q * cos) + (rotate_half(q) * sin)\n    k_embed = (k * cos) + (rotate_half(k) * sin)\n    return q_embed, k_embed\n\n# Usage\nrope = RotaryEmbedding(dim=64, max_seq_len=8192)\ncos, sin = rope(seq_len=2048, device='cuda')\n\n# In attention layer\nq_rotated, k_rotated = apply_rotary_pos_emb(query, key, cos, sin)\n```\n\n### ALiBi (Attention with Linear Biases)\n\n```python\ndef get_alibi_slopes(num_heads):\n    \"\"\"Get ALiBi slope values for each attention head.\"\"\"\n    def get_slopes_power_of_2(n):\n        start = 2 ** (-(2 ** -(math.log2(n) - 3)))\n        ratio = start\n        return [start * (ratio ** i) for i in range(n)]\n\n    if math.log2(num_heads).is_integer():\n        return get_slopes_power_of_2(num_heads)\n    else:\n        # Closest power of 2\n        closest_power = 2 ** math.floor(math.log2(num_heads))\n        slopes = get_slopes_power_of_2(closest_power)\n        # Add extra slopes\n        extra = get_slopes_power_of_2(2 * closest_power)\n        slopes.extend(extra[0::2][:num_heads - closest_power])\n        return slopes\n\ndef create_alibi_bias(seq_len, num_heads):\n    \"\"\"Create ALiBi attention bias.\"\"\"\n    # Distance matrix\n    context_position = torch.arange(seq_len)\n    memory_position = torch.arange(seq_len)\n    relative_position = memory_position[None, :] - context_position[:, None]\n\n    # Get slopes\n    slopes = torch.tensor(get_alibi_slopes(num_heads))\n\n    # Apply slopes to distances\n    alibi = slopes[:, None, None] * relative_position[None, :, :]\n    return alibi  # (num_heads, seq_len, seq_len)\n\n# Usage in attention\nnum_heads = 8\nseq_len = 2048\nalibi_bias = create_alibi_bias(seq_len, num_heads).to('cuda')\n\n# Add bias to attention scores\n# attn_scores shape: (batch, num_heads, seq_len, seq_len)\nattn_scores = attn_scores + alibi_bias\nattn_weights = torch.softmax(attn_scores, dim=-1)\n```\n\n### Position Interpolation for LLaMA\n\n```python\nfrom transformers import LlamaForCausalLM, LlamaTokenizer\n\n# Original context: 2048 tokens\nmodel = LlamaForCausalLM.from_pretrained(\"meta-llama\u002FLlama-2-7b-hf\")\n\n# Extend to 32k with position interpolation\n# Modify RoPE base frequency\nmodel.config.rope_scaling = {\n    \"type\": \"linear\",\n    \"factor\": 16.0  # 2048 * 16 = 32768\n}\n\n# Or use dynamic scaling\nmodel.config.rope_scaling = {\n    \"type\": \"dynamic\",\n    \"factor\": 16.0\n}\n\n# Fine-tune with long documents (minimal steps needed)\n# Position interpolation works out-of-the-box after this config change\n```\n\n## Core Concepts\n\n### 1. RoPE (Rotary Position Embeddings)\n\n**How it works:**\n- Encodes absolute position via rotation matrix\n- Provides relative position dependency in attention\n- Enables length extrapolation\n\n**Mathematical formulation:**\n```\nq_m = (W_q * x_m) * e^(imθ)\nk_n = (W_k * x_n) * e^(inθ)\n\nwhere θ_j = base^(-2j\u002Fd) for j ∈ [0, d\u002F2)\n```\n\n**Advantages:**\n- Decaying inter-token dependency with distance\n- Compatible with linear attention\n- Better extrapolation than absolute position encodings\n\n### 2. YaRN (Yet another RoPE extensioN)\n\n**Key innovation:**\n- NTK-aware interpolation (Neural Tangent Kernel)\n- Attention temperature scaling\n- Efficient context extension (10× less tokens vs baselines)\n\n**Parameters:**\n```python\n# YaRN configuration\nyarn_config = {\n    \"scale\": 16,                    # Extension factor\n    \"original_max_position\": 2048,  # Base context\n    \"extrapolation_factor\": 1.0,    # NTK parameter\n    \"attn_factor\": 1.0,             # Attention scaling\n    \"beta_fast\": 32,                # High-frequency scale\n    \"beta_slow\": 1,                 # Low-frequency scale\n}\n```\n\n**Performance:**\n- Extends LLaMA to 128k tokens\n- 2.5× less training steps than baselines\n- State-of-the-art context window extension\n\n### 3. ALiBi (Attention with Linear Biases)\n\n**Core idea:**\n- No positional embeddings added to tokens\n- Apply distance penalty directly to attention scores\n- Bias proportional to key-query distance\n\n**Formula:**\n```\nattention_bias[i, j] = -m * |i - j|\n\nwhere m = slope for each attention head\n```\n\n**Advantages:**\n- 11% faster training vs sinusoidal embeddings\n- 11% less memory usage\n- Strong length extrapolation (train 1k, test 2k+)\n- Inductive bias towards recency\n\n### 4. Position Interpolation\n\n**Technique:**\n- Linearly down-scale position indices\n- Interpolate within trained range (vs extrapolate beyond)\n- Minimal fine-tuning required\n\n**Formula:**\n```\n# Original: position indices [0, 1, 2, ..., L]\n# Extended: position indices [0, 0.5, 1.0, ..., L\u002F2]\n# (for 2× extension)\n\nscaled_position[i] = i \u002F extension_factor\n```\n\n**Results:**\n- LLaMA 7B-65B extended to 32k tokens\n- 1000 fine-tuning steps sufficient\n- 600× better stability than extrapolation\n\n## Method Comparison\n\n| Method | Max Context | Training Needed | Memory | Extrapolation | Best For |\n|--------|-------------|-----------------|--------|---------------|----------|\n| **RoPE** | 8k-32k | Full pre-training | Moderate | Good | New models |\n| **YaRN** | 32k-128k | Minimal (10× efficient) | Moderate | Excellent | Extending existing models |\n| **ALiBi** | Unlimited | Full pre-training | Low (-11%) | Excellent | Training from scratch |\n| **Position Interpolation** | 32k+ | Minimal (1k steps) | Moderate | Poor (by design) | Quick extension |\n\n## Implementation Patterns\n\n### HuggingFace Transformers Integration\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoConfig\n\n# RoPE with YaRN scaling\nconfig = AutoConfig.from_pretrained(\"mistralai\u002FMistral-7B-v0.1\")\nconfig.rope_scaling = {\n    \"type\": \"yarn\",\n    \"factor\": 8.0,\n    \"original_max_position_embeddings\": 8192,\n    \"attention_factor\": 1.0\n}\n\nmodel = AutoModelForCausalLM.from_config(config)\n\n# Position interpolation (simpler)\nconfig.rope_scaling = {\n    \"type\": \"linear\",\n    \"factor\": 4.0\n}\n\n# Dynamic scaling (adjusts based on input length)\nconfig.rope_scaling = {\n    \"type\": \"dynamic\",\n    \"factor\": 8.0\n}\n```\n\n### Custom RoPE Implementation\n\n```python\nclass LongContextAttention(nn.Module):\n    \"\"\"Multi-head attention with RoPE.\"\"\"\n\n    def __init__(self, hidden_size, num_heads, max_seq_len=32768):\n        super().__init__()\n        self.num_heads = num_heads\n        self.head_dim = hidden_size \u002F\u002F num_heads\n\n        # Q, K, V projections\n        self.q_proj = nn.Linear(hidden_size, hidden_size)\n        self.k_proj = nn.Linear(hidden_size, hidden_size)\n        self.v_proj = nn.Linear(hidden_size, hidden_size)\n        self.o_proj = nn.Linear(hidden_size, hidden_size)\n\n        # RoPE\n        self.rotary_emb = RotaryEmbedding(\n            dim=self.head_dim,\n            max_seq_len=max_seq_len\n        )\n\n    def forward(self, hidden_states):\n        batch_size, seq_len, _ = hidden_states.shape\n\n        # Project to Q, K, V\n        q = self.q_proj(hidden_states)\n        k = self.k_proj(hidden_states)\n        v = self.v_proj(hidden_states)\n\n        # Reshape for multi-head\n        q = q.view(batch_size, seq_len, self.num_heads, self.head_dim).transpose(1, 2)\n        k = k.view(batch_size, seq_len, self.num_heads, self.head_dim).transpose(1, 2)\n        v = v.view(batch_size, seq_len, self.num_heads, self.head_dim).transpose(1, 2)\n\n        # Apply RoPE\n        cos, sin = self.rotary_emb(seq_len, device=hidden_states.device)\n        q, k = apply_rotary_pos_emb(q, k, cos, sin)\n\n        # Standard attention\n        attn_output = F.scaled_dot_product_attention(q, k, v)\n\n        # Reshape and project\n        attn_output = attn_output.transpose(1, 2).contiguous()\n        attn_output = attn_output.view(batch_size, seq_len, -1)\n        output = self.o_proj(attn_output)\n\n        return output\n```\n\n## Fine-tuning for Long Context\n\n### Minimal Fine-tuning (Position Interpolation)\n\n```python\nfrom transformers import Trainer, TrainingArguments\n\n# Extend model config\nmodel.config.max_position_embeddings = 32768\nmodel.config.rope_scaling = {\"type\": \"linear\", \"factor\": 16.0}\n\n# Training args (minimal steps needed)\ntraining_args = TrainingArguments(\n    output_dir=\".\u002Fllama-32k\",\n    num_train_epochs=1,\n    max_steps=1000,           # Only 1000 steps!\n    per_device_train_batch_size=1,\n    gradient_accumulation_steps=16,\n    learning_rate=2e-5,\n    warmup_steps=100,\n    logging_steps=10,\n    save_steps=500,\n)\n\n# Train on long documents\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=long_document_dataset,  # 32k token sequences\n)\n\ntrainer.train()\n```\n\n### YaRN Fine-tuning\n\n```bash\n# Clone YaRN implementation\ngit clone https:\u002F\u002Fgithub.com\u002Fjquesnelle\u002Fyarn\ncd yarn\n\n# Fine-tune LLaMA with YaRN\npython scripts\u002Ftrain.py \\\n    --model meta-llama\u002FLlama-2-7b-hf \\\n    --scale 16 \\\n    --rope_theta 10000 \\\n    --max_length 32768 \\\n    --batch_size 1 \\\n    --gradient_accumulation 16 \\\n    --steps 400 \\\n    --learning_rate 2e-5\n```\n\n## Best Practices\n\n### 1. Choose the Right Method\n\n```python\n# For NEW models (training from scratch)\nuse_method = \"ALiBi\"  # Best extrapolation, lowest memory\n\n# For EXTENDING existing RoPE models\nuse_method = \"YaRN\"  # Most efficient extension (10× less data)\n\n# For QUICK extension with minimal compute\nuse_method = \"Position Interpolation\"  # 1000 steps\n\n# For MODERATE extension with good efficiency\nuse_method = \"Linear RoPE Scaling\"  # Built-in, simple\n```\n\n### 2. Scaling Factor Selection\n\n```python\n# Conservative (safer, better quality)\nscaling_factor = 2.0  # 8k → 16k\n\n# Moderate (good balance)\nscaling_factor = 4.0  # 8k → 32k\n\n# Aggressive (requires more fine-tuning)\nscaling_factor = 8.0  # 8k → 64k\nscaling_factor = 16.0  # 8k → 128k\n\n# Rule: Larger factors need more fine-tuning steps\nsteps_needed = 100 * scaling_factor  # Rough estimate\n```\n\n### 3. Fine-tuning Data\n\n```python\n# ✅ Good: Long documents matching target length\ntrain_data = [\n    {\"text\": long_doc_32k_tokens},  # Full 32k\n    {\"text\": long_doc_24k_tokens},  # Varied lengths\n    {\"text\": long_doc_16k_tokens},\n]\n\n# ❌ Bad: Short documents (won't learn long context)\ntrain_data = [\n    {\"text\": short_doc_2k_tokens},\n]\n\n# Use datasets like:\n# - PG-19 (books, long texts)\n# - arXiv papers\n# - Long-form conversations\n# - GitHub repositories (concatenated files)\n```\n\n### 4. Avoid Common Pitfalls\n\n```python\n# ❌ Bad: Applying position interpolation without fine-tuning\nmodel.config.rope_scaling = {\"type\": \"linear\", \"factor\": 16.0}\n# Model will perform poorly without fine-tuning!\n\n# ✅ Good: Fine-tune after scaling\nmodel.config.rope_scaling = {\"type\": \"linear\", \"factor\": 16.0}\nfine_tune(model, long_documents, steps=1000)\n\n# ❌ Bad: Too aggressive scaling without data\nscale_to_1M_tokens()  # Won't work without massive fine-tuning\n\n# ✅ Good: Incremental scaling\n# 8k → 16k → 32k → 64k (fine-tune at each step)\n```\n\n## Production Deployment\n\n### Inference with Long Context\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load long-context model\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"togethercomputer\u002FLLaMA-2-7B-32K\",  # 32k context\n    torch_dtype=torch.float16,\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"togethercomputer\u002FLLaMA-2-7B-32K\")\n\n# Process long document\nlong_text = \"...\" * 30000  # 30k tokens\ninputs = tokenizer(long_text, return_tensors=\"pt\", truncation=False).to('cuda')\n\n# Generate\noutputs = model.generate(\n    **inputs,\n    max_new_tokens=512,\n    temperature=0.7,\n)\n\nresponse = tokenizer.decode(outputs[0], skip_special_tokens=True)\n```\n\n### Memory Optimization\n\n```python\n# Use gradient checkpointing for fine-tuning\nmodel.gradient_checkpointing_enable()\n\n# Use Flash Attention 2\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    attn_implementation=\"flash_attention_2\",  # 2-3× faster\n    torch_dtype=torch.float16\n)\n\n# Use paged attention (vLLM)\nfrom vllm import LLM\n\nllm = LLM(\n    model=\"togethercomputer\u002FLLaMA-2-7B-32K\",\n    max_model_len=32768,  # 32k context\n    gpu_memory_utilization=0.9\n)\n```\n\n## Resources\n\n- **RoPE Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2104.09864 (RoFormer)\n- **YaRN Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2309.00071\n- **ALiBi Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2108.12409 (Train Short, Test Long)\n- **Position Interpolation**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2306.15595\n- **HuggingFace RoPE Utils**: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftransformers\u002Fblob\u002Fmain\u002Fsrc\u002Ftransformers\u002Fmodeling_rope_utils.py\n- **YaRN Implementation**: https:\u002F\u002Fgithub.com\u002Fjquesnelle\u002Fyarn\n- **Together AI Blog**: https:\u002F\u002Fwww.together.ai\u002Fblog\u002Fllama-2-7b-32k\n\n## See Also\n\n- `references\u002Frope.md` - Detailed RoPE implementation and theory\n- `references\u002Fextension_methods.md` - YaRN, ALiBi, Position Interpolation comparisons\n- `references\u002Ffine_tuning.md` - Complete fine-tuning guide for context extension\n\n\n","# Long Context: Extending Transformer Context Windows\n\n## When to Use This Skill\n\nUse Long Context techniques when you need to:\n- **Process long documents** (32k, 64k, 128k+ tokens) with transformer models\n- **Extend context windows** of pre-trained models (LLaMA, Mistral, etc.)\n- **Implement efficient positional encodings** (RoPE, ALiBi)\n- **Train models** with length extrapolation capabilities\n- **Deploy models** that handle variable-length inputs efficiently\n- **Fine-tune** existing models for longer contexts with minimal compute\n\n**Key Techniques**: RoPE (Rotary Position Embeddings), YaRN, ALiBi (Attention with Linear Biases), Position Interpolation\n\n**Papers**: RoFormer (arXiv 2104.09864), YaRN (arXiv 2309.00071), ALiBi (arXiv 2108.12409), Position Interpolation (arXiv 2306.15595)\n\n## Installation\n\n```bash\n# HuggingFace Transformers (includes RoPE, YaRN support)\npip install transformers torch\n\n# For custom implementations\npip install einops  # Tensor operations\npip install rotary-embedding-torch  # Standalone RoPE\n\n# Optional: FlashAttention for efficiency\npip install flash-attn --no-build-isolation\n```\n\n## Quick Start\n\n### RoPE (Rotary Position Embeddings)\n\n```python\nimport torch\nimport torch.nn as nn\n\nclass RotaryEmbedding(nn.Module):\n    \"\"\"Rotary Position Embeddings (RoPE).\"\"\"\n\n    def __init__(self, dim, max_seq_len=8192, base=10000):\n        super().__init__()\n        # Compute inverse frequencies\n        inv_freq = 1.0 \u002F (base ** (torch.arange(0, dim, 2).float() \u002F dim))\n        self.register_buffer(\"inv_freq\", inv_freq)\n        self.max_seq_len = max_seq_len\n\n    def forward(self, seq_len, device):\n        # Position indices\n        t = torch.arange(seq_len, device=device).type_as(self.inv_freq)\n\n        # Compute frequencies\n        freqs = torch.outer(t, self.inv_freq)  # (seq_len, dim\u002F2)\n\n        # Compute sin and cos\n        emb = torch.cat((freqs, freqs), dim=-1)  # (seq_len, dim)\n        return emb.cos(), emb.sin()\n\ndef rotate_half(x):\n    \"\"\"Rotate half the hidden dimensions.\"\"\"\n    x1, x2 = x.chunk(2, dim=-1)\n    return torch.cat((-x2, x1), dim=-1)\n\ndef apply_rotary_pos_emb(q, k, cos, sin):\n    \"\"\"Apply rotary embeddings to queries and keys.\"\"\"\n    # q, k shape: (batch, heads, seq_len, dim)\n    q_embed = (q * cos) + (rotate_half(q) * sin)\n    k_embed = (k * cos) + (rotate_half(k) * sin)\n    return q_embed, k_embed\n\n# Usage\nrope = RotaryEmbedding(dim=64, max_seq_len=8192)\ncos, sin = rope(seq_len=2048, device='cuda')\n\n# In attention layer\nq_rotated, k_rotated = apply_rotary_pos_emb(query, key, cos, sin)\n```\n\n### ALiBi (Attention with Linear Biases)\n\n```python\ndef get_alibi_slopes(num_heads):\n    \"\"\"Get ALiBi slope values for each attention head.\"\"\"\n    def get_slopes_power_of_2(n):\n        start = 2 ** (-(2 ** -(math.log2(n) - 3)))\n        ratio = start\n        return [start * (ratio ** i) for i in range(n)]\n\n    if math.log2(num_heads).is_integer():\n        return get_slopes_power_of_2(num_heads)\n    else:\n        # Closest power of 2\n        closest_power = 2 ** math.floor(math.log2(num_heads))\n        slopes = get_slopes_power_of_2(closest_power)\n        # Add extra slopes\n        extra = get_slopes_power_of_2(2 * closest_power)\n        slopes.extend(extra[0::2][:num_heads - closest_power])\n        return slopes\n\ndef create_alibi_bias(seq_len, num_heads):\n    \"\"\"Create ALiBi attention bias.\"\"\"\n    # Distance matrix\n    context_position = torch.arange(seq_len)\n    memory_position = torch.arange(seq_len)\n    relative_position = memory_position[None, :] - context_position[:, None]\n\n    # Get slopes\n    slopes = torch.tensor(get_alibi_slopes(num_heads))\n\n    # Apply slopes to distances\n    alibi = slopes[:, None, None] * relative_position[None, :, :]\n    return alibi  # (num_heads, seq_len, seq_len)\n\n# Usage in attention\nnum_heads = 8\nseq_len = 2048\nalibi_bias = create_alibi_bias(seq_len, num_heads).to('cuda')\n\n# Add bias to attention scores\n# attn_scores shape: (batch, num_heads, seq_len, seq_len)\nattn_scores = attn_scores + alibi_bias\nattn_weights = torch.softmax(attn_scores, dim=-1)\n```\n\n### Position Interpolation for LLaMA\n\n```python\nfrom transformers import LlamaForCausalLM, LlamaTokenizer\n\n# Original context: 2048 tokens\nmodel = LlamaForCausalLM.from_pretrained(\"meta-llama\u002FLlama-2-7b-hf\")\n\n# Extend to 32k with position interpolation\n# Modify RoPE base frequency\nmodel.config.rope_scaling = {\n    \"type\": \"linear\",\n    \"factor\": 16.0  # 2048 * 16 = 32768\n}\n\n# Or use dynamic scaling\nmodel.config.rope_scaling = {\n    \"type\": \"dynamic\",\n    \"factor\": 16.0\n}\n\n# Fine-tune with long documents (minimal steps needed)\n# Position interpolation works out-of-the-box after this config change\n```\n\n## Core Concepts\n\n### 1. RoPE (Rotary Position Embeddings)\n\n**How it works:**\n- Encodes absolute position via rotation matrix\n- Provides relative position dependency in attention\n- Enables length extrapolation\n\n**Mathematical formulation:**\n```\nq_m = (W_q * x_m) * e^(imθ)\nk_n = (W_k * x_n) * e^(inθ)\n\nwhere θ_j = base^(-2j\u002Fd) for j ∈ [0, d\u002F2)\n```\n\n**Advantages:**\n- Decaying inter-token dependency with distance\n- Compatible with linear attention\n- Better extrapolation than absolute position encodings\n\n### 2. YaRN (Yet another RoPE extensioN)\n\n**Key innovation:**\n- NTK-aware interpolation (Neural Tangent Kernel)\n- Attention temperature scaling\n- Efficient context extension (10× less tokens vs baselines)\n\n**Parameters:**\n```python\n# YaRN configuration\nyarn_config = {\n    \"scale\": 16,                    # Extension factor\n    \"original_max_position\": 2048,  # Base context\n    \"extrapolation_factor\": 1.0,    # NTK parameter\n    \"attn_factor\": 1.0,             # Attention scaling\n    \"beta_fast\": 32,                # High-frequency scale\n    \"beta_slow\": 1,                 # Low-frequency scale\n}\n```\n\n**Performance:**\n- Extends LLaMA to 128k tokens\n- 2.5× less training steps than baselines\n- State-of-the-art context window extension\n\n### 3. ALiBi (Attention with Linear Biases)\n\n**Core idea:**\n- No positional embeddings added to tokens\n- Apply distance penalty directly to attention scores\n- Bias proportional to key-query distance\n\n**Formula:**\n```\nattention_bias[i, j] = -m * |i - j|\n\nwhere m = slope for each attention head\n```\n\n**Advantages:**\n- 11% faster training vs sinusoidal embeddings\n- 11% less memory usage\n- Strong length extrapolation (train 1k, test 2k+)\n- Inductive bias towards recency\n\n### 4. Position Interpolation\n\n**Technique:**\n- Linearly down-scale position indices\n- Interpolate within trained range (vs extrapolate beyond)\n- Minimal fine-tuning required\n\n**Formula:**\n```\n# Original: position indices [0, 1, 2, ..., L]\n# Extended: position indices [0, 0.5, 1.0, ..., L\u002F2]\n# (for 2× extension)\n\nscaled_position[i] = i \u002F extension_factor\n```\n\n**Results:**\n- LLaMA 7B-65B extended to 32k tokens\n- 1000 fine-tuning steps sufficient\n- 600× better stability than extrapolation\n\n## Method Comparison\n\n| Method | Max Context | Training Needed | Memory | Extrapolation | Best For |\n|--------|-------------|-----------------|--------|---------------|----------|\n| **RoPE** | 8k-32k | Full pre-training | Moderate | Good | New models |\n| **YaRN** | 32k-128k | Minimal (10× efficient) | Moderate | Excellent | Extending existing models |\n| **ALiBi** | Unlimited | Full pre-training | Low (-11%) | Excellent | Training from scratch |\n| **Position Interpolation** | 32k+ | Minimal (1k steps) | Moderate | Poor (by design) | Quick extension |\n\n## Implementation Patterns\n\n### HuggingFace Transformers Integration\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoConfig\n\n# RoPE with YaRN scaling\nconfig = AutoConfig.from_pretrained(\"mistralai\u002FMistral-7B-v0.1\")\nconfig.rope_scaling = {\n    \"type\": \"yarn\",\n    \"factor\": 8.0,\n    \"original_max_position_embeddings\": 8192,\n    \"attention_factor\": 1.0\n}\n\nmodel = AutoModelForCausalLM.from_config(config)\n\n# Position interpolation (simpler)\nconfig.rope_scaling = {\n    \"type\": \"linear\",\n    \"factor\": 4.0\n}\n\n# Dynamic scaling (adjusts based on input length)\nconfig.rope_scaling = {\n    \"type\": \"dynamic\",\n    \"factor\": 8.0\n}\n```\n\n### Custom RoPE Implementation\n\n```python\nclass LongContextAttention(nn.Module):\n    \"\"\"Multi-head attention with RoPE.\"\"\"\n\n    def __init__(self, hidden_size, num_heads, max_seq_len=32768):\n        super().__init__()\n        self.num_heads = num_heads\n        self.head_dim = hidden_size \u002F\u002F num_heads\n\n        # Q, K, V projections\n        self.q_proj = nn.Linear(hidden_size, hidden_size)\n        self.k_proj = nn.Linear(hidden_size, hidden_size)\n        self.v_proj = nn.Linear(hidden_size, hidden_size)\n        self.o_proj = nn.Linear(hidden_size, hidden_size)\n\n        # RoPE\n        self.rotary_emb = RotaryEmbedding(\n            dim=self.head_dim,\n            max_seq_len=max_seq_len\n        )\n\n    def forward(self, hidden_states):\n        batch_size, seq_len, _ = hidden_states.shape\n\n        # Project to Q, K, V\n        q = self.q_proj(hidden_states)\n        k = self.k_proj(hidden_states)\n        v = self.v_proj(hidden_states)\n\n        # Reshape for multi-head\n        q = q.view(batch_size, seq_len, self.num_heads, self.head_dim).transpose(1, 2)\n        k = k.view(batch_size, seq_len, self.num_heads, self.head_dim).transpose(1, 2)\n        v = v.view(batch_size, seq_len, self.num_heads, self.head_dim).transpose(1, 2)\n\n        # Apply RoPE\n        cos, sin = self.rotary_emb(seq_len, device=hidden_states.device)\n        q, k = apply_rotary_pos_emb(q, k, cos, sin)\n\n        # Standard attention\n        attn_output = F.scaled_dot_product_attention(q, k, v)\n\n        # Reshape and project\n        attn_output = attn_output.transpose(1, 2).contiguous()\n        attn_output = attn_output.view(batch_size, seq_len, -1)\n        output = self.o_proj(attn_output)\n\n        return output\n```\n\n## Fine-tuning for Long Context\n\n### Minimal Fine-tuning (Position Interpolation)\n\n```python\nfrom transformers import Trainer, TrainingArguments\n\n# Extend model config\nmodel.config.max_position_embeddings = 32768\nmodel.config.rope_scaling = {\"type\": \"linear\", \"factor\": 16.0}\n\n# Training args (minimal steps needed)\ntraining_args = TrainingArguments(\n    output_dir=\".\u002Fllama-32k\",\n    num_train_epochs=1,\n    max_steps=1000,           # Only 1000 steps!\n    per_device_train_batch_size=1,\n    gradient_accumulation_steps=16,\n    learning_rate=2e-5,\n    warmup_steps=100,\n    logging_steps=10,\n    save_steps=500,\n)\n\n# Train on long documents\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=long_document_dataset,  # 32k token sequences\n)\n\ntrainer.train()\n```\n\n### YaRN Fine-tuning\n\n```bash\n# Clone YaRN implementation\ngit clone https:\u002F\u002Fgithub.com\u002Fjquesnelle\u002Fyarn\ncd yarn\n\n# Fine-tune LLaMA with YaRN\npython scripts\u002Ftrain.py \\\n    --model meta-llama\u002FLlama-2-7b-hf \\\n    --scale 16 \\\n    --rope_theta 10000 \\\n    --max_length 32768 \\\n    --batch_size 1 \\\n    --gradient_accumulation 16 \\\n    --steps 400 \\\n    --learning_rate 2e-5\n```\n\n## Best Practices\n\n### 1. Choose the Right Method\n\n```python\n# For NEW models (training from scratch)\nuse_method = \"ALiBi\"  # Best extrapolation, lowest memory\n\n# For EXTENDING existing RoPE models\nuse_method = \"YaRN\"  # Most efficient extension (10× less data)\n\n# For QUICK extension with minimal compute\nuse_method = \"Position Interpolation\"  # 1000 steps\n\n# For MODERATE extension with good efficiency\nuse_method = \"Linear RoPE Scaling\"  # Built-in, simple\n```\n\n### 2. Scaling Factor Selection\n\n```python\n# Conservative (safer, better quality)\nscaling_factor = 2.0  # 8k → 16k\n\n# Moderate (good balance)\nscaling_factor = 4.0  # 8k → 32k\n\n# Aggressive (requires more fine-tuning)\nscaling_factor = 8.0  # 8k → 64k\nscaling_factor = 16.0  # 8k → 128k\n\n# Rule: Larger factors need more fine-tuning steps\nsteps_needed = 100 * scaling_factor  # Rough estimate\n```\n\n### 3. Fine-tuning Data\n\n```python\n# ✅ Good: Long documents matching target length\ntrain_data = [\n    {\"text\": long_doc_32k_tokens},  # Full 32k\n    {\"text\": long_doc_24k_tokens},  # Varied lengths\n    {\"text\": long_doc_16k_tokens},\n]\n\n# ❌ Bad: Short documents (won't learn long context)\ntrain_data = [\n    {\"text\": short_doc_2k_tokens},\n]\n\n# Use datasets like:\n# - PG-19 (books, long texts)\n# - arXiv papers\n# - Long-form conversations\n# - GitHub repositories (concatenated files)\n```\n\n### 4. Avoid Common Pitfalls\n\n```python\n# ❌ Bad: Applying position interpolation without fine-tuning\nmodel.config.rope_scaling = {\"type\": \"linear\", \"factor\": 16.0}\n# Model will perform poorly without fine-tuning!\n\n# ✅ Good: Fine-tune after scaling\nmodel.config.rope_scaling = {\"type\": \"linear\", \"factor\": 16.0}\nfine_tune(model, long_documents, steps=1000)\n\n# ❌ Bad: Too aggressive scaling without data\nscale_to_1M_tokens()  # Won't work without massive fine-tuning\n\n# ✅ Good: Incremental scaling\n# 8k → 16k → 32k → 64k (fine-tune at each step)\n```\n\n## Production Deployment\n\n### Inference with Long Context\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load long-context model\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"togethercomputer\u002FLLaMA-2-7B-32K\",  # 32k context\n    torch_dtype=torch.float16,\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"togethercomputer\u002FLLaMA-2-7B-32K\")\n\n# Process long document\nlong_text = \"...\" * 30000  # 30k tokens\ninputs = tokenizer(long_text, return_tensors=\"pt\", truncation=False).to('cuda')\n\n# Generate\noutputs = model.generate(\n    **inputs,\n    max_new_tokens=512,\n    temperature=0.7,\n)\n\nresponse = tokenizer.decode(outputs[0], skip_special_tokens=True)\n```\n\n### Memory Optimization\n\n```python\n# Use gradient checkpointing for fine-tuning\nmodel.gradient_checkpointing_enable()\n\n# Use Flash Attention 2\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    attn_implementation=\"flash_attention_2\",  # 2-3× faster\n    torch_dtype=torch.float16\n)\n\n# Use paged attention (vLLM)\nfrom vllm import LLM\n\nllm = LLM(\n    model=\"togethercomputer\u002FLLaMA-2-7B-32K\",\n    max_model_len=32768,  # 32k context\n    gpu_memory_utilization=0.9\n)\n```\n\n## Resources\n\n- **RoPE Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2104.09864 (RoFormer)\n- **YaRN Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2309.00071\n- **ALiBi Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2108.12409 (Train Short, Test Long)\n- **Position Interpolation**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2306.15595\n- **HuggingFace RoPE Utils**: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftransformers\u002Fblob\u002Fmain\u002Fsrc\u002Ftransformers\u002Fmodeling_rope_utils.py\n- **YaRN Implementation**: https:\u002F\u002Fgithub.com\u002Fjquesnelle\u002Fyarn\n- **Together AI Blog**: https:\u002F\u002Fwww.together.ai\u002Fblog\u002Fllama-2-7b-32k\n\n## See Also\n\n- `references\u002Frope.md` - Detailed RoPE implementation and theory\n- `references\u002Fextension_methods.md` - YaRN, ALiBi, Position Interpolation comparisons\n- `references\u002Ffine_tuning.md` - Complete fine-tuning guide for context extension\n\n\n",{"name":5,"tags":45,"author":56,"license":15,"version":57,"category":58,"description":7,"dependencies":59},[29,46,47,48,49,50,51,52,53,54,55],"Long Context","RoPE","YaRN","ALiBi","Position Interpolation","Extended Context","Rotary Embeddings","Attention Bias","Context Extension","Positional Encoding","Synthetic Sciences","1.0.0","llm-tools",[60,61,62],"transformers","torch","flash-attn","main","4aa39efd1cdab0b4657ecee16c92a92011a5b10f","openscience\u002Fbackend\u002Fcli\u002Fskills\u002Fllm-tools\u002Flong-context","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub","https:\u002F\u002Fgithub.com\u002Ffadrienne","2026-10-28 00:43:09",{"source":8,"skillName":5,"commands":70,"prompt":80,"zipAvailable":81},[71,74,77],{"tool":72,"command":73},"npx","npx skills add fadrienne\u002FMyGitHub --skill long-context",{"tool":75,"command":76},"bunx","bunx skills add fadrienne\u002FMyGitHub --skill long-context",{"tool":78,"command":79},"pnpm","pnpm dlx skills add fadrienne\u002FMyGitHub --skill long-context","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fopenscience\u002Fbackend\u002Fcli\u002Fskills\u002Fllm-tools\u002Flong-context\u002FSKILL.md",true,[83,89,93],{"path":84,"resourceType":85,"resourceTypeDesc":86,"sizeBytes":87,"sourceHtmlUrl":88},"references\u002Fextension_methods.md","REFERENCE","参考文档",14537,"https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fopenscience\u002Fbackend\u002Fcli\u002Fskills\u002Fllm-tools\u002Flong-context\u002Freferences\u002Fextension_methods.md",{"path":90,"resourceType":85,"resourceTypeDesc":86,"sizeBytes":91,"sourceHtmlUrl":92},"references\u002Ffine_tuning.md",16431,"https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fopenscience\u002Fbackend\u002Fcli\u002Fskills\u002Fllm-tools\u002Flong-context\u002Freferences\u002Ffine_tuning.md",{"path":94,"resourceType":85,"resourceTypeDesc":86,"sizeBytes":95,"sourceHtmlUrl":96},"references\u002Frope.md",12121,"https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fopenscience\u002Fbackend\u002Fcli\u002Fskills\u002Fllm-tools\u002Flong-context\u002Freferences\u002Frope.md",[98,129,155,189,216,243],{"skillNo":99,"name":100,"nameZh":6,"description":101,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":102,"categories":103,"keywords":109,"publishedAt":40},"01M7167B28PK20HP7R7ZED2DY0","statistical-power","Sample size calculation, power analysis, and effect size conversion for scientific studies. Use BEFORE data collection (a priori) to determine required N, or after (post-hoc) to compute achieved power. Covers parametric tests (t-test, ANOVA, chi-square, proportions, correlation, regression), simulation-based power for complex\u002Fclustered\u002Fnested designs, and power curves. Use this skill when someone asks 'how many samples do I need', 'what is my power', 'what effect size can I detect', or any variant. Pairs with \u002Fexperimental-design (choose design first, then compute power) and \u002Fstatistical-analysis (analyze data after collection).","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fscientific-agent-skills\u002Fskills\u002Fstatistical-power\u002FSKILL.md",[104],{"categoryNo":105,"categoryName":106,"taxonomyType":107,"taxonomyTypeDesc":108},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[110,111,112,113,114,115,116,117,24,118,119,120,121,122,123,124,125,126,127,128],"power","statistical power","effect","scipy","regression","choose","design","uv","Bash","collection","analysis","calculation","ANOVA","conversion","analyze","t-test","chi-square","simulation","sample",{"skillNo":130,"name":131,"nameZh":6,"description":132,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":133,"categories":134,"keywords":138,"publishedAt":40},"01M71677EBCM8EPCE9BZCK8505","statistical-analysis","Guided statistical analysis with test selection and reporting. Use when you need help choosing appropriate tests for your data, assumption checking, power analysis, and APA-formatted results. Best for academic research reporting, test selection guidance. For implementing specific models programmatically use statsmodels.","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fscientific-agent-skills\u002Fskills\u002Fstatistical-analysis\u002FSKILL.md",[135],{"categoryNo":136,"categoryName":137,"taxonomyType":107,"taxonomyTypeDesc":108},"01JBSEEDCAT000000000000008","测试与质量",[120,139,110,140,28,141,24,142,143,144,145,146,147,148,149,150,151,152,153,154],"statistical analysis","academic research","selection","you","reporting","data","for","tests","use","with","test","research","results","statsmodels","models","help",{"skillNo":156,"name":157,"nameZh":6,"description":158,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":159,"categories":160,"keywords":170,"publishedAt":40},"01M71673R93XAHFH81DBNVNG7H","scientific-writing","Core skill for the deep research and writing tool. Write scientific manuscripts in full paragraphs (never bullet points). Use two-stage process with (1) section outlines with key points using research-lookup then (2) convert to flowing prose. IMRAD structure, citations (APA\u002FAMA\u002FVancouver), figures\u002Ftables, reporting guidelines (CONSORT\u002FSTROBE\u002FPRISMA), for research papers and journal submissions.","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fscientific-agent-skills\u002Fskills\u002Fscientific-writing\u002FSKILL.md",[161,164,165],{"categoryNo":162,"categoryName":163,"taxonomyType":107,"taxonomyTypeDesc":108},"01JBSEEDCAT000000000000018","写作与文案",{"categoryNo":105,"categoryName":106,"taxonomyType":107,"taxonomyTypeDesc":108},{"categoryNo":166,"categoryName":167,"taxonomyType":168,"taxonomyTypeDesc":169},"01JBSEEDCAT000000000000036","内容创作者","PROFESSION","适用职业",[171,172,173,174,175,176,177,178,179,180,118,181,182,183,184,185,186,187,143,188],"writing","scientific","scientific writing","Citations","deep","deep research","never","papers","journal","false","api","prisma","structure","openrouter","Outlines","core skill","read","features",{"skillNo":190,"name":191,"nameZh":6,"description":192,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":193,"categories":194,"keywords":199,"publishedAt":40},"01M7166ZPKXBF4EBPHVTJPQBB1","scientific-visualization","Meta-skill for publication-ready figures. Use when creating journal submission figures requiring multi-panel layouts, significance annotations, error bars, colorblind-safe palettes, and specific journal formatting (Nature, Science, Cell). Orchestrates matplotlib\u002Fseaborn\u002Fplotly with publication styles. For quick exploration use seaborn or plotly directly.","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fscientific-agent-skills\u002Fskills\u002Fscientific-visualization\u002FSKILL.md",[195,198],{"categoryNo":196,"categoryName":197,"taxonomyType":107,"taxonomyTypeDesc":108},"01JBSEEDCAT000000000000013","分析与可视化",{"categoryNo":105,"categoryName":106,"taxonomyType":107,"taxonomyTypeDesc":108},[200,172,201,202,203,204,205,206,207,179,24,208,209,210,211,212,213,214,145,215],"visualization","scientific visualization","error","ready","quick","exploration","meta-skill","Meta","safe","plotly","annotations","formatting","Publication","figures","panel","submission",{"skillNo":217,"name":218,"nameZh":6,"description":219,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":220,"categories":221,"keywords":228,"publishedAt":40},"01M7166VXJWQSSC3RQZP60WBGR","research-code","Debug, profile, review, and restructure Python research and analysis code. Use when analysis scripts crash, produce wrong results, run too slowly, or need a correctness review. Covers: reading tracebacks and diagnosing root causes, hypothesis-driven debugging, reviewing code for common statistical\u002Fnumerical errors, profiling data pipelines, improving reproducibility (seeds, environments, determinism), and structuring analysis scripts for sharing. Distinct from \u002Fexploratory-data-analysis (which understands data) and \u002Fstatistical-analysis (which chooses and runs tests) — this skill focuses on the code itself.","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fscientific-agent-skills\u002Fskills\u002Fresearch-code\u002FSKILL.md",[222,225],{"categoryNo":223,"categoryName":224,"taxonomyType":107,"taxonomyTypeDesc":108},"01JBSEEDCAT000000000000021","资料检索与研究",{"categoryNo":226,"categoryName":227,"taxonomyType":168,"taxonomyTypeDesc":169},"01JBSEEDCAT000000000000030","开发工程师",[229,230,150,231,232,233,234,117,235,24,118,120,236,237,238,239,240,241,187,242],"code","research code","which","memory","data-analysis","errors","debugging","common","review","data pipelines","run","scripts","line","write",{"skillNo":244,"name":245,"nameZh":6,"description":246,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":247,"licenseFileUrl":6,"sourceHtmlUrl":248,"categories":249,"keywords":256,"publishedAt":40},"01M7166CF5HSR7SVZYZFWTDX55","markdown-mermaid-writing","Comprehensive markdown and Mermaid diagram writing skill. Use when creating any scientific document, report, analysis, or visualization. Establishes text-based diagrams as the default documentation standard with full style guides (markdown + mermaid), 24 diagram type references, and 9 document templates.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fscientific-agent-skills\u002Fskills\u002Fmarkdown-mermaid-writing\u002FSKILL.md",[250,253,254,255],{"categoryNo":251,"categoryName":252,"taxonomyType":107,"taxonomyTypeDesc":108},"01JBSEEDCAT000000000000002","文档处理",{"categoryNo":162,"categoryName":163,"taxonomyType":107,"taxonomyTypeDesc":108},{"categoryNo":105,"categoryName":106,"taxonomyType":107,"taxonomyTypeDesc":108},{"categoryNo":166,"categoryName":167,"taxonomyType":168,"taxonomyTypeDesc":169},[171,257,258,259,260,261,200,262,263,24,118,120,264,265,266,267,268,269,172,187],"markdown mermaid writing","mermaid","markdown","community","diagrams","project","default","team","guides","text","documentation","references","diagram"]