[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6KQ69GZ7KV64CFA78B5HTP3":3,"skill-same:01M6KQ69GZ7KV64CFA78B5HTP3":99},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":43,"linkOnly":44,"markdownContent":45,"markdownBody":46,"frontMatter":47,"refName":65,"commitSha":66,"skillDirectory":67,"forksCount":13,"repositoryHtmlUrl":68,"repositoryHomepageUrl":6,"ownerHtmlUrl":69,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":70,"install":71,"resources":84},"01M6KQ69GZ7KV64CFA78B5HTP3","long-context",null,"Extend context windows of transformer models using RoPE, YaRN, ALiBi, and position interpolation techniques. Use when processing long documents (32k-128k+ tokens), extending pre-trained models beyond original context limits, or implementing efficient positional encodings. Covers rotary embeddings, attention biases, interpolation methods, and extrapolation strategies for LLMs.","endman100\u002Fskill-router-nexus","endman100","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F31997019?v=4","01M6KPDX17BYDWW4RRB0TNGGMX","01M6KPDX64Y2QBEWVFG5W5PWPF",0,"MIT","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Models-and-Architecture\u002Flong-context\u002Flong-context\u002FSKILL.md",[],[18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42],"context","long context","long","extending","llms","when","limits","interpolation","extend","for","implementing","orchestra","techniques","Emerging Techniques","processing","tokens","attention","strategies","methods","flash","Transformers","embeddings","yarn","documents","windows","2026-11-10 10:12:30",false,"---\nname: long-context\ndescription: Extend context windows of transformer models using RoPE, YaRN, ALiBi, and position interpolation techniques. Use when processing long documents (32k-128k+ tokens), extending pre-trained models beyond original context limits, or implementing efficient positional encodings. Covers rotary embeddings, attention biases, interpolation methods, and extrapolation strategies for LLMs.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Emerging Techniques, Long Context, RoPE, YaRN, ALiBi, Position Interpolation, Extended Context, Rotary Embeddings, Attention Bias, Context Extension, Positional Encoding]\ndependencies: [transformers, torch, flash-attn]\n---\n\n# Long Context: Extending Transformer Context Windows\n\n## When to Use This Skill\n\nUse Long Context techniques when you need to:\n- **Process long documents** (32k, 64k, 128k+ tokens) with transformer models\n- **Extend context windows** of pre-trained models (LLaMA, Mistral, etc.)\n- **Implement efficient positional encodings** (RoPE, ALiBi)\n- **Train models** with length extrapolation capabilities\n- **Deploy models** that handle variable-length inputs efficiently\n- **Fine-tune** existing models for longer contexts with minimal compute\n\n**Key Techniques**: RoPE (Rotary Position Embeddings), YaRN, ALiBi (Attention with Linear Biases), Position Interpolation\n\n**Papers**: RoFormer (arXiv 2104.09864), YaRN (arXiv 2309.00071), ALiBi (arXiv 2108.12409), Position Interpolation (arXiv 2306.15595)\n\n## Installation\n\n```bash\n# HuggingFace Transformers (includes RoPE, YaRN support)\npip install transformers torch\n\n# For custom implementations\npip install einops  # Tensor operations\npip install rotary-embedding-torch  # Standalone RoPE\n\n# Optional: FlashAttention for efficiency\npip install flash-attn --no-build-isolation\n```\n\n## Quick Start\n\n### RoPE (Rotary Position Embeddings)\n\n```python\nimport torch\nimport torch.nn as nn\n\nclass RotaryEmbedding(nn.Module):\n    \"\"\"Rotary Position Embeddings (RoPE).\"\"\"\n\n    def __init__(self, dim, max_seq_len=8192, base=10000):\n        super().__init__()\n        # Compute inverse frequencies\n        inv_freq = 1.0 \u002F (base ** (torch.arange(0, dim, 2).float() \u002F dim))\n        self.register_buffer(\"inv_freq\", inv_freq)\n        self.max_seq_len = max_seq_len\n\n    def forward(self, seq_len, device):\n        # Position indices\n        t = torch.arange(seq_len, device=device).type_as(self.inv_freq)\n\n        # Compute frequencies\n        freqs = torch.outer(t, self.inv_freq)  # (seq_len, dim\u002F2)\n\n        # Compute sin and cos\n        emb = torch.cat((freqs, freqs), dim=-1)  # (seq_len, dim)\n        return emb.cos(), emb.sin()\n\ndef rotate_half(x):\n    \"\"\"Rotate half the hidden dimensions.\"\"\"\n    x1, x2 = x.chunk(2, dim=-1)\n    return torch.cat((-x2, x1), dim=-1)\n\ndef apply_rotary_pos_emb(q, k, cos, sin):\n    \"\"\"Apply rotary embeddings to queries and keys.\"\"\"\n    # q, k shape: (batch, heads, seq_len, dim)\n    q_embed = (q * cos) + (rotate_half(q) * sin)\n    k_embed = (k * cos) + (rotate_half(k) * sin)\n    return q_embed, k_embed\n\n# Usage\nrope = RotaryEmbedding(dim=64, max_seq_len=8192)\ncos, sin = rope(seq_len=2048, device='cuda')\n\n# In attention layer\nq_rotated, k_rotated = apply_rotary_pos_emb(query, key, cos, sin)\n```\n\n### ALiBi (Attention with Linear Biases)\n\n```python\ndef get_alibi_slopes(num_heads):\n    \"\"\"Get ALiBi slope values for each attention head.\"\"\"\n    def get_slopes_power_of_2(n):\n        start = 2 ** (-(2 ** -(math.log2(n) - 3)))\n        ratio = start\n        return [start * (ratio ** i) for i in range(n)]\n\n    if math.log2(num_heads).is_integer():\n        return get_slopes_power_of_2(num_heads)\n    else:\n        # Closest power of 2\n        closest_power = 2 ** math.floor(math.log2(num_heads))\n        slopes = get_slopes_power_of_2(closest_power)\n        # Add extra slopes\n        extra = get_slopes_power_of_2(2 * closest_power)\n        slopes.extend(extra[0::2][:num_heads - closest_power])\n        return slopes\n\ndef create_alibi_bias(seq_len, num_heads):\n    \"\"\"Create ALiBi attention bias.\"\"\"\n    # Distance matrix\n    context_position = torch.arange(seq_len)\n    memory_position = torch.arange(seq_len)\n    relative_position = memory_position[None, :] - context_position[:, None]\n\n    # Get slopes\n    slopes = torch.tensor(get_alibi_slopes(num_heads))\n\n    # Apply slopes to distances\n    alibi = slopes[:, None, None] * relative_position[None, :, :]\n    return alibi  # (num_heads, seq_len, seq_len)\n\n# Usage in attention\nnum_heads = 8\nseq_len = 2048\nalibi_bias = create_alibi_bias(seq_len, num_heads).to('cuda')\n\n# Add bias to attention scores\n# attn_scores shape: (batch, num_heads, seq_len, seq_len)\nattn_scores = attn_scores + alibi_bias\nattn_weights = torch.softmax(attn_scores, dim=-1)\n```\n\n### Position Interpolation for LLaMA\n\n```python\nfrom transformers import LlamaForCausalLM, LlamaTokenizer\n\n# Original context: 2048 tokens\nmodel = LlamaForCausalLM.from_pretrained(\"meta-llama\u002FLlama-2-7b-hf\")\n\n# Extend to 32k with position interpolation\n# Modify RoPE base frequency\nmodel.config.rope_scaling = {\n    \"type\": \"linear\",\n    \"factor\": 16.0  # 2048 * 16 = 32768\n}\n\n# Or use dynamic scaling\nmodel.config.rope_scaling = {\n    \"type\": \"dynamic\",\n    \"factor\": 16.0\n}\n\n# Fine-tune with long documents (minimal steps needed)\n# Position interpolation works out-of-the-box after this config change\n```\n\n## Core Concepts\n\n### 1. RoPE (Rotary Position Embeddings)\n\n**How it works:**\n- Encodes absolute position via rotation matrix\n- Provides relative position dependency in attention\n- Enables length extrapolation\n\n**Mathematical formulation:**\n```\nq_m = (W_q * x_m) * e^(imθ)\nk_n = (W_k * x_n) * e^(inθ)\n\nwhere θ_j = base^(-2j\u002Fd) for j ∈ [0, d\u002F2)\n```\n\n**Advantages:**\n- Decaying inter-token dependency with distance\n- Compatible with linear attention\n- Better extrapolation than absolute position encodings\n\n### 2. YaRN (Yet another RoPE extensioN)\n\n**Key innovation:**\n- NTK-aware interpolation (Neural Tangent Kernel)\n- Attention temperature scaling\n- Efficient context extension (10× less tokens vs baselines)\n\n**Parameters:**\n```python\n# YaRN configuration\nyarn_config = {\n    \"scale\": 16,                    # Extension factor\n    \"original_max_position\": 2048,  # Base context\n    \"extrapolation_factor\": 1.0,    # NTK parameter\n    \"attn_factor\": 1.0,             # Attention scaling\n    \"beta_fast\": 32,                # High-frequency scale\n    \"beta_slow\": 1,                 # Low-frequency scale\n}\n```\n\n**Performance:**\n- Extends LLaMA to 128k tokens\n- 2.5× less training steps than baselines\n- State-of-the-art context window extension\n\n### 3. ALiBi (Attention with Linear Biases)\n\n**Core idea:**\n- No positional embeddings added to tokens\n- Apply distance penalty directly to attention scores\n- Bias proportional to key-query distance\n\n**Formula:**\n```\nattention_bias[i, j] = -m * |i - j|\n\nwhere m = slope for each attention head\n```\n\n**Advantages:**\n- 11% faster training vs sinusoidal embeddings\n- 11% less memory usage\n- Strong length extrapolation (train 1k, test 2k+)\n- Inductive bias towards recency\n\n### 4. Position Interpolation\n\n**Technique:**\n- Linearly down-scale position indices\n- Interpolate within trained range (vs extrapolate beyond)\n- Minimal fine-tuning required\n\n**Formula:**\n```\n# Original: position indices [0, 1, 2, ..., L]\n# Extended: position indices [0, 0.5, 1.0, ..., L\u002F2]\n# (for 2× extension)\n\nscaled_position[i] = i \u002F extension_factor\n```\n\n**Results:**\n- LLaMA 7B-65B extended to 32k tokens\n- 1000 fine-tuning steps sufficient\n- 600× better stability than extrapolation\n\n## Method Comparison\n\n| Method | Max Context | Training Needed | Memory | Extrapolation | Best For |\n|--------|-------------|-----------------|--------|---------------|----------|\n| **RoPE** | 8k-32k | Full pre-training | Moderate | Good | New models |\n| **YaRN** | 32k-128k | Minimal (10× efficient) | Moderate | Excellent | Extending existing models |\n| **ALiBi** | Unlimited | Full pre-training | Low (-11%) | Excellent | Training from scratch |\n| **Position Interpolation** | 32k+ | Minimal (1k steps) | Moderate | Poor (by design) | Quick extension |\n\n## Implementation Patterns\n\n### HuggingFace Transformers Integration\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoConfig\n\n# RoPE with YaRN scaling\nconfig = AutoConfig.from_pretrained(\"mistralai\u002FMistral-7B-v0.1\")\nconfig.rope_scaling = {\n    \"type\": \"yarn\",\n    \"factor\": 8.0,\n    \"original_max_position_embeddings\": 8192,\n    \"attention_factor\": 1.0\n}\n\nmodel = AutoModelForCausalLM.from_config(config)\n\n# Position interpolation (simpler)\nconfig.rope_scaling = {\n    \"type\": \"linear\",\n    \"factor\": 4.0\n}\n\n# Dynamic scaling (adjusts based on input length)\nconfig.rope_scaling = {\n    \"type\": \"dynamic\",\n    \"factor\": 8.0\n}\n```\n\n### Custom RoPE Implementation\n\n```python\nclass LongContextAttention(nn.Module):\n    \"\"\"Multi-head attention with RoPE.\"\"\"\n\n    def __init__(self, hidden_size, num_heads, max_seq_len=32768):\n        super().__init__()\n        self.num_heads = num_heads\n        self.head_dim = hidden_size \u002F\u002F num_heads\n\n        # Q, K, V projections\n        self.q_proj = nn.Linear(hidden_size, hidden_size)\n        self.k_proj = nn.Linear(hidden_size, hidden_size)\n        self.v_proj = nn.Linear(hidden_size, hidden_size)\n        self.o_proj = nn.Linear(hidden_size, hidden_size)\n\n        # RoPE\n        self.rotary_emb = RotaryEmbedding(\n            dim=self.head_dim,\n            max_seq_len=max_seq_len\n        )\n\n    def forward(self, hidden_states):\n        batch_size, seq_len, _ = hidden_states.shape\n\n        # Project to Q, K, V\n        q = self.q_proj(hidden_states)\n        k = self.k_proj(hidden_states)\n        v = self.v_proj(hidden_states)\n\n        # Reshape for multi-head\n        q = q.view(batch_size, seq_len, self.num_heads, self.head_dim).transpose(1, 2)\n        k = k.view(batch_size, seq_len, self.num_heads, self.head_dim).transpose(1, 2)\n        v = v.view(batch_size, seq_len, self.num_heads, self.head_dim).transpose(1, 2)\n\n        # Apply RoPE\n        cos, sin = self.rotary_emb(seq_len, device=hidden_states.device)\n        q, k = apply_rotary_pos_emb(q, k, cos, sin)\n\n        # Standard attention\n        attn_output = F.scaled_dot_product_attention(q, k, v)\n\n        # Reshape and project\n        attn_output = attn_output.transpose(1, 2).contiguous()\n        attn_output = attn_output.view(batch_size, seq_len, -1)\n        output = self.o_proj(attn_output)\n\n        return output\n```\n\n## Fine-tuning for Long Context\n\n### Minimal Fine-tuning (Position Interpolation)\n\n```python\nfrom transformers import Trainer, TrainingArguments\n\n# Extend model config\nmodel.config.max_position_embeddings = 32768\nmodel.config.rope_scaling = {\"type\": \"linear\", \"factor\": 16.0}\n\n# Training args (minimal steps needed)\ntraining_args = TrainingArguments(\n    output_dir=\".\u002Fllama-32k\",\n    num_train_epochs=1,\n    max_steps=1000,           # Only 1000 steps!\n    per_device_train_batch_size=1,\n    gradient_accumulation_steps=16,\n    learning_rate=2e-5,\n    warmup_steps=100,\n    logging_steps=10,\n    save_steps=500,\n)\n\n# Train on long documents\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=long_document_dataset,  # 32k token sequences\n)\n\ntrainer.train()\n```\n\n### YaRN Fine-tuning\n\n```bash\n# Clone YaRN implementation\ngit clone https:\u002F\u002Fgithub.com\u002Fjquesnelle\u002Fyarn\ncd yarn\n\n# Fine-tune LLaMA with YaRN\npython scripts\u002Ftrain.py \\\n    --model meta-llama\u002FLlama-2-7b-hf \\\n    --scale 16 \\\n    --rope_theta 10000 \\\n    --max_length 32768 \\\n    --batch_size 1 \\\n    --gradient_accumulation 16 \\\n    --steps 400 \\\n    --learning_rate 2e-5\n```\n\n## Best Practices\n\n### 1. Choose the Right Method\n\n```python\n# For NEW models (training from scratch)\nuse_method = \"ALiBi\"  # Best extrapolation, lowest memory\n\n# For EXTENDING existing RoPE models\nuse_method = \"YaRN\"  # Most efficient extension (10× less data)\n\n# For QUICK extension with minimal compute\nuse_method = \"Position Interpolation\"  # 1000 steps\n\n# For MODERATE extension with good efficiency\nuse_method = \"Linear RoPE Scaling\"  # Built-in, simple\n```\n\n### 2. Scaling Factor Selection\n\n```python\n# Conservative (safer, better quality)\nscaling_factor = 2.0  # 8k → 16k\n\n# Moderate (good balance)\nscaling_factor = 4.0  # 8k → 32k\n\n# Aggressive (requires more fine-tuning)\nscaling_factor = 8.0  # 8k → 64k\nscaling_factor = 16.0  # 8k → 128k\n\n# Rule: Larger factors need more fine-tuning steps\nsteps_needed = 100 * scaling_factor  # Rough estimate\n```\n\n### 3. Fine-tuning Data\n\n```python\n# ✅ Good: Long documents matching target length\ntrain_data = [\n    {\"text\": long_doc_32k_tokens},  # Full 32k\n    {\"text\": long_doc_24k_tokens},  # Varied lengths\n    {\"text\": long_doc_16k_tokens},\n]\n\n# ❌ Bad: Short documents (won't learn long context)\ntrain_data = [\n    {\"text\": short_doc_2k_tokens},\n]\n\n# Use datasets like:\n# - PG-19 (books, long texts)\n# - arXiv papers\n# - Long-form conversations\n# - GitHub repositories (concatenated files)\n```\n\n### 4. Avoid Common Pitfalls\n\n```python\n# ❌ Bad: Applying position interpolation without fine-tuning\nmodel.config.rope_scaling = {\"type\": \"linear\", \"factor\": 16.0}\n# Model will perform poorly without fine-tuning!\n\n# ✅ Good: Fine-tune after scaling\nmodel.config.rope_scaling = {\"type\": \"linear\", \"factor\": 16.0}\nfine_tune(model, long_documents, steps=1000)\n\n# ❌ Bad: Too aggressive scaling without data\nscale_to_1M_tokens()  # Won't work without massive fine-tuning\n\n# ✅ Good: Incremental scaling\n# 8k → 16k → 32k → 64k (fine-tune at each step)\n```\n\n## Production Deployment\n\n### Inference with Long Context\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load long-context model\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"togethercomputer\u002FLLaMA-2-7B-32K\",  # 32k context\n    torch_dtype=torch.float16,\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"togethercomputer\u002FLLaMA-2-7B-32K\")\n\n# Process long document\nlong_text = \"...\" * 30000  # 30k tokens\ninputs = tokenizer(long_text, return_tensors=\"pt\", truncation=False).to('cuda')\n\n# Generate\noutputs = model.generate(\n    **inputs,\n    max_new_tokens=512,\n    temperature=0.7,\n)\n\nresponse = tokenizer.decode(outputs[0], skip_special_tokens=True)\n```\n\n### Memory Optimization\n\n```python\n# Use gradient checkpointing for fine-tuning\nmodel.gradient_checkpointing_enable()\n\n# Use Flash Attention 2\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    attn_implementation=\"flash_attention_2\",  # 2-3× faster\n    torch_dtype=torch.float16\n)\n\n# Use paged attention (vLLM)\nfrom vllm import LLM\n\nllm = LLM(\n    model=\"togethercomputer\u002FLLaMA-2-7B-32K\",\n    max_model_len=32768,  # 32k context\n    gpu_memory_utilization=0.9\n)\n```\n\n## Resources\n\n- **RoPE Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2104.09864 (RoFormer)\n- **YaRN Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2309.00071\n- **ALiBi Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2108.12409 (Train Short, Test Long)\n- **Position Interpolation**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2306.15595\n- **HuggingFace RoPE Utils**: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftransformers\u002Fblob\u002Fmain\u002Fsrc\u002Ftransformers\u002Fmodeling_rope_utils.py\n- **YaRN Implementation**: https:\u002F\u002Fgithub.com\u002Fjquesnelle\u002Fyarn\n- **Together AI Blog**: https:\u002F\u002Fwww.together.ai\u002Fblog\u002Fllama-2-7b-32k\n\n## See Also\n\n- `references\u002Frope.md` - Detailed RoPE implementation and theory\n- `references\u002Fextension_methods.md` - YaRN, ALiBi, Position Interpolation comparisons\n- `references\u002Ffine_tuning.md` - Complete fine-tuning guide for context extension\n\n\n","# Long Context: Extending Transformer Context Windows\n\n## When to Use This Skill\n\nUse Long Context techniques when you need to:\n- **Process long documents** (32k, 64k, 128k+ tokens) with transformer models\n- **Extend context windows** of pre-trained models (LLaMA, Mistral, etc.)\n- **Implement efficient positional encodings** (RoPE, ALiBi)\n- **Train models** with length extrapolation capabilities\n- **Deploy models** that handle variable-length inputs efficiently\n- **Fine-tune** existing models for longer contexts with minimal compute\n\n**Key Techniques**: RoPE (Rotary Position Embeddings), YaRN, ALiBi (Attention with Linear Biases), Position Interpolation\n\n**Papers**: RoFormer (arXiv 2104.09864), YaRN (arXiv 2309.00071), ALiBi (arXiv 2108.12409), Position Interpolation (arXiv 2306.15595)\n\n## Installation\n\n```bash\n# HuggingFace Transformers (includes RoPE, YaRN support)\npip install transformers torch\n\n# For custom implementations\npip install einops  # Tensor operations\npip install rotary-embedding-torch  # Standalone RoPE\n\n# Optional: FlashAttention for efficiency\npip install flash-attn --no-build-isolation\n```\n\n## Quick Start\n\n### RoPE (Rotary Position Embeddings)\n\n```python\nimport torch\nimport torch.nn as nn\n\nclass RotaryEmbedding(nn.Module):\n    \"\"\"Rotary Position Embeddings (RoPE).\"\"\"\n\n    def __init__(self, dim, max_seq_len=8192, base=10000):\n        super().__init__()\n        # Compute inverse frequencies\n        inv_freq = 1.0 \u002F (base ** (torch.arange(0, dim, 2).float() \u002F dim))\n        self.register_buffer(\"inv_freq\", inv_freq)\n        self.max_seq_len = max_seq_len\n\n    def forward(self, seq_len, device):\n        # Position indices\n        t = torch.arange(seq_len, device=device).type_as(self.inv_freq)\n\n        # Compute frequencies\n        freqs = torch.outer(t, self.inv_freq)  # (seq_len, dim\u002F2)\n\n        # Compute sin and cos\n        emb = torch.cat((freqs, freqs), dim=-1)  # (seq_len, dim)\n        return emb.cos(), emb.sin()\n\ndef rotate_half(x):\n    \"\"\"Rotate half the hidden dimensions.\"\"\"\n    x1, x2 = x.chunk(2, dim=-1)\n    return torch.cat((-x2, x1), dim=-1)\n\ndef apply_rotary_pos_emb(q, k, cos, sin):\n    \"\"\"Apply rotary embeddings to queries and keys.\"\"\"\n    # q, k shape: (batch, heads, seq_len, dim)\n    q_embed = (q * cos) + (rotate_half(q) * sin)\n    k_embed = (k * cos) + (rotate_half(k) * sin)\n    return q_embed, k_embed\n\n# Usage\nrope = RotaryEmbedding(dim=64, max_seq_len=8192)\ncos, sin = rope(seq_len=2048, device='cuda')\n\n# In attention layer\nq_rotated, k_rotated = apply_rotary_pos_emb(query, key, cos, sin)\n```\n\n### ALiBi (Attention with Linear Biases)\n\n```python\ndef get_alibi_slopes(num_heads):\n    \"\"\"Get ALiBi slope values for each attention head.\"\"\"\n    def get_slopes_power_of_2(n):\n        start = 2 ** (-(2 ** -(math.log2(n) - 3)))\n        ratio = start\n        return [start * (ratio ** i) for i in range(n)]\n\n    if math.log2(num_heads).is_integer():\n        return get_slopes_power_of_2(num_heads)\n    else:\n        # Closest power of 2\n        closest_power = 2 ** math.floor(math.log2(num_heads))\n        slopes = get_slopes_power_of_2(closest_power)\n        # Add extra slopes\n        extra = get_slopes_power_of_2(2 * closest_power)\n        slopes.extend(extra[0::2][:num_heads - closest_power])\n        return slopes\n\ndef create_alibi_bias(seq_len, num_heads):\n    \"\"\"Create ALiBi attention bias.\"\"\"\n    # Distance matrix\n    context_position = torch.arange(seq_len)\n    memory_position = torch.arange(seq_len)\n    relative_position = memory_position[None, :] - context_position[:, None]\n\n    # Get slopes\n    slopes = torch.tensor(get_alibi_slopes(num_heads))\n\n    # Apply slopes to distances\n    alibi = slopes[:, None, None] * relative_position[None, :, :]\n    return alibi  # (num_heads, seq_len, seq_len)\n\n# Usage in attention\nnum_heads = 8\nseq_len = 2048\nalibi_bias = create_alibi_bias(seq_len, num_heads).to('cuda')\n\n# Add bias to attention scores\n# attn_scores shape: (batch, num_heads, seq_len, seq_len)\nattn_scores = attn_scores + alibi_bias\nattn_weights = torch.softmax(attn_scores, dim=-1)\n```\n\n### Position Interpolation for LLaMA\n\n```python\nfrom transformers import LlamaForCausalLM, LlamaTokenizer\n\n# Original context: 2048 tokens\nmodel = LlamaForCausalLM.from_pretrained(\"meta-llama\u002FLlama-2-7b-hf\")\n\n# Extend to 32k with position interpolation\n# Modify RoPE base frequency\nmodel.config.rope_scaling = {\n    \"type\": \"linear\",\n    \"factor\": 16.0  # 2048 * 16 = 32768\n}\n\n# Or use dynamic scaling\nmodel.config.rope_scaling = {\n    \"type\": \"dynamic\",\n    \"factor\": 16.0\n}\n\n# Fine-tune with long documents (minimal steps needed)\n# Position interpolation works out-of-the-box after this config change\n```\n\n## Core Concepts\n\n### 1. RoPE (Rotary Position Embeddings)\n\n**How it works:**\n- Encodes absolute position via rotation matrix\n- Provides relative position dependency in attention\n- Enables length extrapolation\n\n**Mathematical formulation:**\n```\nq_m = (W_q * x_m) * e^(imθ)\nk_n = (W_k * x_n) * e^(inθ)\n\nwhere θ_j = base^(-2j\u002Fd) for j ∈ [0, d\u002F2)\n```\n\n**Advantages:**\n- Decaying inter-token dependency with distance\n- Compatible with linear attention\n- Better extrapolation than absolute position encodings\n\n### 2. YaRN (Yet another RoPE extensioN)\n\n**Key innovation:**\n- NTK-aware interpolation (Neural Tangent Kernel)\n- Attention temperature scaling\n- Efficient context extension (10× less tokens vs baselines)\n\n**Parameters:**\n```python\n# YaRN configuration\nyarn_config = {\n    \"scale\": 16,                    # Extension factor\n    \"original_max_position\": 2048,  # Base context\n    \"extrapolation_factor\": 1.0,    # NTK parameter\n    \"attn_factor\": 1.0,             # Attention scaling\n    \"beta_fast\": 32,                # High-frequency scale\n    \"beta_slow\": 1,                 # Low-frequency scale\n}\n```\n\n**Performance:**\n- Extends LLaMA to 128k tokens\n- 2.5× less training steps than baselines\n- State-of-the-art context window extension\n\n### 3. ALiBi (Attention with Linear Biases)\n\n**Core idea:**\n- No positional embeddings added to tokens\n- Apply distance penalty directly to attention scores\n- Bias proportional to key-query distance\n\n**Formula:**\n```\nattention_bias[i, j] = -m * |i - j|\n\nwhere m = slope for each attention head\n```\n\n**Advantages:**\n- 11% faster training vs sinusoidal embeddings\n- 11% less memory usage\n- Strong length extrapolation (train 1k, test 2k+)\n- Inductive bias towards recency\n\n### 4. Position Interpolation\n\n**Technique:**\n- Linearly down-scale position indices\n- Interpolate within trained range (vs extrapolate beyond)\n- Minimal fine-tuning required\n\n**Formula:**\n```\n# Original: position indices [0, 1, 2, ..., L]\n# Extended: position indices [0, 0.5, 1.0, ..., L\u002F2]\n# (for 2× extension)\n\nscaled_position[i] = i \u002F extension_factor\n```\n\n**Results:**\n- LLaMA 7B-65B extended to 32k tokens\n- 1000 fine-tuning steps sufficient\n- 600× better stability than extrapolation\n\n## Method Comparison\n\n| Method | Max Context | Training Needed | Memory | Extrapolation | Best For |\n|--------|-------------|-----------------|--------|---------------|----------|\n| **RoPE** | 8k-32k | Full pre-training | Moderate | Good | New models |\n| **YaRN** | 32k-128k | Minimal (10× efficient) | Moderate | Excellent | Extending existing models |\n| **ALiBi** | Unlimited | Full pre-training | Low (-11%) | Excellent | Training from scratch |\n| **Position Interpolation** | 32k+ | Minimal (1k steps) | Moderate | Poor (by design) | Quick extension |\n\n## Implementation Patterns\n\n### HuggingFace Transformers Integration\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoConfig\n\n# RoPE with YaRN scaling\nconfig = AutoConfig.from_pretrained(\"mistralai\u002FMistral-7B-v0.1\")\nconfig.rope_scaling = {\n    \"type\": \"yarn\",\n    \"factor\": 8.0,\n    \"original_max_position_embeddings\": 8192,\n    \"attention_factor\": 1.0\n}\n\nmodel = AutoModelForCausalLM.from_config(config)\n\n# Position interpolation (simpler)\nconfig.rope_scaling = {\n    \"type\": \"linear\",\n    \"factor\": 4.0\n}\n\n# Dynamic scaling (adjusts based on input length)\nconfig.rope_scaling = {\n    \"type\": \"dynamic\",\n    \"factor\": 8.0\n}\n```\n\n### Custom RoPE Implementation\n\n```python\nclass LongContextAttention(nn.Module):\n    \"\"\"Multi-head attention with RoPE.\"\"\"\n\n    def __init__(self, hidden_size, num_heads, max_seq_len=32768):\n        super().__init__()\n        self.num_heads = num_heads\n        self.head_dim = hidden_size \u002F\u002F num_heads\n\n        # Q, K, V projections\n        self.q_proj = nn.Linear(hidden_size, hidden_size)\n        self.k_proj = nn.Linear(hidden_size, hidden_size)\n        self.v_proj = nn.Linear(hidden_size, hidden_size)\n        self.o_proj = nn.Linear(hidden_size, hidden_size)\n\n        # RoPE\n        self.rotary_emb = RotaryEmbedding(\n            dim=self.head_dim,\n            max_seq_len=max_seq_len\n        )\n\n    def forward(self, hidden_states):\n        batch_size, seq_len, _ = hidden_states.shape\n\n        # Project to Q, K, V\n        q = self.q_proj(hidden_states)\n        k = self.k_proj(hidden_states)\n        v = self.v_proj(hidden_states)\n\n        # Reshape for multi-head\n        q = q.view(batch_size, seq_len, self.num_heads, self.head_dim).transpose(1, 2)\n        k = k.view(batch_size, seq_len, self.num_heads, self.head_dim).transpose(1, 2)\n        v = v.view(batch_size, seq_len, self.num_heads, self.head_dim).transpose(1, 2)\n\n        # Apply RoPE\n        cos, sin = self.rotary_emb(seq_len, device=hidden_states.device)\n        q, k = apply_rotary_pos_emb(q, k, cos, sin)\n\n        # Standard attention\n        attn_output = F.scaled_dot_product_attention(q, k, v)\n\n        # Reshape and project\n        attn_output = attn_output.transpose(1, 2).contiguous()\n        attn_output = attn_output.view(batch_size, seq_len, -1)\n        output = self.o_proj(attn_output)\n\n        return output\n```\n\n## Fine-tuning for Long Context\n\n### Minimal Fine-tuning (Position Interpolation)\n\n```python\nfrom transformers import Trainer, TrainingArguments\n\n# Extend model config\nmodel.config.max_position_embeddings = 32768\nmodel.config.rope_scaling = {\"type\": \"linear\", \"factor\": 16.0}\n\n# Training args (minimal steps needed)\ntraining_args = TrainingArguments(\n    output_dir=\".\u002Fllama-32k\",\n    num_train_epochs=1,\n    max_steps=1000,           # Only 1000 steps!\n    per_device_train_batch_size=1,\n    gradient_accumulation_steps=16,\n    learning_rate=2e-5,\n    warmup_steps=100,\n    logging_steps=10,\n    save_steps=500,\n)\n\n# Train on long documents\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=long_document_dataset,  # 32k token sequences\n)\n\ntrainer.train()\n```\n\n### YaRN Fine-tuning\n\n```bash\n# Clone YaRN implementation\ngit clone https:\u002F\u002Fgithub.com\u002Fjquesnelle\u002Fyarn\ncd yarn\n\n# Fine-tune LLaMA with YaRN\npython scripts\u002Ftrain.py \\\n    --model meta-llama\u002FLlama-2-7b-hf \\\n    --scale 16 \\\n    --rope_theta 10000 \\\n    --max_length 32768 \\\n    --batch_size 1 \\\n    --gradient_accumulation 16 \\\n    --steps 400 \\\n    --learning_rate 2e-5\n```\n\n## Best Practices\n\n### 1. Choose the Right Method\n\n```python\n# For NEW models (training from scratch)\nuse_method = \"ALiBi\"  # Best extrapolation, lowest memory\n\n# For EXTENDING existing RoPE models\nuse_method = \"YaRN\"  # Most efficient extension (10× less data)\n\n# For QUICK extension with minimal compute\nuse_method = \"Position Interpolation\"  # 1000 steps\n\n# For MODERATE extension with good efficiency\nuse_method = \"Linear RoPE Scaling\"  # Built-in, simple\n```\n\n### 2. Scaling Factor Selection\n\n```python\n# Conservative (safer, better quality)\nscaling_factor = 2.0  # 8k → 16k\n\n# Moderate (good balance)\nscaling_factor = 4.0  # 8k → 32k\n\n# Aggressive (requires more fine-tuning)\nscaling_factor = 8.0  # 8k → 64k\nscaling_factor = 16.0  # 8k → 128k\n\n# Rule: Larger factors need more fine-tuning steps\nsteps_needed = 100 * scaling_factor  # Rough estimate\n```\n\n### 3. Fine-tuning Data\n\n```python\n# ✅ Good: Long documents matching target length\ntrain_data = [\n    {\"text\": long_doc_32k_tokens},  # Full 32k\n    {\"text\": long_doc_24k_tokens},  # Varied lengths\n    {\"text\": long_doc_16k_tokens},\n]\n\n# ❌ Bad: Short documents (won't learn long context)\ntrain_data = [\n    {\"text\": short_doc_2k_tokens},\n]\n\n# Use datasets like:\n# - PG-19 (books, long texts)\n# - arXiv papers\n# - Long-form conversations\n# - GitHub repositories (concatenated files)\n```\n\n### 4. Avoid Common Pitfalls\n\n```python\n# ❌ Bad: Applying position interpolation without fine-tuning\nmodel.config.rope_scaling = {\"type\": \"linear\", \"factor\": 16.0}\n# Model will perform poorly without fine-tuning!\n\n# ✅ Good: Fine-tune after scaling\nmodel.config.rope_scaling = {\"type\": \"linear\", \"factor\": 16.0}\nfine_tune(model, long_documents, steps=1000)\n\n# ❌ Bad: Too aggressive scaling without data\nscale_to_1M_tokens()  # Won't work without massive fine-tuning\n\n# ✅ Good: Incremental scaling\n# 8k → 16k → 32k → 64k (fine-tune at each step)\n```\n\n## Production Deployment\n\n### Inference with Long Context\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load long-context model\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"togethercomputer\u002FLLaMA-2-7B-32K\",  # 32k context\n    torch_dtype=torch.float16,\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"togethercomputer\u002FLLaMA-2-7B-32K\")\n\n# Process long document\nlong_text = \"...\" * 30000  # 30k tokens\ninputs = tokenizer(long_text, return_tensors=\"pt\", truncation=False).to('cuda')\n\n# Generate\noutputs = model.generate(\n    **inputs,\n    max_new_tokens=512,\n    temperature=0.7,\n)\n\nresponse = tokenizer.decode(outputs[0], skip_special_tokens=True)\n```\n\n### Memory Optimization\n\n```python\n# Use gradient checkpointing for fine-tuning\nmodel.gradient_checkpointing_enable()\n\n# Use Flash Attention 2\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    attn_implementation=\"flash_attention_2\",  # 2-3× faster\n    torch_dtype=torch.float16\n)\n\n# Use paged attention (vLLM)\nfrom vllm import LLM\n\nllm = LLM(\n    model=\"togethercomputer\u002FLLaMA-2-7B-32K\",\n    max_model_len=32768,  # 32k context\n    gpu_memory_utilization=0.9\n)\n```\n\n## Resources\n\n- **RoPE Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2104.09864 (RoFormer)\n- **YaRN Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2309.00071\n- **ALiBi Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2108.12409 (Train Short, Test Long)\n- **Position Interpolation**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2306.15595\n- **HuggingFace RoPE Utils**: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftransformers\u002Fblob\u002Fmain\u002Fsrc\u002Ftransformers\u002Fmodeling_rope_utils.py\n- **YaRN Implementation**: https:\u002F\u002Fgithub.com\u002Fjquesnelle\u002Fyarn\n- **Together AI Blog**: https:\u002F\u002Fwww.together.ai\u002Fblog\u002Fllama-2-7b-32k\n\n## See Also\n\n- `references\u002Frope.md` - Detailed RoPE implementation and theory\n- `references\u002Fextension_methods.md` - YaRN, ALiBi, Position Interpolation comparisons\n- `references\u002Ffine_tuning.md` - Complete fine-tuning guide for context extension\n\n\n",{"name":5,"tags":48,"author":59,"license":14,"version":60,"description":7,"dependencies":61},[31,49,50,51,52,53,54,55,56,57,58],"Long Context","RoPE","YaRN","ALiBi","Position Interpolation","Extended Context","Rotary Embeddings","Attention Bias","Context Extension","Positional Encoding","Orchestra Research","1.0.0",[62,63,64],"transformers","torch","flash-attn","main","f19b254e8975d0bfddb394686706106930d7964d","AI-Models-and-Architecture\u002Flong-context\u002Flong-context","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus","https:\u002F\u002Fgithub.com\u002Fendman100","2026-11-11 18:19:10",{"source":8,"skillName":5,"commands":72,"prompt":82,"zipAvailable":83},[73,76,79],{"tool":74,"command":75},"npx","npx skills add endman100\u002Fskill-router-nexus --skill long-context",{"tool":77,"command":78},"bunx","bunx skills add endman100\u002Fskill-router-nexus --skill long-context",{"tool":80,"command":81},"pnpm","pnpm dlx skills add endman100\u002Fskill-router-nexus --skill long-context","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Models-and-Architecture\u002Flong-context\u002Flong-context\u002FSKILL.md",true,[85,91,95],{"path":86,"resourceType":87,"resourceTypeDesc":88,"sizeBytes":89,"sourceHtmlUrl":90},"references\u002Fextension_methods.md","REFERENCE","参考文档",14537,"https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Models-and-Architecture\u002Flong-context\u002Flong-context\u002Freferences\u002Fextension_methods.md",{"path":92,"resourceType":87,"resourceTypeDesc":88,"sizeBytes":93,"sourceHtmlUrl":94},"references\u002Ffine_tuning.md",16431,"https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Models-and-Architecture\u002Flong-context\u002Flong-context\u002Freferences\u002Ffine_tuning.md",{"path":96,"resourceType":87,"resourceTypeDesc":88,"sizeBytes":97,"sourceHtmlUrl":98},"references\u002Frope.md",12121,"https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Models-and-Architecture\u002Flong-context\u002Flong-context\u002Freferences\u002Frope.md",[100,124,161,183,192,216],{"skillNo":101,"name":102,"nameZh":6,"description":103,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":104,"sourceHtmlUrl":105,"categories":106,"keywords":112,"publishedAt":43},"01M6MPCT5BXVER8HQZTW3QE9AS","create-master","基于佛教经典文献，生成特定高僧大德的 AI 教学角色","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FMetaphysics-and-Spirituality\u002FMaster-skill\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FMetaphysics-and-Spirituality\u002FMaster-skill\u002FSKILL.md",[107],{"categoryNo":108,"categoryName":109,"taxonomyType":110,"taxonomyTypeDesc":111},"01JBSEEDCAT000000000000039","教师与教育","PROFESSION","适用职业",[113,114,115,116,117,118,119,120,121,122,123],"create master","master","create","生成","true","webfetch","edit","write","read","grep","Bash",{"skillNo":125,"name":126,"nameZh":6,"description":127,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":128,"sourceHtmlUrl":129,"categories":130,"keywords":142,"publishedAt":43},"01M6M43AKE95Y64E1JRXEF5QQ3","writing-dna-skill","从至少 20 篇完整文章中蒸馏可复用的写作 DNA，分析语言、文章结构、选题逻辑、素材策略、认知框架和视觉风格，并生成 Writing-DNA.md；按该 DNA 写作时会读取全部蒸馏产物和 5 篇相关原文。用于中英文作者、账号、品牌或出版物的风格分析与一致性写作。 Distill reusable Writing DNA from at least 20 complete articles for Chinese or English authors, publications, brands, and accounts; use for language, structure, topic logic, source strategy, cognitive-frame, and visual-style analysis.","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fwriting-dna-skill\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fwriting-dna-skill\u002FSKILL.md",[131,136,139],{"categoryNo":132,"categoryName":133,"taxonomyType":134,"taxonomyTypeDesc":135},"01JBSEEDCAT000000000000018","写作与文案","SCENARIO","使用场景",{"categoryNo":137,"categoryName":138,"taxonomyType":134,"taxonomyTypeDesc":135},"01JBSEEDCAT000000000000023","Skill与智能体开发",{"categoryNo":140,"categoryName":141,"taxonomyType":110,"taxonomyTypeDesc":111},"01JBSEEDCAT000000000000036","内容创作者",[143,144,145,146,147,148,149,150,151,152,153,154,155,116,156,157,158,27,159,160],"writing dna skill","DNA","writing","strategy","english","logic","visual","structure","chinese","reusable","use","style","distill","analysis","frame","source","at","complete",{"skillNo":162,"name":163,"nameZh":6,"description":164,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":165,"sourceHtmlUrl":166,"categories":167,"keywords":170,"publishedAt":43},"01M6M428TTCGQ4ME3KY99VN037","stop-slop","Remove AI writing patterns from prose. Use when drafting, editing, or reviewing text to eliminate predictable AI tells.","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fstop-slop\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fstop-slop\u002FSKILL.md",[168,169],{"categoryNo":132,"categoryName":133,"taxonomyType":134,"taxonomyTypeDesc":135},{"categoryNo":140,"categoryName":141,"taxonomyType":110,"taxonomyTypeDesc":111},[171,172,173,174,23,175,176,177,178,153,179,180,181,27,182,145],"slop","stop slop","stop","drafting","drafts","https","patterns","prose","remove","editing","text","content",{"skillNo":184,"name":185,"nameZh":6,"description":186,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":187,"sourceHtmlUrl":188,"categories":189,"keywords":190,"publishedAt":43},"01M6M41X0B7EBX42TK435DMGGR","shuorenhua","检查和清理中英文文本里的 AI 套路，适用于“去 AI 味”“说人话”“自然一点”“别像模板”“先标问题”这类改写和审稿需求；按场景控制力度，同时保留事实、术语、语域和责任主体。","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fshuorenhua\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fshuorenhua\u002FSKILL.md",[],[185,191],"文本",{"skillNo":193,"name":194,"nameZh":6,"description":195,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":196,"sourceHtmlUrl":197,"categories":198,"keywords":199,"publishedAt":43},"01M6M3ZD12WBEW5KXKDM4N6DFG","naming","Name products, SaaS, brands, open source projects, bots, and apps. Use when the user needs to name something, find a brand name, or pick a product name. Metaphor-driven process that produces memorable, meaningful names and avoids AI slop.","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fnaming-main\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fnaming-main\u002FSKILL.md",[],[194,200,201,202,203,204,153,205,121,158,206,207,208,209,123,210,211,212,213,23,214,215,171],"repo","websearch","product name","apps","bots","view","brand","pick","gh","projects","open","saas","open source","product","describe","ai slop",{"skillNo":217,"name":218,"nameZh":6,"description":219,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":220,"sourceHtmlUrl":221,"categories":222,"keywords":224,"publishedAt":43},"01M6M3YVXAFAAWJ7F6JMPFREYE","humanizer-zh","去除文本中的 AI 生成痕迹。适用于编辑或审阅文本，使其听起来更自然、更像人类书写。\n基于维基百科的\"AI 写作特征\"综合指南。检测并修复以下模式：夸大的象征意义、\n宣传性语言、以 -ing 结尾的肤浅分析、模糊的归因、破折号过度使用、三段式法则、\nAI 词汇、否定式排比、过多的连接性短语。","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fhumanizer-zh\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fhumanizer-zh\u002FSKILL.md",[223],{"categoryNo":132,"categoryName":133,"taxonomyType":134,"taxonomyTypeDesc":135},[225,226,227,121,120,119,145,116,191,228,171,173],"humanizer","zh","humanizer zh","翻译"]