[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6XG9MWSNEDBTRZ39MDH4K00":3,"skill-same:01M6XG9MWSNEDBTRZ39MDH4K00":86},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":56,"commitSha":57,"skillDirectory":58,"forksCount":14,"repositoryHtmlUrl":59,"repositoryHomepageUrl":6,"ownerHtmlUrl":60,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":61,"install":62,"resources":75},"01M6XG9MWSNEDBTRZ39MDH4K00","optimizing-attention-flash",null,"Optimizes transformer attention with Flash Attention for 2-4x speedup and 10-20x memory reduction. Use when training\u002Frunning transformers with long sequences (>512 tokens), encountering GPU memory issues with attention, or need faster inference. Supports PyTorch native SDPA, flash-attn library, H100 FP8, and sliding window attention.","fadrienne\u002FMyGitHub","fadrienne","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F197874571?v=4","01M6X64V8VWCRSMFZ4EPS2BWXG","Francesca Adrienne","01M6X64VD5MW4F9PHDWQKS7ZAF",0,"MIT","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fopenscience\u002Fbackend\u002Fcli\u002Fskills\u002Fml-training\u002Fflash-attention\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"flash","optimizing attention flash","attention","Flash Attention","sequences","when","window","native","H100","SDPA","Speed Optimization","Memory Efficiency","Attention Optimization","memory","synthetic","efficiency","Transformers","issues","training","pytorch","2026-11-10 10:12:30",false,"---\nname: optimizing-attention-flash\ndescription: Optimizes transformer attention with Flash Attention for 2-4x speedup and 10-20x memory reduction. Use when training\u002Frunning transformers with long sequences (>512 tokens), encountering GPU memory issues with attention, or need faster inference. Supports PyTorch native SDPA, flash-attn library, H100 FP8, and sliding window attention.\ncategory: ml-training\nversion: 1.0.0\nauthor: Synthetic Sciences\nlicense: MIT\ntags: [Optimization, Flash Attention, Attention Optimization, Memory Efficiency, Speed Optimization, Long Context, PyTorch, SDPA, H100, FP8, Transformers]\ndependencies: [flash-attn, torch, transformers]\n---\n\n# Flash Attention - Fast Memory-Efficient Attention\n\n## Quick start\n\nFlash Attention provides 2-4x speedup and 10-20x memory reduction for transformer attention through IO-aware tiling and recomputation.\n\n**PyTorch native (easiest, PyTorch 2.2+)**:\n```python\nimport torch\nimport torch.nn.functional as F\n\nq = torch.randn(2, 8, 512, 64, device='cuda', dtype=torch.float16)  # [batch, heads, seq, dim]\nk = torch.randn(2, 8, 512, 64, device='cuda', dtype=torch.float16)\nv = torch.randn(2, 8, 512, 64, device='cuda', dtype=torch.float16)\n\n# Automatically uses Flash Attention if available\nout = F.scaled_dot_product_attention(q, k, v)\n```\n\n**flash-attn library (more features)**:\n```bash\npip install flash-attn --no-build-isolation\n```\n\n```python\nfrom flash_attn import flash_attn_func\n\n# q, k, v: [batch, seqlen, nheads, headdim]\nout = flash_attn_func(q, k, v, dropout_p=0.0, causal=True)\n```\n\n## Common workflows\n\n### Workflow 1: Enable in existing PyTorch model\n\nCopy this checklist:\n\n```\nFlash Attention Integration:\n- [ ] Step 1: Check PyTorch version (≥2.2)\n- [ ] Step 2: Enable Flash Attention backend\n- [ ] Step 3: Verify speedup with profiling\n- [ ] Step 4: Test accuracy matches baseline\n```\n\n**Step 1: Check PyTorch version**\n\n```bash\npython -c \"import torch; print(torch.__version__)\"\n# Should be ≥2.2.0\n```\n\nIf \u003C2.2, upgrade:\n```bash\npip install --upgrade torch\n```\n\n**Step 2: Enable Flash Attention backend**\n\nReplace standard attention:\n```python\n# Before (standard attention)\nattn_weights = torch.softmax(q @ k.transpose(-2, -1) \u002F math.sqrt(d_k), dim=-1)\nout = attn_weights @ v\n\n# After (Flash Attention)\nimport torch.nn.functional as F\nout = F.scaled_dot_product_attention(q, k, v, attn_mask=mask)\n```\n\nForce Flash Attention backend:\n```python\nwith torch.backends.cuda.sdp_kernel(\n    enable_flash=True,\n    enable_math=False,\n    enable_mem_efficient=False\n):\n    out = F.scaled_dot_product_attention(q, k, v)\n```\n\n**Step 3: Verify speedup with profiling**\n\n```python\nimport torch.utils.benchmark as benchmark\n\ndef test_attention(use_flash):\n    q, k, v = [torch.randn(2, 8, 2048, 64, device='cuda', dtype=torch.float16) for _ in range(3)]\n\n    if use_flash:\n        with torch.backends.cuda.sdp_kernel(enable_flash=True):\n            return F.scaled_dot_product_attention(q, k, v)\n    else:\n        attn = (q @ k.transpose(-2, -1) \u002F 8.0).softmax(dim=-1)\n        return attn @ v\n\n# Benchmark\nt_flash = benchmark.Timer(stmt='test_attention(True)', globals=globals())\nt_standard = benchmark.Timer(stmt='test_attention(False)', globals=globals())\n\nprint(f\"Flash: {t_flash.timeit(100).mean:.3f}s\")\nprint(f\"Standard: {t_standard.timeit(100).mean:.3f}s\")\n```\n\nExpected: 2-4x speedup for sequences >512 tokens.\n\n**Step 4: Test accuracy matches baseline**\n\n```python\n# Compare outputs\nq, k, v = [torch.randn(1, 8, 512, 64, device='cuda', dtype=torch.float16) for _ in range(3)]\n\n# Flash Attention\nout_flash = F.scaled_dot_product_attention(q, k, v)\n\n# Standard attention\nattn_weights = torch.softmax(q @ k.transpose(-2, -1) \u002F 8.0, dim=-1)\nout_standard = attn_weights @ v\n\n# Check difference\ndiff = (out_flash - out_standard).abs().max()\nprint(f\"Max difference: {diff:.6f}\")\n# Should be \u003C1e-3 for float16\n```\n\n### Workflow 2: Use flash-attn library for advanced features\n\nFor multi-query attention, sliding window, or H100 FP8.\n\nCopy this checklist:\n\n```\nflash-attn Library Setup:\n- [ ] Step 1: Install flash-attn library\n- [ ] Step 2: Modify attention code\n- [ ] Step 3: Enable advanced features\n- [ ] Step 4: Benchmark performance\n```\n\n**Step 1: Install flash-attn library**\n\n```bash\n# NVIDIA GPUs (CUDA 12.0+)\npip install flash-attn --no-build-isolation\n\n# Verify installation\npython -c \"from flash_attn import flash_attn_func; print('Success')\"\n```\n\n**Step 2: Modify attention code**\n\n```python\nfrom flash_attn import flash_attn_func\n\n# Input: [batch_size, seq_len, num_heads, head_dim]\n# Transpose from [batch, heads, seq, dim] if needed\nq = q.transpose(1, 2)  # [batch, seq, heads, dim]\nk = k.transpose(1, 2)\nv = v.transpose(1, 2)\n\nout = flash_attn_func(\n    q, k, v,\n    dropout_p=0.1,\n    causal=True,  # For autoregressive models\n    window_size=(-1, -1),  # No sliding window\n    softmax_scale=None  # Auto-scale\n)\n\nout = out.transpose(1, 2)  # Back to [batch, heads, seq, dim]\n```\n\n**Step 3: Enable advanced features**\n\nMulti-query attention (shared K\u002FV across heads):\n```python\nfrom flash_attn import flash_attn_func\n\n# q: [batch, seq, num_q_heads, dim]\n# k, v: [batch, seq, num_kv_heads, dim]  # Fewer KV heads\nout = flash_attn_func(q, k, v)  # Automatically handles MQA\n```\n\nSliding window attention (local attention):\n```python\n# Only attend to window of 256 tokens before\u002Fafter\nout = flash_attn_func(\n    q, k, v,\n    window_size=(256, 256),  # (left, right) window\n    causal=True\n)\n```\n\n**Step 4: Benchmark performance**\n\n```python\nimport torch\nfrom flash_attn import flash_attn_func\nimport time\n\nq, k, v = [torch.randn(4, 4096, 32, 64, device='cuda', dtype=torch.float16) for _ in range(3)]\n\n# Warmup\nfor _ in range(10):\n    _ = flash_attn_func(q, k, v)\n\n# Benchmark\ntorch.cuda.synchronize()\nstart = time.time()\nfor _ in range(100):\n    out = flash_attn_func(q, k, v)\n    torch.cuda.synchronize()\nend = time.time()\n\nprint(f\"Time per iteration: {(end-start)\u002F100*1000:.2f}ms\")\nprint(f\"Memory allocated: {torch.cuda.max_memory_allocated()\u002F1e9:.2f}GB\")\n```\n\n### Workflow 3: H100 FP8 optimization (FlashAttention-3)\n\nFor maximum performance on H100 GPUs.\n\n```\nFP8 Setup:\n- [ ] Step 1: Verify H100 GPU available\n- [ ] Step 2: Install flash-attn with FP8 support\n- [ ] Step 3: Convert inputs to FP8\n- [ ] Step 4: Run with FP8 attention\n```\n\n**Step 1: Verify H100 GPU**\n\n```bash\nnvidia-smi --query-gpu=name --format=csv\n# Should show \"H100\" or \"H800\"\n```\n\n**Step 2: Install flash-attn with FP8 support**\n\n```bash\npip install flash-attn --no-build-isolation\n# FP8 support included for H100\n```\n\n**Step 3: Convert inputs to FP8**\n\n```python\nimport torch\n\nq = torch.randn(2, 4096, 32, 64, device='cuda', dtype=torch.float16)\nk = torch.randn(2, 4096, 32, 64, device='cuda', dtype=torch.float16)\nv = torch.randn(2, 4096, 32, 64, device='cuda', dtype=torch.float16)\n\n# Convert to float8_e4m3 (FP8)\nq_fp8 = q.to(torch.float8_e4m3fn)\nk_fp8 = k.to(torch.float8_e4m3fn)\nv_fp8 = v.to(torch.float8_e4m3fn)\n```\n\n**Step 4: Run with FP8 attention**\n\n```python\nfrom flash_attn import flash_attn_func\n\n# FlashAttention-3 automatically uses FP8 kernels on H100\nout = flash_attn_func(q_fp8, k_fp8, v_fp8)\n# Result: ~1.2 PFLOPS, 1.5-2x faster than FP16\n```\n\n## When to use vs alternatives\n\n**Use Flash Attention when:**\n- Training transformers with sequences >512 tokens\n- Running inference with long context (>2K tokens)\n- GPU memory constrained (OOM with standard attention)\n- Need 2-4x speedup without accuracy loss\n- Using PyTorch 2.2+ or can install flash-attn\n\n**Use alternatives instead:**\n- **Standard attention**: Sequences \u003C256 tokens (overhead not worth it)\n- **xFormers**: Need more attention variants (not just speed)\n- **Memory-efficient attention**: CPU inference (Flash Attention needs GPU)\n\n## Common issues\n\n**Issue: ImportError: cannot import flash_attn**\n\nInstall with no-build-isolation flag:\n```bash\npip install flash-attn --no-build-isolation\n```\n\nOr install CUDA toolkit first:\n```bash\nconda install cuda -c nvidia\npip install flash-attn --no-build-isolation\n```\n\n**Issue: Slower than expected (no speedup)**\n\nFlash Attention benefits increase with sequence length:\n- \u003C512 tokens: Minimal speedup (10-20%)\n- 512-2K tokens: 2-3x speedup\n- >2K tokens: 3-4x speedup\n\nCheck sequence length is sufficient.\n\n**Issue: RuntimeError: CUDA error**\n\nVerify GPU supports Flash Attention:\n```python\nimport torch\nprint(torch.cuda.get_device_capability())\n# Should be ≥(7, 5) for Turing+\n```\n\nFlash Attention requires:\n- Ampere (A100, A10): ✅ Full support\n- Turing (T4): ✅ Supported\n- Volta (V100): ❌ Not supported\n\n**Issue: Accuracy degradation**\n\nCheck dtype is float16 or bfloat16 (not float32):\n```python\nq = q.to(torch.float16)  # Or torch.bfloat16\n```\n\nFlash Attention uses float16\u002Fbfloat16 for speed. Float32 not supported.\n\n## Advanced topics\n\n**Integration with HuggingFace Transformers**: See [references\u002Ftransformers-integration.md](references\u002Ftransformers-integration.md) for enabling Flash Attention in BERT, GPT, Llama models.\n\n**Performance benchmarks**: See [references\u002Fbenchmarks.md](references\u002Fbenchmarks.md) for detailed speed and memory comparisons across GPUs and sequence lengths.\n\n**Algorithm details**: See [references\u002Falgorithm.md](references\u002Falgorithm.md) for tiling strategy, recomputation, and IO complexity analysis.\n\n**Advanced features**: See [references\u002Fadvanced-features.md](references\u002Fadvanced-features.md) for rotary embeddings, ALiBi, paged KV cache, and custom attention masks.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA Ampere+ (A100, A10, A30) or AMD MI200+\n- **VRAM**: Same as standard attention (Flash Attention doesn't increase memory)\n- **CUDA**: 12.0+ (11.8 minimum)\n- **PyTorch**: 2.2+ for native support\n\n**Not supported**: V100 (Volta), CPU inference\n\n\n## Known Conflicts\n\n- **Do not install alongside `mamba`** in the same environment. Both compile custom CUDA kernels (flash-attn vs causal-conv1d) that may conflict. Use separate virtual environments or Modal containers.\n\n## Resources\n\n- Paper: \"FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness\" (NeurIPS 2022)\n- Paper: \"FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning\" (ICLR 2024)\n- Blog: https:\u002F\u002Ftridao.me\u002Fblog\u002F2024\u002Fflash3\u002F\n- GitHub: https:\u002F\u002Fgithub.com\u002FDao-AILab\u002Fflash-attention\n- PyTorch docs: https:\u002F\u002Fpytorch.org\u002Fdocs\u002Fstable\u002Fgenerated\u002Ftorch.nn.functional.scaled_dot_product_attention.html\n\n\n\n","# Flash Attention - Fast Memory-Efficient Attention\n\n## Quick start\n\nFlash Attention provides 2-4x speedup and 10-20x memory reduction for transformer attention through IO-aware tiling and recomputation.\n\n**PyTorch native (easiest, PyTorch 2.2+)**:\n```python\nimport torch\nimport torch.nn.functional as F\n\nq = torch.randn(2, 8, 512, 64, device='cuda', dtype=torch.float16)  # [batch, heads, seq, dim]\nk = torch.randn(2, 8, 512, 64, device='cuda', dtype=torch.float16)\nv = torch.randn(2, 8, 512, 64, device='cuda', dtype=torch.float16)\n\n# Automatically uses Flash Attention if available\nout = F.scaled_dot_product_attention(q, k, v)\n```\n\n**flash-attn library (more features)**:\n```bash\npip install flash-attn --no-build-isolation\n```\n\n```python\nfrom flash_attn import flash_attn_func\n\n# q, k, v: [batch, seqlen, nheads, headdim]\nout = flash_attn_func(q, k, v, dropout_p=0.0, causal=True)\n```\n\n## Common workflows\n\n### Workflow 1: Enable in existing PyTorch model\n\nCopy this checklist:\n\n```\nFlash Attention Integration:\n- [ ] Step 1: Check PyTorch version (≥2.2)\n- [ ] Step 2: Enable Flash Attention backend\n- [ ] Step 3: Verify speedup with profiling\n- [ ] Step 4: Test accuracy matches baseline\n```\n\n**Step 1: Check PyTorch version**\n\n```bash\npython -c \"import torch; print(torch.__version__)\"\n# Should be ≥2.2.0\n```\n\nIf \u003C2.2, upgrade:\n```bash\npip install --upgrade torch\n```\n\n**Step 2: Enable Flash Attention backend**\n\nReplace standard attention:\n```python\n# Before (standard attention)\nattn_weights = torch.softmax(q @ k.transpose(-2, -1) \u002F math.sqrt(d_k), dim=-1)\nout = attn_weights @ v\n\n# After (Flash Attention)\nimport torch.nn.functional as F\nout = F.scaled_dot_product_attention(q, k, v, attn_mask=mask)\n```\n\nForce Flash Attention backend:\n```python\nwith torch.backends.cuda.sdp_kernel(\n    enable_flash=True,\n    enable_math=False,\n    enable_mem_efficient=False\n):\n    out = F.scaled_dot_product_attention(q, k, v)\n```\n\n**Step 3: Verify speedup with profiling**\n\n```python\nimport torch.utils.benchmark as benchmark\n\ndef test_attention(use_flash):\n    q, k, v = [torch.randn(2, 8, 2048, 64, device='cuda', dtype=torch.float16) for _ in range(3)]\n\n    if use_flash:\n        with torch.backends.cuda.sdp_kernel(enable_flash=True):\n            return F.scaled_dot_product_attention(q, k, v)\n    else:\n        attn = (q @ k.transpose(-2, -1) \u002F 8.0).softmax(dim=-1)\n        return attn @ v\n\n# Benchmark\nt_flash = benchmark.Timer(stmt='test_attention(True)', globals=globals())\nt_standard = benchmark.Timer(stmt='test_attention(False)', globals=globals())\n\nprint(f\"Flash: {t_flash.timeit(100).mean:.3f}s\")\nprint(f\"Standard: {t_standard.timeit(100).mean:.3f}s\")\n```\n\nExpected: 2-4x speedup for sequences >512 tokens.\n\n**Step 4: Test accuracy matches baseline**\n\n```python\n# Compare outputs\nq, k, v = [torch.randn(1, 8, 512, 64, device='cuda', dtype=torch.float16) for _ in range(3)]\n\n# Flash Attention\nout_flash = F.scaled_dot_product_attention(q, k, v)\n\n# Standard attention\nattn_weights = torch.softmax(q @ k.transpose(-2, -1) \u002F 8.0, dim=-1)\nout_standard = attn_weights @ v\n\n# Check difference\ndiff = (out_flash - out_standard).abs().max()\nprint(f\"Max difference: {diff:.6f}\")\n# Should be \u003C1e-3 for float16\n```\n\n### Workflow 2: Use flash-attn library for advanced features\n\nFor multi-query attention, sliding window, or H100 FP8.\n\nCopy this checklist:\n\n```\nflash-attn Library Setup:\n- [ ] Step 1: Install flash-attn library\n- [ ] Step 2: Modify attention code\n- [ ] Step 3: Enable advanced features\n- [ ] Step 4: Benchmark performance\n```\n\n**Step 1: Install flash-attn library**\n\n```bash\n# NVIDIA GPUs (CUDA 12.0+)\npip install flash-attn --no-build-isolation\n\n# Verify installation\npython -c \"from flash_attn import flash_attn_func; print('Success')\"\n```\n\n**Step 2: Modify attention code**\n\n```python\nfrom flash_attn import flash_attn_func\n\n# Input: [batch_size, seq_len, num_heads, head_dim]\n# Transpose from [batch, heads, seq, dim] if needed\nq = q.transpose(1, 2)  # [batch, seq, heads, dim]\nk = k.transpose(1, 2)\nv = v.transpose(1, 2)\n\nout = flash_attn_func(\n    q, k, v,\n    dropout_p=0.1,\n    causal=True,  # For autoregressive models\n    window_size=(-1, -1),  # No sliding window\n    softmax_scale=None  # Auto-scale\n)\n\nout = out.transpose(1, 2)  # Back to [batch, heads, seq, dim]\n```\n\n**Step 3: Enable advanced features**\n\nMulti-query attention (shared K\u002FV across heads):\n```python\nfrom flash_attn import flash_attn_func\n\n# q: [batch, seq, num_q_heads, dim]\n# k, v: [batch, seq, num_kv_heads, dim]  # Fewer KV heads\nout = flash_attn_func(q, k, v)  # Automatically handles MQA\n```\n\nSliding window attention (local attention):\n```python\n# Only attend to window of 256 tokens before\u002Fafter\nout = flash_attn_func(\n    q, k, v,\n    window_size=(256, 256),  # (left, right) window\n    causal=True\n)\n```\n\n**Step 4: Benchmark performance**\n\n```python\nimport torch\nfrom flash_attn import flash_attn_func\nimport time\n\nq, k, v = [torch.randn(4, 4096, 32, 64, device='cuda', dtype=torch.float16) for _ in range(3)]\n\n# Warmup\nfor _ in range(10):\n    _ = flash_attn_func(q, k, v)\n\n# Benchmark\ntorch.cuda.synchronize()\nstart = time.time()\nfor _ in range(100):\n    out = flash_attn_func(q, k, v)\n    torch.cuda.synchronize()\nend = time.time()\n\nprint(f\"Time per iteration: {(end-start)\u002F100*1000:.2f}ms\")\nprint(f\"Memory allocated: {torch.cuda.max_memory_allocated()\u002F1e9:.2f}GB\")\n```\n\n### Workflow 3: H100 FP8 optimization (FlashAttention-3)\n\nFor maximum performance on H100 GPUs.\n\n```\nFP8 Setup:\n- [ ] Step 1: Verify H100 GPU available\n- [ ] Step 2: Install flash-attn with FP8 support\n- [ ] Step 3: Convert inputs to FP8\n- [ ] Step 4: Run with FP8 attention\n```\n\n**Step 1: Verify H100 GPU**\n\n```bash\nnvidia-smi --query-gpu=name --format=csv\n# Should show \"H100\" or \"H800\"\n```\n\n**Step 2: Install flash-attn with FP8 support**\n\n```bash\npip install flash-attn --no-build-isolation\n# FP8 support included for H100\n```\n\n**Step 3: Convert inputs to FP8**\n\n```python\nimport torch\n\nq = torch.randn(2, 4096, 32, 64, device='cuda', dtype=torch.float16)\nk = torch.randn(2, 4096, 32, 64, device='cuda', dtype=torch.float16)\nv = torch.randn(2, 4096, 32, 64, device='cuda', dtype=torch.float16)\n\n# Convert to float8_e4m3 (FP8)\nq_fp8 = q.to(torch.float8_e4m3fn)\nk_fp8 = k.to(torch.float8_e4m3fn)\nv_fp8 = v.to(torch.float8_e4m3fn)\n```\n\n**Step 4: Run with FP8 attention**\n\n```python\nfrom flash_attn import flash_attn_func\n\n# FlashAttention-3 automatically uses FP8 kernels on H100\nout = flash_attn_func(q_fp8, k_fp8, v_fp8)\n# Result: ~1.2 PFLOPS, 1.5-2x faster than FP16\n```\n\n## When to use vs alternatives\n\n**Use Flash Attention when:**\n- Training transformers with sequences >512 tokens\n- Running inference with long context (>2K tokens)\n- GPU memory constrained (OOM with standard attention)\n- Need 2-4x speedup without accuracy loss\n- Using PyTorch 2.2+ or can install flash-attn\n\n**Use alternatives instead:**\n- **Standard attention**: Sequences \u003C256 tokens (overhead not worth it)\n- **xFormers**: Need more attention variants (not just speed)\n- **Memory-efficient attention**: CPU inference (Flash Attention needs GPU)\n\n## Common issues\n\n**Issue: ImportError: cannot import flash_attn**\n\nInstall with no-build-isolation flag:\n```bash\npip install flash-attn --no-build-isolation\n```\n\nOr install CUDA toolkit first:\n```bash\nconda install cuda -c nvidia\npip install flash-attn --no-build-isolation\n```\n\n**Issue: Slower than expected (no speedup)**\n\nFlash Attention benefits increase with sequence length:\n- \u003C512 tokens: Minimal speedup (10-20%)\n- 512-2K tokens: 2-3x speedup\n- >2K tokens: 3-4x speedup\n\nCheck sequence length is sufficient.\n\n**Issue: RuntimeError: CUDA error**\n\nVerify GPU supports Flash Attention:\n```python\nimport torch\nprint(torch.cuda.get_device_capability())\n# Should be ≥(7, 5) for Turing+\n```\n\nFlash Attention requires:\n- Ampere (A100, A10): ✅ Full support\n- Turing (T4): ✅ Supported\n- Volta (V100): ❌ Not supported\n\n**Issue: Accuracy degradation**\n\nCheck dtype is float16 or bfloat16 (not float32):\n```python\nq = q.to(torch.float16)  # Or torch.bfloat16\n```\n\nFlash Attention uses float16\u002Fbfloat16 for speed. Float32 not supported.\n\n## Advanced topics\n\n**Integration with HuggingFace Transformers**: See [references\u002Ftransformers-integration.md](references\u002Ftransformers-integration.md) for enabling Flash Attention in BERT, GPT, Llama models.\n\n**Performance benchmarks**: See [references\u002Fbenchmarks.md](references\u002Fbenchmarks.md) for detailed speed and memory comparisons across GPUs and sequence lengths.\n\n**Algorithm details**: See [references\u002Falgorithm.md](references\u002Falgorithm.md) for tiling strategy, recomputation, and IO complexity analysis.\n\n**Advanced features**: See [references\u002Fadvanced-features.md](references\u002Fadvanced-features.md) for rotary embeddings, ALiBi, paged KV cache, and custom attention masks.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA Ampere+ (A100, A10, A30) or AMD MI200+\n- **VRAM**: Same as standard attention (Flash Attention doesn't increase memory)\n- **CUDA**: 12.0+ (11.8 minimum)\n- **PyTorch**: 2.2+ for native support\n\n**Not supported**: V100 (Volta), CPU inference\n\n\n## Known Conflicts\n\n- **Do not install alongside `mamba`** in the same environment. Both compile custom CUDA kernels (flash-attn vs causal-conv1d) that may conflict. Use separate virtual environments or Modal containers.\n\n## Resources\n\n- Paper: \"FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness\" (NeurIPS 2022)\n- Paper: \"FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning\" (ICLR 2024)\n- Blog: https:\u002F\u002Ftridao.me\u002Fblog\u002F2024\u002Fflash3\u002F\n- GitHub: https:\u002F\u002Fgithub.com\u002FDao-AILab\u002Fflash-attention\n- PyTorch docs: https:\u002F\u002Fpytorch.org\u002Fdocs\u002Fstable\u002Fgenerated\u002Ftorch.nn.functional.scaled_dot_product_attention.html\n\n\n\n",{"name":5,"tags":44,"author":49,"license":15,"version":50,"category":51,"description":7,"dependencies":52},[45,22,31,30,29,46,47,28,27,48,35],"Optimization","Long Context","PyTorch","FP8","Synthetic Sciences","1.0.0","ml-training",[53,54,55],"flash-attn","torch","transformers","main","4aa39efd1cdab0b4657ecee16c92a92011a5b10f","openscience\u002Fbackend\u002Fcli\u002Fskills\u002Fml-training\u002Fflash-attention","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub","https:\u002F\u002Fgithub.com\u002Ffadrienne","2026-10-28 00:43:09",{"source":8,"skillName":5,"commands":63,"prompt":73,"zipAvailable":74},[64,67,70],{"tool":65,"command":66},"npx","npx skills add fadrienne\u002FMyGitHub --skill optimizing-attention-flash",{"tool":68,"command":69},"bunx","bunx skills add fadrienne\u002FMyGitHub --skill optimizing-attention-flash",{"tool":71,"command":72},"pnpm","pnpm dlx skills add fadrienne\u002FMyGitHub --skill optimizing-attention-flash","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fopenscience\u002Fbackend\u002Fcli\u002Fskills\u002Fml-training\u002Fflash-attention\u002FSKILL.md",true,[76,82],{"path":77,"resourceType":78,"resourceTypeDesc":79,"sizeBytes":80,"sourceHtmlUrl":81},"references\u002Fbenchmarks.md","REFERENCE","参考文档",7129,"https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fopenscience\u002Fbackend\u002Fcli\u002Fskills\u002Fml-training\u002Fflash-attention\u002Freferences\u002Fbenchmarks.md",{"path":83,"resourceType":78,"resourceTypeDesc":79,"sizeBytes":84,"sourceHtmlUrl":85},"references\u002Ftransformers-integration.md",7427,"https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fopenscience\u002Fbackend\u002Fcli\u002Fskills\u002Fml-training\u002Fflash-attention\u002Freferences\u002Ftransformers-integration.md",[87,118,145,179,206,232],{"skillNo":88,"name":89,"nameZh":6,"description":90,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":91,"categories":92,"keywords":98,"publishedAt":39},"01M7167B28PK20HP7R7ZED2DY0","statistical-power","Sample size calculation, power analysis, and effect size conversion for scientific studies. Use BEFORE data collection (a priori) to determine required N, or after (post-hoc) to compute achieved power. Covers parametric tests (t-test, ANOVA, chi-square, proportions, correlation, regression), simulation-based power for complex\u002Fclustered\u002Fnested designs, and power curves. Use this skill when someone asks 'how many samples do I need', 'what is my power', 'what effect size can I detect', or any variant. Pairs with \u002Fexperimental-design (choose design first, then compute power) and \u002Fstatistical-analysis (analyze data after collection).","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fscientific-agent-skills\u002Fskills\u002Fstatistical-power\u002FSKILL.md",[93],{"categoryNo":94,"categoryName":95,"taxonomyType":96,"taxonomyTypeDesc":97},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[99,100,101,102,103,104,105,106,24,107,108,109,110,111,112,113,114,115,116,117],"power","statistical power","effect","scipy","regression","choose","design","uv","Bash","collection","analysis","calculation","ANOVA","conversion","analyze","t-test","chi-square","simulation","sample",{"skillNo":119,"name":120,"nameZh":6,"description":121,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":122,"categories":123,"keywords":127,"publishedAt":39},"01M71677EBCM8EPCE9BZCK8505","statistical-analysis","Guided statistical analysis with test selection and reporting. Use when you need help choosing appropriate tests for your data, assumption checking, power analysis, and APA-formatted results. Best for academic research reporting, test selection guidance. For implementing specific models programmatically use statsmodels.","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fscientific-agent-skills\u002Fskills\u002Fstatistical-analysis\u002FSKILL.md",[124],{"categoryNo":125,"categoryName":126,"taxonomyType":96,"taxonomyTypeDesc":97},"01JBSEEDCAT000000000000008","测试与质量",[109,128,99,129,130,131,24,132,133,134,135,136,137,138,139,140,141,142,143,144],"statistical analysis","academic research","implementing","selection","you","reporting","data","for","tests","use","with","test","research","results","statsmodels","models","help",{"skillNo":146,"name":147,"nameZh":6,"description":148,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":149,"categories":150,"keywords":160,"publishedAt":39},"01M71673R93XAHFH81DBNVNG7H","scientific-writing","Core skill for the deep research and writing tool. Write scientific manuscripts in full paragraphs (never bullet points). Use two-stage process with (1) section outlines with key points using research-lookup then (2) convert to flowing prose. IMRAD structure, citations (APA\u002FAMA\u002FVancouver), figures\u002Ftables, reporting guidelines (CONSORT\u002FSTROBE\u002FPRISMA), for research papers and journal submissions.","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fscientific-agent-skills\u002Fskills\u002Fscientific-writing\u002FSKILL.md",[151,154,155],{"categoryNo":152,"categoryName":153,"taxonomyType":96,"taxonomyTypeDesc":97},"01JBSEEDCAT000000000000018","写作与文案",{"categoryNo":94,"categoryName":95,"taxonomyType":96,"taxonomyTypeDesc":97},{"categoryNo":156,"categoryName":157,"taxonomyType":158,"taxonomyTypeDesc":159},"01JBSEEDCAT000000000000036","内容创作者","PROFESSION","适用职业",[161,162,163,164,165,166,167,168,169,170,107,171,172,173,174,175,176,177,133,178],"writing","scientific","scientific writing","Citations","deep","deep research","never","papers","journal","false","api","prisma","structure","openrouter","Outlines","core skill","read","features",{"skillNo":180,"name":181,"nameZh":6,"description":182,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":183,"categories":184,"keywords":189,"publishedAt":39},"01M7166ZPKXBF4EBPHVTJPQBB1","scientific-visualization","Meta-skill for publication-ready figures. Use when creating journal submission figures requiring multi-panel layouts, significance annotations, error bars, colorblind-safe palettes, and specific journal formatting (Nature, Science, Cell). Orchestrates matplotlib\u002Fseaborn\u002Fplotly with publication styles. For quick exploration use seaborn or plotly directly.","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fscientific-agent-skills\u002Fskills\u002Fscientific-visualization\u002FSKILL.md",[185,188],{"categoryNo":186,"categoryName":187,"taxonomyType":96,"taxonomyTypeDesc":97},"01JBSEEDCAT000000000000013","分析与可视化",{"categoryNo":94,"categoryName":95,"taxonomyType":96,"taxonomyTypeDesc":97},[190,162,191,192,193,194,195,196,197,169,24,198,199,200,201,202,203,204,135,205],"visualization","scientific visualization","error","ready","quick","exploration","meta-skill","Meta","safe","plotly","annotations","formatting","Publication","figures","panel","submission",{"skillNo":207,"name":208,"nameZh":6,"description":209,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":210,"categories":211,"keywords":218,"publishedAt":39},"01M7166VXJWQSSC3RQZP60WBGR","research-code","Debug, profile, review, and restructure Python research and analysis code. Use when analysis scripts crash, produce wrong results, run too slowly, or need a correctness review. Covers: reading tracebacks and diagnosing root causes, hypothesis-driven debugging, reviewing code for common statistical\u002Fnumerical errors, profiling data pipelines, improving reproducibility (seeds, environments, determinism), and structuring analysis scripts for sharing. Distinct from \u002Fexploratory-data-analysis (which understands data) and \u002Fstatistical-analysis (which chooses and runs tests) — this skill focuses on the code itself.","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fscientific-agent-skills\u002Fskills\u002Fresearch-code\u002FSKILL.md",[212,215],{"categoryNo":213,"categoryName":214,"taxonomyType":96,"taxonomyTypeDesc":97},"01JBSEEDCAT000000000000021","资料检索与研究",{"categoryNo":216,"categoryName":217,"taxonomyType":158,"taxonomyTypeDesc":159},"01JBSEEDCAT000000000000030","开发工程师",[219,220,140,221,32,222,223,106,224,24,107,109,225,226,227,228,229,230,177,231],"code","research code","which","data-analysis","errors","debugging","common","review","data pipelines","run","scripts","line","write",{"skillNo":233,"name":234,"nameZh":6,"description":235,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":236,"licenseFileUrl":6,"sourceHtmlUrl":237,"categories":238,"keywords":245,"publishedAt":39},"01M7166CF5HSR7SVZYZFWTDX55","markdown-mermaid-writing","Comprehensive markdown and Mermaid diagram writing skill. Use when creating any scientific document, report, analysis, or visualization. Establishes text-based diagrams as the default documentation standard with full style guides (markdown + mermaid), 24 diagram type references, and 9 document templates.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Ffadrienne\u002FMyGitHub\u002Fblob\u002Fmain\u002Fscientific-agent-skills\u002Fskills\u002Fmarkdown-mermaid-writing\u002FSKILL.md",[239,242,243,244],{"categoryNo":240,"categoryName":241,"taxonomyType":96,"taxonomyTypeDesc":97},"01JBSEEDCAT000000000000002","文档处理",{"categoryNo":152,"categoryName":153,"taxonomyType":96,"taxonomyTypeDesc":97},{"categoryNo":94,"categoryName":95,"taxonomyType":96,"taxonomyTypeDesc":97},{"categoryNo":156,"categoryName":157,"taxonomyType":158,"taxonomyTypeDesc":159},[161,246,247,248,249,250,190,251,252,24,107,109,253,254,255,256,257,258,162,177],"markdown mermaid writing","mermaid","markdown","community","diagrams","project","default","team","guides","text","documentation","references","diagram"]