AGENT SKILL

attention-mechanisms

Attention mechanisms in neural networks

License 许可
MIT
最近更新
2026-03-01
Stars
★ 3
Forks
2

01 前言 Front Matter

name
attention-mechanisms
license
MIT
metadata
{ "audience": "machine-learning-engineers", "category": "artificial-intelligence" }
description
Attention mechanisms in neural networks
compatibility
opencode

03 SKILL.md 正文

What I do

  • Implement attention mechanisms
  • Design transformer architectures
  • Build sequence-to-sequence models
  • Optimize attention computations
  • Apply self-attention and cross-attention
  • Create efficient attention patterns

When to use me

Use me when:

  • Building transformer models
  • Working with sequence data
  • Implementing NLP models
  • Creating vision transformers
  • Multi-modal learning

Key Concepts

Attention Formula

Attention(Q, K, V) = softmax(QK^T / √d_k)V

Q = Query (what we're looking for)
K = Key (what we're searching in)
V = Value (content to retrieve)
d_k = dimension of keys

Multi-Head Attention

import torch
import torch.nn as nn
import math

class MultiHeadAttention(nn.Module):
    def __init__(self, d_model, num_heads):
        super().__init__()
        self.d_model = d_model
        self.num_heads = num_heads
        self.d_k = d_model // num_heads
        
        self.W_q = nn.Linear(d_model, d_model)
        self.W_k = nn.Linear(d_model, d_model)
        self.W_v = nn.Linear(d_model, d_model)
        self.W_o = nn.Linear(d_model, d_model)
    
    def split_heads(self, x):
        batch_size = x.size(0)
        x = x.view(batch_size, -1, self.num_heads, self.d_k)
        return x.permute(0, 2, 1, 3)
    
    def forward(self, q, k, v, mask=None):
        Q = self.split_heads(self.W_q(q))
        K = self.split_heads(self.W_k(k))
        V = self.split_heads(self.W_v(v))
        
        # Attention scores
        scores = torch.matmul(Q, K.permute(0, 1, 3, 2))
        scores = scores / math.sqrt(self.d_k)
        
        if mask is not None:
            scores = scores.masked_fill(mask == 0, -1e9)
        
        attn_weights = torch.softmax(scores, dim=-1)
        attn_output = torch.matmul(attn_weights, V)
        
        # Merge heads
        attn_output = attn_output.permute(0, 2, 1, 3).contiguous()
        attn_output = attn_output.view(attn_output.size(0), -1, self.d_model)
        
        return self.W_o(attn_output)

Attention Types

  • Self-Attention: Q, K, V from same sequence
  • Cross-Attention: Q from one, K, V from another
  • Causal Attention: Masked for language modeling
  • Sparse Attention: Computational efficiency
  • Linear Attention: Kernel-based approximation

Transformer Applications

  • BERT: Bidirectional encoders
  • GPT: Autoregressive decoders
  • T5: Encoder-decoder
  • ViT: Vision transformers
  • Stable Diffusion: Cross-attention in diffusion

04 同仓库更多 Skills