[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6KQKFN8JM39GVHFZ97EJXSW":3,"skill-same:01M6KQKFN8JM39GVHFZ97EJXSW":103},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":57,"commitSha":58,"skillDirectory":59,"forksCount":13,"repositoryHtmlUrl":60,"repositoryHomepageUrl":6,"ownerHtmlUrl":61,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":62,"install":63,"resources":76},"01M6KQKFN8JM39GVHFZ97EJXSW","ml-training-recipes",null,"Battle-tested PyTorch training recipes for all domains — LLMs, vision, diffusion, medical imaging, protein\u002Fdrug discovery, spatial omics, genomics. Covers training loops, optimizer selection (AdamW, Muon), LR scheduling, mixed precision, debugging, and systematic experimentation. Use when training or fine-tuning neural networks, debugging loss spikes or OOM, choosing architectures, or optimizing GPU throughput.","endman100\u002Fskill-router-nexus","endman100","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F31997019?v=4","01M6KPDX17BYDWW4RRB0TNGGMX","01M6KPDX64Y2QBEWVFG5W5PWPF",0,"MIT","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Training\u002Fml-training-recipes\u002FSKILL.md",[],[18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"ml training recipes","ml","training","recipes","systematic","when","AdamW","Muon","loops","imaging","omics","experimentation","domains","llms","medical","optimizer","selection","tuning","debugging","pytorch","neural networks","2026-11-10 10:12:30",false,"---\nname: ml-training-recipes\ndescription: Battle-tested PyTorch training recipes for all domains — LLMs, vision, diffusion, medical imaging, protein\u002Fdrug discovery, spatial omics, genomics. Covers training loops, optimizer selection (AdamW, Muon), LR scheduling, mixed precision, debugging, and systematic experimentation. Use when training or fine-tuning neural networks, debugging loss spikes or OOM, choosing architectures, or optimizing GPU throughput.\nversion: 1.0.0\nauthor: dailycafi\nlicense: MIT\ntags: [PyTorch, Training, Optimization, LLM, Vision, Diffusion, Biomedical, Muon, AdamW, Debugging]\ndependencies: [torch>=2.0.0]\n---\n\n# ML Training Recipes\n\nBattle-tested patterns for PyTorch training across domains. Drawn from production codebases\n(Karpathy's autoresearch\u002Fnanochat, torchvision, HuggingFace) and modern training practice.\n\n## Reference files (read when needed)\n\n- `references\u002Farchitecture.md` — Transformer\u002FLLM architecture code patterns, weight init\n- `references\u002Foptimizers.md` — Muon, AdamW hybrid, per-group LR, compiled optimizer steps\n- `references\u002Fdomain-specific.md` — Vision, diffusion, contrastive, distributed, checkpointing, data loading\n- `references\u002Fscaling-and-selection.md` — Scaling laws, compute budget tables, decision trees, DGX Spark\n- `references\u002Fbiomedical.md` — Drug discovery, protein models, medical imaging, genomics, clinical NLP\n- `references\u002Fexperiment-loop.md` — Autonomous experiment loop (autoresearch keep\u002Fdiscard\u002Frevert)\n\n---\n\n## Architecture Selection\n\nPick the right model by **data type** and **data scale**:\n\n| Data Type | \u003C 10K samples | 10K-100K | > 100K |\n|-----------|--------------|----------|--------|\n| **Images** | Pretrained CNN + fine-tune | Fine-tune ViT or CNN | ViT from scratch |\n| **Text (gen)** | Few-shot prompting | Fine-tune GPT\u002FLLaMA (LoRA) | Pretrain from scratch |\n| **Tabular** | XGBoost\u002FLightGBM | Still XGBoost | Neural viable |\n| **Audio** | Pretrained Whisper | Fine-tune AST | Train from scratch |\n| **Molecules** | Pretrained GNN | Fine-tune molecular LM | Train GNN from scratch |\n| **Proteins** | ESM-2 embeddings + head | Fine-tune ESM-2 | Train protein LM |\n| **Medical img** | Pretrained CNN | nnU-Net (auto-config) | Swin-UNETR \u002F MedSAM |\n\n**Key principle**: architecture matters less than training recipe at equal compute. A well-tuned\nResNet beats a poorly-tuned ViT (ref: \"ResNet Strikes Back\", Wightman 2021).\n\nFor biomedical domains, see `references\u002Fbiomedical.md`.\nFor sequence model selection and compute planning, see `references\u002Fscaling-and-selection.md`.\n\n---\n\n## Scaling Laws\n\n### Chinchilla rule (Hoffmann et al., 2022)\n\nCompute-optimal training: **~20 tokens per parameter**.\n\n| Model Size | Compute-Optimal | Inference-Optimal (100×) |\n|-----------|----------------|--------------------------|\n| 125M | 2.5B tokens | 12.5B tokens |\n| 1B | 20B tokens | 100B tokens |\n| 7B | 140B tokens | 700B tokens |\n\n**FLOPs ≈ 6 × N × D** (N=params, D=tokens). Data repetition limit: ~4 epochs before diminishing returns.\n\n---\n\n## Training Loop\n\n```python\nimport gc, time, torch\n\ntorch.manual_seed(42)\ntorch.set_float32_matmul_precision(\"high\")  # TF32 on Ampere+\nautocast_ctx = torch.amp.autocast(device_type=\"cuda\", dtype=torch.bfloat16)\n\ngrad_accum_steps = total_batch_size \u002F\u002F (batch_size * seq_len)\nstep = 0\n\nwhile not done:\n    t0 = time.time()\n    for micro_step in range(grad_accum_steps):\n        with autocast_ctx:\n            loss = model(x, y)\n        (loss \u002F grad_accum_steps).backward()\n        x, y = next(train_loader)\n\n    update_lr(optimizer, progress)\n    optimizer.step()\n    model.zero_grad(set_to_none=True)  # frees memory vs zeroing\n\n    if loss.item() > 100:  # fast-fail on divergence\n        print(\"FAIL: loss exploded\"); exit(1)\n\n    torch.cuda.synchronize()\n    if step == 0:\n        gc.collect(); gc.freeze(); gc.disable()  # avoid ~500ms GC stalls\n    step += 1\n```\n\n### Key principles\n\n- **Gradient clipping**: `clip_grad_norm_(params, 1.0)` — near-universal for Transformers.\n  Exception: Muon optimizer normalizes updates via orthogonalization, so clipping is optional.\n- **Tensor Core alignment**: batch size, hidden dims should be multiples of 8 (bf16) or 64 (A100).\n- **Time-based budgets** make experiments comparable across hardware.\n- **`cudnn.benchmark = True`** for fixed-size vision inputs.\n\n---\n\n## Optimizer Configuration\n\nModern LLM training uses different optimizers per parameter group:\n\n| Parameter Type | Optimizer | LR (base) | Weight Decay |\n|---------------|-----------|-----------|--------------|\n| 2D weight matrices | Muon | 0.04 | 0.2 |\n| Token embeddings | AdamW | 0.6 × scale | 0.0 |\n| Unembedding (lm_head) | AdamW | 0.004 × scale | 0.0 |\n| Per-layer scalars | AdamW | 0.005 × scale | 0.0 |\n\n**LR scaling by dimension**: `lr * (d_model \u002F 768)^(-0.5)` — keeps dynamics stable across sizes.\n\n### Rules of thumb\n\n- Embeddings need higher LR (sparse updates). Never weight-decay embeddings.\n- Weight decay scheduling: linearly decay WD to 0 over training.\n- AdamW defaults: β1=0.9, β2=0.95, eps=1e-10 (not default 1e-8 — prevents stale updates in bf16).\n\nFor Muon details (polar express orthogonalization, NorMuon), see `references\u002Foptimizers.md`.\n\n---\n\n## Learning Rate Scheduling\n\n### Time-based (autoresearch style)\n\n```python\ndef get_lr_multiplier(progress):  # progress = elapsed_time \u002F time_budget\n    if progress \u003C warmup_ratio:\n        return progress \u002F warmup_ratio\n    elif progress \u003C 1.0 - warmdown_ratio:\n        return 1.0\n    else:\n        cooldown = (1.0 - progress) \u002F warmdown_ratio\n        return cooldown + (1 - cooldown) * final_lr_frac\n```\n\n### Cosine decay\n\n```python\ndef get_lr(step, total_steps, max_lr, min_lr, warmup_steps):\n    if step \u003C warmup_steps:\n        return max_lr * step \u002F warmup_steps\n    progress = (step - warmup_steps) \u002F (total_steps - warmup_steps)\n    return min_lr + 0.5 * (max_lr - min_lr) * (1 + math.cos(math.pi * progress))\n```\n\n**WSD (Warmup-Stable-Decay)**: gaining traction — easier to resume training mid-run.\n\n### Guidance\n\n- **Warmup**: 1-5% of training. Zero warmup valid with Muon (autoresearch uses `WARMUP_RATIO=0.0`).\n- **Warmdown**: 30-50% of training in LR decay. Matters more than warmup for final quality.\n- **Final LR**: 0 or ~10% of peak. Zero is simpler.\n\n---\n\n## Mixed Precision & Compilation\n\n```python\nimport os\nos.environ[\"PYTORCH_ALLOC_CONF\"] = \"expandable_segments:True\"  # before torch import\n\nimport torch\ntorch.set_float32_matmul_precision(\"high\")\nautocast_ctx = torch.amp.autocast(device_type=\"cuda\", dtype=torch.bfloat16)\nmodel = torch.compile(model, dynamic=False)\n```\n\n- **bf16** (Ampere+): same exponent as fp32, no loss scaling needed. Preferred over fp16.\n- **fp16**: needs GradScaler. Use only on V100 or older.\n- `dynamic=False` enables max optimization. Add `fullgraph=True` if no graph breaks.\n- First steps are slow (JIT) — exclude from timing.\n\n---\n\n## Memory & Performance\n\n### Meta device init (large models)\n\n```python\nwith torch.device(\"meta\"):\n    model = GPT(config)          # zero memory\nmodel.to_empty(device=\"cuda\")\nmodel.init_weights()\n```\n\n### MFU (Model FLOPs Utilization)\n\n```python\nachieved_flops = model_flops_per_token * batch_tokens \u002F step_time\nmfu = achieved_flops \u002F gpu_peak_flops\n# H100 SXM: 989.5 TFLOPS | A100: 312 | RTX 4090: 165\n```\n\nGood targets: >30% decent, >40% good, >50% excellent (single-GPU).\n\n### OOM solutions (in order)\n\n1. Reduce `DEVICE_BATCH_SIZE`, increase `grad_accum_steps`\n2. `PYTORCH_ALLOC_CONF=expandable_segments:True`\n3. `model.zero_grad(set_to_none=True)`\n4. Meta device init → `to_empty`\n5. Activation checkpointing: `torch.utils.checkpoint.checkpoint()`\n6. 8-bit optimizer (bitsandbytes): ~30% savings on optimizer states\n\n---\n\n## Hyperparameter Search\n\n### Priority order (tune first → last)\n\n1. **Learning rate** — most impactful. Always tune first.\n2. **Batch size** — largest that fits. Speed knob, not quality knob.\n3. **Weight decay** — 0.01-0.1 for AdamW.\n4. **Warmup steps** — 1-5% of training.\n\n### The 2025 default recipe\n\n| Setting | Value |\n|---------|-------|\n| Optimizer | AdamW (β1=0.9, β2=0.95, eps=1e-10) |\n| Weight decay | 0.1 |\n| LR schedule | Cosine decay or WSD |\n| Peak LR | 3e-4 (scale down for larger models) |\n| Precision | bf16 |\n| Grad clipping | max_norm=1.0 |\n| Normalization | RMSNorm (pre-norm) |\n| Activation | SwiGLU |\n| Position encoding | RoPE |\n| Attention | Flash Attention, optionally GQA |\n\n---\n\n## Debugging Checklist\n\n### Karpathy's recipe (still canonical)\n\n1. **Become one with the data** — visualize, check distributions, verify labels\n2. **Get end-to-end running first** — verify on a trivial case\n3. **Overfit one batch** — if you can't, you have a bug\n4. **Then regularize** — add regularization only after overfitting works\n5. **Tune hyperparameters** — start with known defaults\n\n### Loss exploding \u002F NaN\n\n1. Reduce LR (3-10× smaller)\n2. Add gradient clipping: `clip_grad_norm_(params, 1.0)`\n3. Check for inf\u002Fnan in inputs\n4. Add logit soft capping: `softcap * tanh(logits \u002F softcap)`\n5. Add QK-norm in attention\n6. Verify weight init (zero-init output projections?)\n7. Check loss reduction with gradient accumulation (`loss \u002F grad_accum_steps`)\n\n### Slow training \u002F Low MFU\n\n1. Verify `torch.compile` is active\n2. Check `torch.set_float32_matmul_precision(\"high\")`\n3. Pin memory + non_blocking transfers\n4. Profile with `torch.profiler`\n5. GC stalls? `gc.freeze(); gc.disable()`\n6. Tensor Core alignment: dims multiples of 8\u002F64\n\n### Loss plateau \u002F Slow convergence\n\n1. LR too low — try 2-5× larger\n2. Warmup too long\n3. Weight decay too high\n4. Verify LR schedule is actually applied (print each step)\n5. Model too small for task\n\n### Silent failures\n\n1. **Data leakage** between train\u002Fval\n2. **Wrong preprocessing at inference** — augmentation mismatch\n3. **Label errors** — use cleanlab to detect\n4. **Shuffling bugs** — correlated batches\n5. **Tokenizer mismatch** with pretrained model\n\n### What to monitor\n\n- **Gradient norms** — spike precedes loss spike\n- **Per-layer activation stats** — reveals exploding\u002Fvanishing\n- **Dead neurons** — >50% zero ReLU = dying ReLU problem\n- **Learning rate** — verify schedule applied (common silent bug)\n\n---\n\n## Experiment Management\n\nTrack experiments in TSV for easy comparison:\n\n```\ncommit  val_bpb  memory_gb  status   description\na1b2c3d 0.9979   44.0       keep     baseline\nb2c3d4e 0.9932   44.2       keep     increase matrix LR to 0.04\nc3d4e5f 1.0050   44.0       discard  switch to GeLU (worse)\n```\n\n**Simplicity criterion**: all else equal, simpler is better. Removing something and getting equal\nresults is a great outcome. For systematic agent-driven experimentation, see `references\u002Fexperiment-loop.md`.\n\n### Evaluation metrics by domain\n\n| Domain | Primary Metric | Notes |\n|--------|---------------|-------|\n| LLM | BPB (bits per byte) | Vocab-size-independent |\n| Classification | Accuracy \u002F F1 | Macro-F1 for imbalanced |\n| Segmentation | mIoU \u002F Dice | Per-class IoU reveals weak spots |\n| Generation | FID | Needs >10k samples |\n| Regression | RMSE \u002F MAE | Log-transform skewed targets |\n","# ML Training Recipes\n\nBattle-tested patterns for PyTorch training across domains. Drawn from production codebases\n(Karpathy's autoresearch\u002Fnanochat, torchvision, HuggingFace) and modern training practice.\n\n## Reference files (read when needed)\n\n- `references\u002Farchitecture.md` — Transformer\u002FLLM architecture code patterns, weight init\n- `references\u002Foptimizers.md` — Muon, AdamW hybrid, per-group LR, compiled optimizer steps\n- `references\u002Fdomain-specific.md` — Vision, diffusion, contrastive, distributed, checkpointing, data loading\n- `references\u002Fscaling-and-selection.md` — Scaling laws, compute budget tables, decision trees, DGX Spark\n- `references\u002Fbiomedical.md` — Drug discovery, protein models, medical imaging, genomics, clinical NLP\n- `references\u002Fexperiment-loop.md` — Autonomous experiment loop (autoresearch keep\u002Fdiscard\u002Frevert)\n\n---\n\n## Architecture Selection\n\nPick the right model by **data type** and **data scale**:\n\n| Data Type | \u003C 10K samples | 10K-100K | > 100K |\n|-----------|--------------|----------|--------|\n| **Images** | Pretrained CNN + fine-tune | Fine-tune ViT or CNN | ViT from scratch |\n| **Text (gen)** | Few-shot prompting | Fine-tune GPT\u002FLLaMA (LoRA) | Pretrain from scratch |\n| **Tabular** | XGBoost\u002FLightGBM | Still XGBoost | Neural viable |\n| **Audio** | Pretrained Whisper | Fine-tune AST | Train from scratch |\n| **Molecules** | Pretrained GNN | Fine-tune molecular LM | Train GNN from scratch |\n| **Proteins** | ESM-2 embeddings + head | Fine-tune ESM-2 | Train protein LM |\n| **Medical img** | Pretrained CNN | nnU-Net (auto-config) | Swin-UNETR \u002F MedSAM |\n\n**Key principle**: architecture matters less than training recipe at equal compute. A well-tuned\nResNet beats a poorly-tuned ViT (ref: \"ResNet Strikes Back\", Wightman 2021).\n\nFor biomedical domains, see `references\u002Fbiomedical.md`.\nFor sequence model selection and compute planning, see `references\u002Fscaling-and-selection.md`.\n\n---\n\n## Scaling Laws\n\n### Chinchilla rule (Hoffmann et al., 2022)\n\nCompute-optimal training: **~20 tokens per parameter**.\n\n| Model Size | Compute-Optimal | Inference-Optimal (100×) |\n|-----------|----------------|--------------------------|\n| 125M | 2.5B tokens | 12.5B tokens |\n| 1B | 20B tokens | 100B tokens |\n| 7B | 140B tokens | 700B tokens |\n\n**FLOPs ≈ 6 × N × D** (N=params, D=tokens). Data repetition limit: ~4 epochs before diminishing returns.\n\n---\n\n## Training Loop\n\n```python\nimport gc, time, torch\n\ntorch.manual_seed(42)\ntorch.set_float32_matmul_precision(\"high\")  # TF32 on Ampere+\nautocast_ctx = torch.amp.autocast(device_type=\"cuda\", dtype=torch.bfloat16)\n\ngrad_accum_steps = total_batch_size \u002F\u002F (batch_size * seq_len)\nstep = 0\n\nwhile not done:\n    t0 = time.time()\n    for micro_step in range(grad_accum_steps):\n        with autocast_ctx:\n            loss = model(x, y)\n        (loss \u002F grad_accum_steps).backward()\n        x, y = next(train_loader)\n\n    update_lr(optimizer, progress)\n    optimizer.step()\n    model.zero_grad(set_to_none=True)  # frees memory vs zeroing\n\n    if loss.item() > 100:  # fast-fail on divergence\n        print(\"FAIL: loss exploded\"); exit(1)\n\n    torch.cuda.synchronize()\n    if step == 0:\n        gc.collect(); gc.freeze(); gc.disable()  # avoid ~500ms GC stalls\n    step += 1\n```\n\n### Key principles\n\n- **Gradient clipping**: `clip_grad_norm_(params, 1.0)` — near-universal for Transformers.\n  Exception: Muon optimizer normalizes updates via orthogonalization, so clipping is optional.\n- **Tensor Core alignment**: batch size, hidden dims should be multiples of 8 (bf16) or 64 (A100).\n- **Time-based budgets** make experiments comparable across hardware.\n- **`cudnn.benchmark = True`** for fixed-size vision inputs.\n\n---\n\n## Optimizer Configuration\n\nModern LLM training uses different optimizers per parameter group:\n\n| Parameter Type | Optimizer | LR (base) | Weight Decay |\n|---------------|-----------|-----------|--------------|\n| 2D weight matrices | Muon | 0.04 | 0.2 |\n| Token embeddings | AdamW | 0.6 × scale | 0.0 |\n| Unembedding (lm_head) | AdamW | 0.004 × scale | 0.0 |\n| Per-layer scalars | AdamW | 0.005 × scale | 0.0 |\n\n**LR scaling by dimension**: `lr * (d_model \u002F 768)^(-0.5)` — keeps dynamics stable across sizes.\n\n### Rules of thumb\n\n- Embeddings need higher LR (sparse updates). Never weight-decay embeddings.\n- Weight decay scheduling: linearly decay WD to 0 over training.\n- AdamW defaults: β1=0.9, β2=0.95, eps=1e-10 (not default 1e-8 — prevents stale updates in bf16).\n\nFor Muon details (polar express orthogonalization, NorMuon), see `references\u002Foptimizers.md`.\n\n---\n\n## Learning Rate Scheduling\n\n### Time-based (autoresearch style)\n\n```python\ndef get_lr_multiplier(progress):  # progress = elapsed_time \u002F time_budget\n    if progress \u003C warmup_ratio:\n        return progress \u002F warmup_ratio\n    elif progress \u003C 1.0 - warmdown_ratio:\n        return 1.0\n    else:\n        cooldown = (1.0 - progress) \u002F warmdown_ratio\n        return cooldown + (1 - cooldown) * final_lr_frac\n```\n\n### Cosine decay\n\n```python\ndef get_lr(step, total_steps, max_lr, min_lr, warmup_steps):\n    if step \u003C warmup_steps:\n        return max_lr * step \u002F warmup_steps\n    progress = (step - warmup_steps) \u002F (total_steps - warmup_steps)\n    return min_lr + 0.5 * (max_lr - min_lr) * (1 + math.cos(math.pi * progress))\n```\n\n**WSD (Warmup-Stable-Decay)**: gaining traction — easier to resume training mid-run.\n\n### Guidance\n\n- **Warmup**: 1-5% of training. Zero warmup valid with Muon (autoresearch uses `WARMUP_RATIO=0.0`).\n- **Warmdown**: 30-50% of training in LR decay. Matters more than warmup for final quality.\n- **Final LR**: 0 or ~10% of peak. Zero is simpler.\n\n---\n\n## Mixed Precision & Compilation\n\n```python\nimport os\nos.environ[\"PYTORCH_ALLOC_CONF\"] = \"expandable_segments:True\"  # before torch import\n\nimport torch\ntorch.set_float32_matmul_precision(\"high\")\nautocast_ctx = torch.amp.autocast(device_type=\"cuda\", dtype=torch.bfloat16)\nmodel = torch.compile(model, dynamic=False)\n```\n\n- **bf16** (Ampere+): same exponent as fp32, no loss scaling needed. Preferred over fp16.\n- **fp16**: needs GradScaler. Use only on V100 or older.\n- `dynamic=False` enables max optimization. Add `fullgraph=True` if no graph breaks.\n- First steps are slow (JIT) — exclude from timing.\n\n---\n\n## Memory & Performance\n\n### Meta device init (large models)\n\n```python\nwith torch.device(\"meta\"):\n    model = GPT(config)          # zero memory\nmodel.to_empty(device=\"cuda\")\nmodel.init_weights()\n```\n\n### MFU (Model FLOPs Utilization)\n\n```python\nachieved_flops = model_flops_per_token * batch_tokens \u002F step_time\nmfu = achieved_flops \u002F gpu_peak_flops\n# H100 SXM: 989.5 TFLOPS | A100: 312 | RTX 4090: 165\n```\n\nGood targets: >30% decent, >40% good, >50% excellent (single-GPU).\n\n### OOM solutions (in order)\n\n1. Reduce `DEVICE_BATCH_SIZE`, increase `grad_accum_steps`\n2. `PYTORCH_ALLOC_CONF=expandable_segments:True`\n3. `model.zero_grad(set_to_none=True)`\n4. Meta device init → `to_empty`\n5. Activation checkpointing: `torch.utils.checkpoint.checkpoint()`\n6. 8-bit optimizer (bitsandbytes): ~30% savings on optimizer states\n\n---\n\n## Hyperparameter Search\n\n### Priority order (tune first → last)\n\n1. **Learning rate** — most impactful. Always tune first.\n2. **Batch size** — largest that fits. Speed knob, not quality knob.\n3. **Weight decay** — 0.01-0.1 for AdamW.\n4. **Warmup steps** — 1-5% of training.\n\n### The 2025 default recipe\n\n| Setting | Value |\n|---------|-------|\n| Optimizer | AdamW (β1=0.9, β2=0.95, eps=1e-10) |\n| Weight decay | 0.1 |\n| LR schedule | Cosine decay or WSD |\n| Peak LR | 3e-4 (scale down for larger models) |\n| Precision | bf16 |\n| Grad clipping | max_norm=1.0 |\n| Normalization | RMSNorm (pre-norm) |\n| Activation | SwiGLU |\n| Position encoding | RoPE |\n| Attention | Flash Attention, optionally GQA |\n\n---\n\n## Debugging Checklist\n\n### Karpathy's recipe (still canonical)\n\n1. **Become one with the data** — visualize, check distributions, verify labels\n2. **Get end-to-end running first** — verify on a trivial case\n3. **Overfit one batch** — if you can't, you have a bug\n4. **Then regularize** — add regularization only after overfitting works\n5. **Tune hyperparameters** — start with known defaults\n\n### Loss exploding \u002F NaN\n\n1. Reduce LR (3-10× smaller)\n2. Add gradient clipping: `clip_grad_norm_(params, 1.0)`\n3. Check for inf\u002Fnan in inputs\n4. Add logit soft capping: `softcap * tanh(logits \u002F softcap)`\n5. Add QK-norm in attention\n6. Verify weight init (zero-init output projections?)\n7. Check loss reduction with gradient accumulation (`loss \u002F grad_accum_steps`)\n\n### Slow training \u002F Low MFU\n\n1. Verify `torch.compile` is active\n2. Check `torch.set_float32_matmul_precision(\"high\")`\n3. Pin memory + non_blocking transfers\n4. Profile with `torch.profiler`\n5. GC stalls? `gc.freeze(); gc.disable()`\n6. Tensor Core alignment: dims multiples of 8\u002F64\n\n### Loss plateau \u002F Slow convergence\n\n1. LR too low — try 2-5× larger\n2. Warmup too long\n3. Weight decay too high\n4. Verify LR schedule is actually applied (print each step)\n5. Model too small for task\n\n### Silent failures\n\n1. **Data leakage** between train\u002Fval\n2. **Wrong preprocessing at inference** — augmentation mismatch\n3. **Label errors** — use cleanlab to detect\n4. **Shuffling bugs** — correlated batches\n5. **Tokenizer mismatch** with pretrained model\n\n### What to monitor\n\n- **Gradient norms** — spike precedes loss spike\n- **Per-layer activation stats** — reveals exploding\u002Fvanishing\n- **Dead neurons** — >50% zero ReLU = dying ReLU problem\n- **Learning rate** — verify schedule applied (common silent bug)\n\n---\n\n## Experiment Management\n\nTrack experiments in TSV for easy comparison:\n\n```\ncommit  val_bpb  memory_gb  status   description\na1b2c3d 0.9979   44.0       keep     baseline\nb2c3d4e 0.9932   44.2       keep     increase matrix LR to 0.04\nc3d4e5f 1.0050   44.0       discard  switch to GeLU (worse)\n```\n\n**Simplicity criterion**: all else equal, simpler is better. Removing something and getting equal\nresults is a great outcome. For systematic agent-driven experimentation, see `references\u002Fexperiment-loop.md`.\n\n### Evaluation metrics by domain\n\n| Domain | Primary Metric | Notes |\n|--------|---------------|-------|\n| LLM | BPB (bits per byte) | Vocab-size-independent |\n| Classification | Accuracy \u002F F1 | Macro-F1 for imbalanced |\n| Segmentation | mIoU \u002F Dice | Per-class IoU reveals weak spots |\n| Generation | FID | Needs >10k samples |\n| Regression | RMSE \u002F MAE | Log-transform skewed targets |\n",{"name":5,"tags":44,"author":53,"license":14,"version":54,"description":7,"dependencies":55},[45,46,47,48,49,50,51,25,24,52],"PyTorch","Training","Optimization","LLM","Vision","Diffusion","Biomedical","Debugging","dailycafi","1.0.0",[56],"torch>=2.0.0","main","f19b254e8975d0bfddb394686706106930d7964d","AI-Training\u002Fml-training-recipes","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus","https:\u002F\u002Fgithub.com\u002Fendman100","2026-11-11 18:19:10",{"source":8,"skillName":5,"commands":64,"prompt":74,"zipAvailable":75},[65,68,71],{"tool":66,"command":67},"npx","npx skills add endman100\u002Fskill-router-nexus --skill ml-training-recipes",{"tool":69,"command":70},"bunx","bunx skills add endman100\u002Fskill-router-nexus --skill ml-training-recipes",{"tool":72,"command":73},"pnpm","pnpm dlx skills add endman100\u002Fskill-router-nexus --skill ml-training-recipes","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Training\u002Fml-training-recipes\u002FSKILL.md",true,[77,83,87,91,95,99],{"path":78,"resourceType":79,"resourceTypeDesc":80,"sizeBytes":81,"sourceHtmlUrl":82},"references\u002Farchitecture.md","REFERENCE","参考文档",10204,"https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Training\u002Fml-training-recipes\u002Freferences\u002Farchitecture.md",{"path":84,"resourceType":79,"resourceTypeDesc":80,"sizeBytes":85,"sourceHtmlUrl":86},"references\u002Fbiomedical.md",20778,"https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Training\u002Fml-training-recipes\u002Freferences\u002Fbiomedical.md",{"path":88,"resourceType":79,"resourceTypeDesc":80,"sizeBytes":89,"sourceHtmlUrl":90},"references\u002Fdomain-specific.md",19018,"https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Training\u002Fml-training-recipes\u002Freferences\u002Fdomain-specific.md",{"path":92,"resourceType":79,"resourceTypeDesc":80,"sizeBytes":93,"sourceHtmlUrl":94},"references\u002Fexperiment-loop.md",4446,"https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Training\u002Fml-training-recipes\u002Freferences\u002Fexperiment-loop.md",{"path":96,"resourceType":79,"resourceTypeDesc":80,"sizeBytes":97,"sourceHtmlUrl":98},"references\u002Foptimizers.md",10564,"https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Training\u002Fml-training-recipes\u002Freferences\u002Foptimizers.md",{"path":100,"resourceType":79,"resourceTypeDesc":80,"sizeBytes":101,"sourceHtmlUrl":102},"references\u002Fscaling-and-selection.md",16832,"https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Training\u002Fml-training-recipes\u002Freferences\u002Fscaling-and-selection.md",[104,128,166,188,197,221],{"skillNo":105,"name":106,"nameZh":6,"description":107,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":108,"sourceHtmlUrl":109,"categories":110,"keywords":116,"publishedAt":39},"01M6MPCT5BXVER8HQZTW3QE9AS","create-master","基于佛教经典文献，生成特定高僧大德的 AI 教学角色","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FMetaphysics-and-Spirituality\u002FMaster-skill\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FMetaphysics-and-Spirituality\u002FMaster-skill\u002FSKILL.md",[111],{"categoryNo":112,"categoryName":113,"taxonomyType":114,"taxonomyTypeDesc":115},"01JBSEEDCAT000000000000039","教师与教育","PROFESSION","适用职业",[117,118,119,120,121,122,123,124,125,126,127],"create master","master","create","生成","true","webfetch","edit","write","read","grep","Bash",{"skillNo":129,"name":130,"nameZh":6,"description":131,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":132,"sourceHtmlUrl":133,"categories":134,"keywords":146,"publishedAt":39},"01M6M43AKE95Y64E1JRXEF5QQ3","writing-dna-skill","从至少 20 篇完整文章中蒸馏可复用的写作 DNA，分析语言、文章结构、选题逻辑、素材策略、认知框架和视觉风格，并生成 Writing-DNA.md；按该 DNA 写作时会读取全部蒸馏产物和 5 篇相关原文。用于中英文作者、账号、品牌或出版物的风格分析与一致性写作。 Distill reusable Writing DNA from at least 20 complete articles for Chinese or English authors, publications, brands, and accounts; use for language, structure, topic logic, source strategy, cognitive-frame, and visual-style analysis.","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fwriting-dna-skill\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fwriting-dna-skill\u002FSKILL.md",[135,140,143],{"categoryNo":136,"categoryName":137,"taxonomyType":138,"taxonomyTypeDesc":139},"01JBSEEDCAT000000000000018","写作与文案","SCENARIO","使用场景",{"categoryNo":141,"categoryName":142,"taxonomyType":138,"taxonomyTypeDesc":139},"01JBSEEDCAT000000000000023","Skill与智能体开发",{"categoryNo":144,"categoryName":145,"taxonomyType":114,"taxonomyTypeDesc":115},"01JBSEEDCAT000000000000036","内容创作者",[147,148,149,150,151,152,153,154,155,156,157,158,159,120,160,161,162,163,164,165],"writing dna skill","DNA","writing","strategy","english","logic","visual","structure","chinese","reusable","use","style","distill","analysis","frame","source","for","at","complete",{"skillNo":167,"name":168,"nameZh":6,"description":169,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":170,"sourceHtmlUrl":171,"categories":172,"keywords":175,"publishedAt":39},"01M6M428TTCGQ4ME3KY99VN037","stop-slop","Remove AI writing patterns from prose. Use when drafting, editing, or reviewing text to eliminate predictable AI tells.","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fstop-slop\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fstop-slop\u002FSKILL.md",[173,174],{"categoryNo":136,"categoryName":137,"taxonomyType":138,"taxonomyTypeDesc":139},{"categoryNo":144,"categoryName":145,"taxonomyType":114,"taxonomyTypeDesc":115},[176,177,178,179,23,180,181,182,183,157,184,185,186,163,187,149],"slop","stop slop","stop","drafting","drafts","https","patterns","prose","remove","editing","text","content",{"skillNo":189,"name":190,"nameZh":6,"description":191,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":192,"sourceHtmlUrl":193,"categories":194,"keywords":195,"publishedAt":39},"01M6M41X0B7EBX42TK435DMGGR","shuorenhua","检查和清理中英文文本里的 AI 套路，适用于“去 AI 味”“说人话”“自然一点”“别像模板”“先标问题”这类改写和审稿需求；按场景控制力度，同时保留事实、术语、语域和责任主体。","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fshuorenhua\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fshuorenhua\u002FSKILL.md",[],[190,196],"文本",{"skillNo":198,"name":199,"nameZh":6,"description":200,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":201,"sourceHtmlUrl":202,"categories":203,"keywords":204,"publishedAt":39},"01M6M3ZD12WBEW5KXKDM4N6DFG","naming","Name products, SaaS, brands, open source projects, bots, and apps. Use when the user needs to name something, find a brand name, or pick a product name. Metaphor-driven process that produces memorable, meaningful names and avoids AI slop.","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fnaming-main\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fnaming-main\u002FSKILL.md",[],[199,205,206,207,208,209,157,210,125,162,211,212,213,214,127,215,216,217,218,23,219,220,176],"repo","websearch","product name","apps","bots","view","brand","pick","gh","projects","open","saas","open source","product","describe","ai slop",{"skillNo":222,"name":223,"nameZh":6,"description":224,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":225,"sourceHtmlUrl":226,"categories":227,"keywords":229,"publishedAt":39},"01M6M3YVXAFAAWJ7F6JMPFREYE","humanizer-zh","去除文本中的 AI 生成痕迹。适用于编辑或审阅文本，使其听起来更自然、更像人类书写。\n基于维基百科的\"AI 写作特征\"综合指南。检测并修复以下模式：夸大的象征意义、\n宣传性语言、以 -ing 结尾的肤浅分析、模糊的归因、破折号过度使用、三段式法则、\nAI 词汇、否定式排比、过多的连接性短语。","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fhumanizer-zh\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fhumanizer-zh\u002FSKILL.md",[228],{"categoryNo":136,"categoryName":137,"taxonomyType":138,"taxonomyTypeDesc":139},[230,231,232,125,124,123,149,120,196,233,176,178],"humanizer","zh","humanizer zh","翻译"]