[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6MAFT2NFMYJN1GCFSTXKH34":3,"skill-same:01M6MAFT2NFMYJN1GCFSTXKH34":102},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":40,"linkOnly":41,"markdownContent":42,"markdownBody":43,"frontMatter":44,"refName":52,"commitSha":53,"skillDirectory":54,"forksCount":55,"repositoryHtmlUrl":56,"repositoryHomepageUrl":57,"ownerHtmlUrl":58,"ownerBlogUrl":59,"ownerDescription":60,"updatedAt":61,"install":62,"resources":75},"01M6MAFT2NFMYJN1GCFSTXKH34","ml-training-recipes",null,"Battle-tested PyTorch training recipes for all domains — LLMs, vision, diffusion, medical imaging, protein\u002Fdrug discovery, spatial omics, genomics. Covers training loops, optimizer selection (AdamW, Muon), LR scheduling, mixed precision, debugging, and systematic experimentation. Use when training or fine-tuning neural networks, debugging loss spikes or OOM, choosing architectures, or optimizing GPU throughput.","hashgraph-online\u002Fawesome-codex-plugins","hashgraph-online","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F160371962?v=4","01M6CNB6NRJVNNGN14TR01Q6D3","HOL (Hashgraph Online)","01M6JT1HSKKG9V68KQAQGBA7XP",1575,"MIT","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fml-training-recipes\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39],"ml","training","ml training recipes","recipes","biomedical","Protein","for","imaging","experimentation","architectures","domains","drug discovery","systematic","neural networks","selection","scheduling","use","discovery","networks","Fine-Tuning","pytorch","2026-11-10 10:12:30",false,"---\nname: ml-training-recipes\ndescription: Battle-tested PyTorch training recipes for all domains — LLMs, vision, diffusion, medical imaging, protein\u002Fdrug discovery, spatial omics, genomics. Covers training loops, optimizer selection (AdamW, Muon), LR scheduling, mixed precision, debugging, and systematic experimentation. Use when training or fine-tuning neural networks, debugging loss spikes or OOM, choosing architectures, or optimizing GPU throughput.\nlicense: MIT\nmetadata:\n  version: 1.0.0\n  category: ml-training\n  maintainer: Kalaris Labs\n  contributor: dailycafi\n  tags: PyTorch, Training, Optimization, LLM, Vision, Diffusion, Biomedical, Muon, AdamW, Debugging\n  dependencies: torch>=2.0.0\n---\n\n# ML Training Recipes\n\nBattle-tested patterns for PyTorch training across domains. Drawn from production codebases\n(Karpathy's autoresearch\u002Fnanochat, torchvision, HuggingFace) and modern training practice.\n\n## Reference files (read when needed)\n\n- `references\u002Farchitecture.md` — Transformer\u002FLLM architecture code patterns, weight init\n- `references\u002Foptimizers.md` — Muon, AdamW hybrid, per-group LR, compiled optimizer steps\n- `references\u002Fdomain-specific.md` — Vision, diffusion, contrastive, distributed, checkpointing, data loading\n- `references\u002Fscaling-and-selection.md` — Scaling laws, compute budget tables, decision trees, DGX Spark\n- `references\u002Fbiomedical.md` — Drug discovery, protein models, medical imaging, genomics, clinical NLP\n- `references\u002Fexperiment-loop.md` — Autonomous experiment loop (autoresearch keep\u002Fdiscard\u002Frevert)\n\n---\n\n## Architecture Selection\n\nPick the right model by **data type** and **data scale**:\n\n| Data Type | \u003C 10K samples | 10K-100K | > 100K |\n|-----------|--------------|----------|--------|\n| **Images** | Pretrained CNN + fine-tune | Fine-tune ViT or CNN | ViT from scratch |\n| **Text (gen)** | Few-shot prompting | Fine-tune GPT\u002FLLaMA (LoRA) | Pretrain from scratch |\n| **Tabular** | XGBoost\u002FLightGBM | Still XGBoost | Neural viable |\n| **Audio** | Pretrained Whisper | Fine-tune AST | Train from scratch |\n| **Molecules** | Pretrained GNN | Fine-tune molecular LM | Train GNN from scratch |\n| **Proteins** | ESM-2 embeddings + head | Fine-tune ESM-2 | Train protein LM |\n| **Medical img** | Pretrained CNN | nnU-Net (auto-config) | Swin-UNETR \u002F MedSAM |\n\n**Key principle**: architecture matters less than training recipe at equal compute. A well-tuned\nResNet beats a poorly-tuned ViT (ref: \"ResNet Strikes Back\", Wightman 2021).\n\nFor biomedical domains, see `references\u002Fbiomedical.md`.\nFor sequence model selection and compute planning, see `references\u002Fscaling-and-selection.md`.\n\n---\n\n## Scaling Laws\n\n### Chinchilla rule (Hoffmann et al., 2022)\n\nCompute-optimal training: **~20 tokens per parameter**.\n\n| Model Size | Compute-Optimal | Inference-Optimal (100×) |\n|-----------|----------------|--------------------------|\n| 125M | 2.5B tokens | 12.5B tokens |\n| 1B | 20B tokens | 100B tokens |\n| 7B | 140B tokens | 700B tokens |\n\n**FLOPs ≈ 6 × N × D** (N=params, D=tokens). Data repetition limit: ~4 epochs before diminishing returns.\n\n---\n\n## Training Loop\n\n```python\nimport gc, time, torch\n\ntorch.manual_seed(42)\ntorch.set_float32_matmul_precision(\"high\")  # TF32 on Ampere+\nautocast_ctx = torch.amp.autocast(device_type=\"cuda\", dtype=torch.bfloat16)\n\ngrad_accum_steps = total_batch_size \u002F\u002F (batch_size * seq_len)\nstep = 0\n\nwhile not done:\n    t0 = time.time()\n    for micro_step in range(grad_accum_steps):\n        with autocast_ctx:\n            loss = model(x, y)\n        (loss \u002F grad_accum_steps).backward()\n        x, y = next(train_loader)\n\n    update_lr(optimizer, progress)\n    optimizer.step()\n    model.zero_grad(set_to_none=True)  # frees memory vs zeroing\n\n    if loss.item() > 100:  # fast-fail on divergence\n        print(\"FAIL: loss exploded\"); exit(1)\n\n    torch.cuda.synchronize()\n    if step == 0:\n        gc.collect(); gc.freeze(); gc.disable()  # avoid ~500ms GC stalls\n    step += 1\n```\n\n### Key principles\n\n- **Gradient clipping**: `clip_grad_norm_(params, 1.0)` — near-universal for Transformers.\n  Exception: Muon optimizer normalizes updates via orthogonalization, so clipping is optional.\n- **Tensor Core alignment**: batch size, hidden dims should be multiples of 8 (bf16) or 64 (A100).\n- **Time-based budgets** make experiments comparable across hardware.\n- **`cudnn.benchmark = True`** for fixed-size vision inputs.\n\n---\n\n## Optimizer Configuration\n\nModern LLM training uses different optimizers per parameter group:\n\n| Parameter Type | Optimizer | LR (base) | Weight Decay |\n|---------------|-----------|-----------|--------------|\n| 2D weight matrices | Muon | 0.04 | 0.2 |\n| Token embeddings | AdamW | 0.6 × scale | 0.0 |\n| Unembedding (lm_head) | AdamW | 0.004 × scale | 0.0 |\n| Per-layer scalars | AdamW | 0.005 × scale | 0.0 |\n\n**LR scaling by dimension**: `lr * (d_model \u002F 768)^(-0.5)` — keeps dynamics stable across sizes.\n\n### Rules of thumb\n\n- Embeddings need higher LR (sparse updates). Never weight-decay embeddings.\n- Weight decay scheduling: linearly decay WD to 0 over training.\n- AdamW defaults: β1=0.9, β2=0.95, eps=1e-10 (not default 1e-8 — prevents stale updates in bf16).\n\nFor Muon details (polar express orthogonalization, NorMuon), see `references\u002Foptimizers.md`.\n\n---\n\n## Learning Rate Scheduling\n\n### Time-based (autoresearch style)\n\n```python\ndef get_lr_multiplier(progress):  # progress = elapsed_time \u002F time_budget\n    if progress \u003C warmup_ratio:\n        return progress \u002F warmup_ratio\n    elif progress \u003C 1.0 - warmdown_ratio:\n        return 1.0\n    else:\n        cooldown = (1.0 - progress) \u002F warmdown_ratio\n        return cooldown + (1 - cooldown) * final_lr_frac\n```\n\n### Cosine decay\n\n```python\ndef get_lr(step, total_steps, max_lr, min_lr, warmup_steps):\n    if step \u003C warmup_steps:\n        return max_lr * step \u002F warmup_steps\n    progress = (step - warmup_steps) \u002F (total_steps - warmup_steps)\n    return min_lr + 0.5 * (max_lr - min_lr) * (1 + math.cos(math.pi * progress))\n```\n\n**WSD (Warmup-Stable-Decay)**: gaining traction — easier to resume training mid-run.\n\n### Guidance\n\n- **Warmup**: 1-5% of training. Zero warmup valid with Muon (autoresearch uses `WARMUP_RATIO=0.0`).\n- **Warmdown**: 30-50% of training in LR decay. Matters more than warmup for final quality.\n- **Final LR**: 0 or ~10% of peak. Zero is simpler.\n\n---\n\n## Mixed Precision & Compilation\n\n```python\nimport os\nos.environ[\"PYTORCH_ALLOC_CONF\"] = \"expandable_segments:True\"  # before torch import\n\nimport torch\ntorch.set_float32_matmul_precision(\"high\")\nautocast_ctx = torch.amp.autocast(device_type=\"cuda\", dtype=torch.bfloat16)\nmodel = torch.compile(model, dynamic=False)\n```\n\n- **bf16** (Ampere+): same exponent as fp32, no loss scaling needed. Preferred over fp16.\n- **fp16**: needs GradScaler. Use only on V100 or older.\n- `dynamic=False` enables max optimization. Add `fullgraph=True` if no graph breaks.\n- First steps are slow (JIT) — exclude from timing.\n\n---\n\n## Memory & Performance\n\n### Meta device init (large models)\n\n```python\nwith torch.device(\"meta\"):\n    model = GPT(config)          # zero memory\nmodel.to_empty(device=\"cuda\")\nmodel.init_weights()\n```\n\n### MFU (Model FLOPs Utilization)\n\n```python\nachieved_flops = model_flops_per_token * batch_tokens \u002F step_time\nmfu = achieved_flops \u002F gpu_peak_flops\n# H100 SXM: 989.5 TFLOPS | A100: 312 | RTX 4090: 165\n```\n\nGood targets: >30% decent, >40% good, >50% excellent (single-GPU).\n\n### OOM solutions (in order)\n\n1. Reduce `DEVICE_BATCH_SIZE`, increase `grad_accum_steps`\n2. `PYTORCH_ALLOC_CONF=expandable_segments:True`\n3. `model.zero_grad(set_to_none=True)`\n4. Meta device init → `to_empty`\n5. Activation checkpointing: `torch.utils.checkpoint.checkpoint()`\n6. 8-bit optimizer (bitsandbytes): ~30% savings on optimizer states\n\n---\n\n## Hyperparameter Search\n\n### Priority order (tune first → last)\n\n1. **Learning rate** — most impactful. Always tune first.\n2. **Batch size** — largest that fits. Speed knob, not quality knob.\n3. **Weight decay** — 0.01-0.1 for AdamW.\n4. **Warmup steps** — 1-5% of training.\n\n### The 2025 default recipe\n\n| Setting | Value |\n|---------|-------|\n| Optimizer | AdamW (β1=0.9, β2=0.95, eps=1e-10) |\n| Weight decay | 0.1 |\n| LR schedule | Cosine decay or WSD |\n| Peak LR | 3e-4 (scale down for larger models) |\n| Precision | bf16 |\n| Grad clipping | max_norm=1.0 |\n| Normalization | RMSNorm (pre-norm) |\n| Activation | SwiGLU |\n| Position encoding | RoPE |\n| Attention | Flash Attention, optionally GQA |\n\n---\n\n## Debugging Checklist\n\n### Karpathy's recipe (still canonical)\n\n1. **Become one with the data** — visualize, check distributions, verify labels\n2. **Get end-to-end running first** — verify on a trivial case\n3. **Overfit one batch** — if you can't, you have a bug\n4. **Then regularize** — add regularization only after overfitting works\n5. **Tune hyperparameters** — start with known defaults\n\n### Loss exploding \u002F NaN\n\n1. Reduce LR (3-10× smaller)\n2. Add gradient clipping: `clip_grad_norm_(params, 1.0)`\n3. Check for inf\u002Fnan in inputs\n4. Add logit soft capping: `softcap * tanh(logits \u002F softcap)`\n5. Add QK-norm in attention\n6. Verify weight init (zero-init output projections?)\n7. Check loss reduction with gradient accumulation (`loss \u002F grad_accum_steps`)\n\n### Slow training \u002F Low MFU\n\n1. Verify `torch.compile` is active\n2. Check `torch.set_float32_matmul_precision(\"high\")`\n3. Pin memory + non_blocking transfers\n4. Profile with `torch.profiler`\n5. GC stalls? `gc.freeze(); gc.disable()`\n6. Tensor Core alignment: dims multiples of 8\u002F64\n\n### Loss plateau \u002F Slow convergence\n\n1. LR too low — try 2-5× larger\n2. Warmup too long\n3. Weight decay too high\n4. Verify LR schedule is actually applied (print each step)\n5. Model too small for task\n\n### Silent failures\n\n1. **Data leakage** between train\u002Fval\n2. **Wrong preprocessing at inference** — augmentation mismatch\n3. **Label errors** — use cleanlab to detect\n4. **Shuffling bugs** — correlated batches\n5. **Tokenizer mismatch** with pretrained model\n\n### What to monitor\n\n- **Gradient norms** — spike precedes loss spike\n- **Per-layer activation stats** — reveals exploding\u002Fvanishing\n- **Dead neurons** — >50% zero ReLU = dying ReLU problem\n- **Learning rate** — verify schedule applied (common silent bug)\n\n---\n\n## Experiment Management\n\nTrack experiments in TSV for easy comparison:\n\n```\ncommit  val_bpb  memory_gb  status   description\na1b2c3d 0.9979   44.0       keep     baseline\nb2c3d4e 0.9932   44.2       keep     increase matrix LR to 0.04\nc3d4e5f 1.0050   44.0       discard  switch to GeLU (worse)\n```\n\n**Simplicity criterion**: all else equal, simpler is better. Removing something and getting equal\nresults is a great outcome. For systematic agent-driven experimentation, see `references\u002Fexperiment-loop.md`.\n\n### Evaluation metrics by domain\n\n| Domain | Primary Metric | Notes |\n|--------|---------------|-------|\n| LLM | BPB (bits per byte) | Vocab-size-independent |\n| Classification | Accuracy \u002F F1 | Macro-F1 for imbalanced |\n| Segmentation | mIoU \u002F Dice | Per-class IoU reveals weak spots |\n| Generation | FID | Needs >10k samples |\n| Regression | RMSE \u002F MAE | Log-transform skewed targets |\n\n## Related skills\n\n- `serving-llms-vllm`: Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching.\n- `llama-factory`: Guides fine-tuning of large language models with LLaMA-Factory, covering the WebUI no-code interface, training across 100+ supported models…\n- `peft-fine-tuning`: Fine-tunes LLMs with Hugging Face PEFT, using LoRA, QLoRA, IA3, AdaLoRA, prefix tuning, and prompt tuning so that under 1% of parameters ar…\n","# ML Training Recipes\n\nBattle-tested patterns for PyTorch training across domains. Drawn from production codebases\n(Karpathy's autoresearch\u002Fnanochat, torchvision, HuggingFace) and modern training practice.\n\n## Reference files (read when needed)\n\n- `references\u002Farchitecture.md` — Transformer\u002FLLM architecture code patterns, weight init\n- `references\u002Foptimizers.md` — Muon, AdamW hybrid, per-group LR, compiled optimizer steps\n- `references\u002Fdomain-specific.md` — Vision, diffusion, contrastive, distributed, checkpointing, data loading\n- `references\u002Fscaling-and-selection.md` — Scaling laws, compute budget tables, decision trees, DGX Spark\n- `references\u002Fbiomedical.md` — Drug discovery, protein models, medical imaging, genomics, clinical NLP\n- `references\u002Fexperiment-loop.md` — Autonomous experiment loop (autoresearch keep\u002Fdiscard\u002Frevert)\n\n---\n\n## Architecture Selection\n\nPick the right model by **data type** and **data scale**:\n\n| Data Type | \u003C 10K samples | 10K-100K | > 100K |\n|-----------|--------------|----------|--------|\n| **Images** | Pretrained CNN + fine-tune | Fine-tune ViT or CNN | ViT from scratch |\n| **Text (gen)** | Few-shot prompting | Fine-tune GPT\u002FLLaMA (LoRA) | Pretrain from scratch |\n| **Tabular** | XGBoost\u002FLightGBM | Still XGBoost | Neural viable |\n| **Audio** | Pretrained Whisper | Fine-tune AST | Train from scratch |\n| **Molecules** | Pretrained GNN | Fine-tune molecular LM | Train GNN from scratch |\n| **Proteins** | ESM-2 embeddings + head | Fine-tune ESM-2 | Train protein LM |\n| **Medical img** | Pretrained CNN | nnU-Net (auto-config) | Swin-UNETR \u002F MedSAM |\n\n**Key principle**: architecture matters less than training recipe at equal compute. A well-tuned\nResNet beats a poorly-tuned ViT (ref: \"ResNet Strikes Back\", Wightman 2021).\n\nFor biomedical domains, see `references\u002Fbiomedical.md`.\nFor sequence model selection and compute planning, see `references\u002Fscaling-and-selection.md`.\n\n---\n\n## Scaling Laws\n\n### Chinchilla rule (Hoffmann et al., 2022)\n\nCompute-optimal training: **~20 tokens per parameter**.\n\n| Model Size | Compute-Optimal | Inference-Optimal (100×) |\n|-----------|----------------|--------------------------|\n| 125M | 2.5B tokens | 12.5B tokens |\n| 1B | 20B tokens | 100B tokens |\n| 7B | 140B tokens | 700B tokens |\n\n**FLOPs ≈ 6 × N × D** (N=params, D=tokens). Data repetition limit: ~4 epochs before diminishing returns.\n\n---\n\n## Training Loop\n\n```python\nimport gc, time, torch\n\ntorch.manual_seed(42)\ntorch.set_float32_matmul_precision(\"high\")  # TF32 on Ampere+\nautocast_ctx = torch.amp.autocast(device_type=\"cuda\", dtype=torch.bfloat16)\n\ngrad_accum_steps = total_batch_size \u002F\u002F (batch_size * seq_len)\nstep = 0\n\nwhile not done:\n    t0 = time.time()\n    for micro_step in range(grad_accum_steps):\n        with autocast_ctx:\n            loss = model(x, y)\n        (loss \u002F grad_accum_steps).backward()\n        x, y = next(train_loader)\n\n    update_lr(optimizer, progress)\n    optimizer.step()\n    model.zero_grad(set_to_none=True)  # frees memory vs zeroing\n\n    if loss.item() > 100:  # fast-fail on divergence\n        print(\"FAIL: loss exploded\"); exit(1)\n\n    torch.cuda.synchronize()\n    if step == 0:\n        gc.collect(); gc.freeze(); gc.disable()  # avoid ~500ms GC stalls\n    step += 1\n```\n\n### Key principles\n\n- **Gradient clipping**: `clip_grad_norm_(params, 1.0)` — near-universal for Transformers.\n  Exception: Muon optimizer normalizes updates via orthogonalization, so clipping is optional.\n- **Tensor Core alignment**: batch size, hidden dims should be multiples of 8 (bf16) or 64 (A100).\n- **Time-based budgets** make experiments comparable across hardware.\n- **`cudnn.benchmark = True`** for fixed-size vision inputs.\n\n---\n\n## Optimizer Configuration\n\nModern LLM training uses different optimizers per parameter group:\n\n| Parameter Type | Optimizer | LR (base) | Weight Decay |\n|---------------|-----------|-----------|--------------|\n| 2D weight matrices | Muon | 0.04 | 0.2 |\n| Token embeddings | AdamW | 0.6 × scale | 0.0 |\n| Unembedding (lm_head) | AdamW | 0.004 × scale | 0.0 |\n| Per-layer scalars | AdamW | 0.005 × scale | 0.0 |\n\n**LR scaling by dimension**: `lr * (d_model \u002F 768)^(-0.5)` — keeps dynamics stable across sizes.\n\n### Rules of thumb\n\n- Embeddings need higher LR (sparse updates). Never weight-decay embeddings.\n- Weight decay scheduling: linearly decay WD to 0 over training.\n- AdamW defaults: β1=0.9, β2=0.95, eps=1e-10 (not default 1e-8 — prevents stale updates in bf16).\n\nFor Muon details (polar express orthogonalization, NorMuon), see `references\u002Foptimizers.md`.\n\n---\n\n## Learning Rate Scheduling\n\n### Time-based (autoresearch style)\n\n```python\ndef get_lr_multiplier(progress):  # progress = elapsed_time \u002F time_budget\n    if progress \u003C warmup_ratio:\n        return progress \u002F warmup_ratio\n    elif progress \u003C 1.0 - warmdown_ratio:\n        return 1.0\n    else:\n        cooldown = (1.0 - progress) \u002F warmdown_ratio\n        return cooldown + (1 - cooldown) * final_lr_frac\n```\n\n### Cosine decay\n\n```python\ndef get_lr(step, total_steps, max_lr, min_lr, warmup_steps):\n    if step \u003C warmup_steps:\n        return max_lr * step \u002F warmup_steps\n    progress = (step - warmup_steps) \u002F (total_steps - warmup_steps)\n    return min_lr + 0.5 * (max_lr - min_lr) * (1 + math.cos(math.pi * progress))\n```\n\n**WSD (Warmup-Stable-Decay)**: gaining traction — easier to resume training mid-run.\n\n### Guidance\n\n- **Warmup**: 1-5% of training. Zero warmup valid with Muon (autoresearch uses `WARMUP_RATIO=0.0`).\n- **Warmdown**: 30-50% of training in LR decay. Matters more than warmup for final quality.\n- **Final LR**: 0 or ~10% of peak. Zero is simpler.\n\n---\n\n## Mixed Precision & Compilation\n\n```python\nimport os\nos.environ[\"PYTORCH_ALLOC_CONF\"] = \"expandable_segments:True\"  # before torch import\n\nimport torch\ntorch.set_float32_matmul_precision(\"high\")\nautocast_ctx = torch.amp.autocast(device_type=\"cuda\", dtype=torch.bfloat16)\nmodel = torch.compile(model, dynamic=False)\n```\n\n- **bf16** (Ampere+): same exponent as fp32, no loss scaling needed. Preferred over fp16.\n- **fp16**: needs GradScaler. Use only on V100 or older.\n- `dynamic=False` enables max optimization. Add `fullgraph=True` if no graph breaks.\n- First steps are slow (JIT) — exclude from timing.\n\n---\n\n## Memory & Performance\n\n### Meta device init (large models)\n\n```python\nwith torch.device(\"meta\"):\n    model = GPT(config)          # zero memory\nmodel.to_empty(device=\"cuda\")\nmodel.init_weights()\n```\n\n### MFU (Model FLOPs Utilization)\n\n```python\nachieved_flops = model_flops_per_token * batch_tokens \u002F step_time\nmfu = achieved_flops \u002F gpu_peak_flops\n# H100 SXM: 989.5 TFLOPS | A100: 312 | RTX 4090: 165\n```\n\nGood targets: >30% decent, >40% good, >50% excellent (single-GPU).\n\n### OOM solutions (in order)\n\n1. Reduce `DEVICE_BATCH_SIZE`, increase `grad_accum_steps`\n2. `PYTORCH_ALLOC_CONF=expandable_segments:True`\n3. `model.zero_grad(set_to_none=True)`\n4. Meta device init → `to_empty`\n5. Activation checkpointing: `torch.utils.checkpoint.checkpoint()`\n6. 8-bit optimizer (bitsandbytes): ~30% savings on optimizer states\n\n---\n\n## Hyperparameter Search\n\n### Priority order (tune first → last)\n\n1. **Learning rate** — most impactful. Always tune first.\n2. **Batch size** — largest that fits. Speed knob, not quality knob.\n3. **Weight decay** — 0.01-0.1 for AdamW.\n4. **Warmup steps** — 1-5% of training.\n\n### The 2025 default recipe\n\n| Setting | Value |\n|---------|-------|\n| Optimizer | AdamW (β1=0.9, β2=0.95, eps=1e-10) |\n| Weight decay | 0.1 |\n| LR schedule | Cosine decay or WSD |\n| Peak LR | 3e-4 (scale down for larger models) |\n| Precision | bf16 |\n| Grad clipping | max_norm=1.0 |\n| Normalization | RMSNorm (pre-norm) |\n| Activation | SwiGLU |\n| Position encoding | RoPE |\n| Attention | Flash Attention, optionally GQA |\n\n---\n\n## Debugging Checklist\n\n### Karpathy's recipe (still canonical)\n\n1. **Become one with the data** — visualize, check distributions, verify labels\n2. **Get end-to-end running first** — verify on a trivial case\n3. **Overfit one batch** — if you can't, you have a bug\n4. **Then regularize** — add regularization only after overfitting works\n5. **Tune hyperparameters** — start with known defaults\n\n### Loss exploding \u002F NaN\n\n1. Reduce LR (3-10× smaller)\n2. Add gradient clipping: `clip_grad_norm_(params, 1.0)`\n3. Check for inf\u002Fnan in inputs\n4. Add logit soft capping: `softcap * tanh(logits \u002F softcap)`\n5. Add QK-norm in attention\n6. Verify weight init (zero-init output projections?)\n7. Check loss reduction with gradient accumulation (`loss \u002F grad_accum_steps`)\n\n### Slow training \u002F Low MFU\n\n1. Verify `torch.compile` is active\n2. Check `torch.set_float32_matmul_precision(\"high\")`\n3. Pin memory + non_blocking transfers\n4. Profile with `torch.profiler`\n5. GC stalls? `gc.freeze(); gc.disable()`\n6. Tensor Core alignment: dims multiples of 8\u002F64\n\n### Loss plateau \u002F Slow convergence\n\n1. LR too low — try 2-5× larger\n2. Warmup too long\n3. Weight decay too high\n4. Verify LR schedule is actually applied (print each step)\n5. Model too small for task\n\n### Silent failures\n\n1. **Data leakage** between train\u002Fval\n2. **Wrong preprocessing at inference** — augmentation mismatch\n3. **Label errors** — use cleanlab to detect\n4. **Shuffling bugs** — correlated batches\n5. **Tokenizer mismatch** with pretrained model\n\n### What to monitor\n\n- **Gradient norms** — spike precedes loss spike\n- **Per-layer activation stats** — reveals exploding\u002Fvanishing\n- **Dead neurons** — >50% zero ReLU = dying ReLU problem\n- **Learning rate** — verify schedule applied (common silent bug)\n\n---\n\n## Experiment Management\n\nTrack experiments in TSV for easy comparison:\n\n```\ncommit  val_bpb  memory_gb  status   description\na1b2c3d 0.9979   44.0       keep     baseline\nb2c3d4e 0.9932   44.2       keep     increase matrix LR to 0.04\nc3d4e5f 1.0050   44.0       discard  switch to GeLU (worse)\n```\n\n**Simplicity criterion**: all else equal, simpler is better. Removing something and getting equal\nresults is a great outcome. For systematic agent-driven experimentation, see `references\u002Fexperiment-loop.md`.\n\n### Evaluation metrics by domain\n\n| Domain | Primary Metric | Notes |\n|--------|---------------|-------|\n| LLM | BPB (bits per byte) | Vocab-size-independent |\n| Classification | Accuracy \u002F F1 | Macro-F1 for imbalanced |\n| Segmentation | mIoU \u002F Dice | Per-class IoU reveals weak spots |\n| Generation | FID | Needs >10k samples |\n| Regression | RMSE \u002F MAE | Log-transform skewed targets |\n\n## Related skills\n\n- `serving-llms-vllm`: Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching.\n- `llama-factory`: Guides fine-tuning of large language models with LLaMA-Factory, covering the WebUI no-code interface, training across 100+ supported models…\n- `peft-fine-tuning`: Fine-tunes LLMs with Hugging Face PEFT, using LoRA, QLoRA, IA3, AdaLoRA, prefix tuning, and prompt tuning so that under 1% of parameters ar…\n",{"name":5,"license":15,"metadata":45,"description":7},{"tags":46,"version":47,"category":48,"maintainer":49,"contributor":50,"dependencies":51},"PyTorch, Training, Optimization, LLM, Vision, Diffusion, Biomedical, Muon, AdamW, Debugging","1.0.0","ml-training","Kalaris Labs","dailycafi","torch>=2.0.0","main","2b63f856ce19c2a797929cacabd9e0fc936812fa","plugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fml-training-recipes",491,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins","https:\u002F\u002Fhol.org\u002Fplugins\u002Fbest-codex-plugins","https:\u002F\u002Fgithub.com\u002Fhashgraph-online","https:\u002F\u002Fhol.org","Open-source standards and developer tools for interoperable AI agents: discovery, identity, communication, payments, privacy and security.","2026-11-14 14:06:13",{"source":8,"skillName":5,"commands":63,"prompt":73,"zipAvailable":74},[64,67,70],{"tool":65,"command":66},"npx","npx skills add hashgraph-online\u002Fawesome-codex-plugins --skill ml-training-recipes",{"tool":68,"command":69},"bunx","bunx skills add hashgraph-online\u002Fawesome-codex-plugins --skill ml-training-recipes",{"tool":71,"command":72},"pnpm","pnpm dlx skills add hashgraph-online\u002Fawesome-codex-plugins --skill ml-training-recipes","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fml-training-recipes\u002FSKILL.md",true,[76,82,86,90,94,98],{"path":77,"resourceType":78,"resourceTypeDesc":79,"sizeBytes":80,"sourceHtmlUrl":81},"references\u002Farchitecture.md","REFERENCE","参考文档",10204,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fml-training-recipes\u002Freferences\u002Farchitecture.md",{"path":83,"resourceType":78,"resourceTypeDesc":79,"sizeBytes":84,"sourceHtmlUrl":85},"references\u002Fbiomedical.md",20778,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fml-training-recipes\u002Freferences\u002Fbiomedical.md",{"path":87,"resourceType":78,"resourceTypeDesc":79,"sizeBytes":88,"sourceHtmlUrl":89},"references\u002Fdomain-specific.md",19018,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fml-training-recipes\u002Freferences\u002Fdomain-specific.md",{"path":91,"resourceType":78,"resourceTypeDesc":79,"sizeBytes":92,"sourceHtmlUrl":93},"references\u002Fexperiment-loop.md",4446,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fml-training-recipes\u002Freferences\u002Fexperiment-loop.md",{"path":95,"resourceType":78,"resourceTypeDesc":79,"sizeBytes":96,"sourceHtmlUrl":97},"references\u002Foptimizers.md",10564,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fml-training-recipes\u002Freferences\u002Foptimizers.md",{"path":99,"resourceType":78,"resourceTypeDesc":79,"sizeBytes":100,"sourceHtmlUrl":101},"references\u002Fscaling-and-selection.md",16832,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fml-training-recipes\u002Freferences\u002Fscaling-and-selection.md",[103,112,120,128,136,144],{"skillNo":104,"name":105,"nameZh":6,"description":106,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":107,"sourceHtmlUrl":108,"categories":109,"keywords":110,"publishedAt":111},"01M7DV332YQMGD8WTRJ9DZQXTB","worktable-review-with-annotations","Create, inspect, reply to, update, or resolve situated Worktable annotations. Use when feedback belongs to a Doc section, block, text range, or HTML Doc, or when the user asks to address existing review feedback and coordinate a requested artifact edit.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-review-with-annotations\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-review-with-annotations\u002FSKILL.md",[],[],"2026-11-15 14:19:17",{"skillNo":113,"name":114,"nameZh":6,"description":115,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":116,"sourceHtmlUrl":117,"categories":118,"keywords":119,"publishedAt":111},"01M7DV2YGHCZVBEB1AHYB6CGHE","worktable-find-and-synthesize-context","Find, assess, and synthesize context from Worktable Docs and Records. Use when the user asks what a workspace contains, wants workspace-backed research, needs current facts reconciled across artifacts, or requests a cited synthesis without changing workspace state.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-find-and-synthesize-context\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-find-and-synthesize-context\u002FSKILL.md",[],[],{"skillNo":121,"name":122,"nameZh":6,"description":123,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":124,"sourceHtmlUrl":125,"categories":126,"keywords":127,"publishedAt":111},"01M7DV2TQN1H11ME2SAF9MXN7V","worktable-create-or-update-docs","Create, edit, rename, or maintain narrative Worktable Docs. Use for plans, decisions, briefs, notes, research syntheses, and other durable prose when the user requests a new artifact or a safe update to an existing one.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-update-docs\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-update-docs\u002FSKILL.md",[],[],{"skillNo":129,"name":130,"nameZh":6,"description":131,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":132,"sourceHtmlUrl":133,"categories":134,"keywords":135,"publishedAt":111},"01M7DV2NE452NEA8TXATKSNAJH","worktable-create-or-manage-records","Organize repeated structured information as durable Worktable Records. Use for research datasets, inventories, directories, pipelines, trackers, and other collections whose items need identity, fields, filtering, links, validation, or individual updates.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-records\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-records\u002FSKILL.md",[],[],{"skillNo":137,"name":138,"nameZh":6,"description":139,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":140,"sourceHtmlUrl":141,"categories":142,"keywords":143,"publishedAt":111},"01M7DV2HENRBR2KSCVM064WH0Y","worktable-create-or-manage-interactive-html","Create or improve durable visual and interactive artifacts in Worktable. Use for dashboards, trackers, calculators, mini apps, slide presentations, and other requests that benefit from a polished HTML Doc.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-interactive-html\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-interactive-html\u002FSKILL.md",[],[],{"skillNo":145,"name":146,"nameZh":6,"description":147,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":148,"sourceHtmlUrl":149,"categories":150,"keywords":151,"publishedAt":111},"01M7DV2CADR5R7RQFFYPNAGR9Y","worktable-collaborate-threads","Read, answer, and hand off messages through durable Worktable threads with users or other authenticated participants. Use when the user asks to check addressed messages, reply to a participant, continue a thread, request another agent's help, wait briefly for a response, or watch for comments and messages sent to the agent.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-collaborate-threads\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-collaborate-threads\u002FSKILL.md",[],[]]