[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6XDDFAE003BQ4M7H8ZWCE0W":3,"skill-same:01M6XDDFAE003BQ4M7H8ZWCE0W":90},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":38,"linkOnly":39,"markdownContent":40,"markdownBody":41,"frontMatter":42,"refName":43,"commitSha":44,"skillDirectory":45,"forksCount":13,"repositoryHtmlUrl":46,"repositoryHomepageUrl":6,"ownerHtmlUrl":47,"ownerBlogUrl":6,"ownerDescription":48,"updatedAt":49,"install":50,"resources":63},"01M6XDDFAE003BQ4M7H8ZWCE0W","nanogpt-training",null,"Train GPT-2 scale models (~124M parameters) efficiently on a single GPU. Covers GPT-124M architecture, tokenized dataset loading (e.g., HuggingFace Hub shards), modern optimizers (Muon, AdamW), mixed precision training, and training loop implementation.","Waaaalnut\u002Fskillsbench","Waaaalnut","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F103995656?v=4","01M6XCS9K1V51EY3JMB7DGMX24","01M6XCS9Q4WPW88W62C55G1H44",0,"Apache-2.0","https:\u002F\u002Fgithub.com\u002FWaaaalnut\u002Fskillsbench\u002Fblob\u002Fmain\u002Ftasks-no-skills\u002Fmhc-layer-impl\u002Fenvironment\u002Fskills\u002Fnanogpt-training\u002FSKILL.md",[],[18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37],"training","nanogpt training","NanoGPT","dataset","architecture","single","AdamW","Muon","loading","GPT-2","gpu","scale","gpt","huggingface hub","hub","models","implementation","loop","HuggingFace","Mixed Precision","2026-11-10 10:12:30",false,"---\nname: nanogpt-training\ndescription: Train GPT-2 scale models (~124M parameters) efficiently on a single GPU. Covers GPT-124M architecture, tokenized dataset loading (e.g., HuggingFace Hub shards), modern optimizers (Muon, AdamW), mixed precision training, and training loop implementation.\n---\n\n# NanoGPT Training\n\n## Overview\n\nTraining GPT-2 scale models (~124M parameters) efficiently on a single GPU. It provides:\n\n- **GPT-124M Architecture**: Standard transformer with RoPE and modern optimizations\n- **Tokenized Datasets**: Loading pre-tokenized shards from HuggingFace Hub or local files\n- **Modern Optimizers**: Muon optimizer with Newton-Schulz orthogonalization\n- **Mixed Precision**: bfloat16 training on A100 for 2x speedup\n\nTraining options:\n- **Baseline GPT**: Standard residual connections\n- **Experimental residual variants**: Optional alternative residual schemes for stability\u002Fefficiency\n\n## Quick Reference\n\n| Topic | Reference |\n|-------|-----------|\n| Model Architecture | [GPT Architecture](references\u002Fgpt-architecture.md) |\n| Data Loading | [Tokenized Data](references\u002Ftokenized-data.md) |\n| Optimizers | [Optimizers](references\u002Foptimizers.md) |\n| Training Loop | [Training Loop](references\u002Ftraining-loop.md) |\n| Hyperparameters | [Hyperparameters](references\u002Fhyperparameters.md) |\n\n## Installation\n\n```bash\npip install torch einops numpy huggingface_hub\n```\n\n## Minimal Example\n\n```python\nimport modal\n\napp = modal.App(\"gpt-training\")\n\nimage = modal.Image.debian_slim(python_version=\"3.11\").pip_install(\n    \"torch\", \"einops\", \"numpy\", \"huggingface_hub\"\n)\n\n@app.function(gpu=\"A100\", image=image, timeout=3600)\ndef train():\n    import torch\n    from dataclasses import dataclass\n\n    @dataclass\n    class GPTConfig:\n        block_size: int = 1024\n        vocab_size: int = 50257\n        n_layer: int = 12\n        n_head: int = 12\n        n_embd: int = 768\n        dropout: float = 0.0\n        bias: bool = False\n\n    # Download data, build model, train\n    # ... (see references for full implementation)\n\n    return {\"final_loss\": final_loss}\n\n@app.local_entrypoint()\ndef main():\n    results = train.remote()\n    print(results)\n```\n\n## Common Imports\n\n```python\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.cuda.amp import autocast, GradScaler\nfrom dataclasses import dataclass\nfrom einops import rearrange, repeat, reduce\nimport numpy as np\nimport math\n```\n\n## When to Use What\n\n| Scenario | Approach |\n|----------|----------|\n| Standard GPT training | Use baseline model with standard residuals |\n| Stability experiments | Try alternative residual variants or extra streams |\n| Small experiments | Use T4\u002FA10G GPU |\n| Full training | Use A100 with bfloat16 |\n| Custom data | Modify the dataset loader class |\n| Different model size | Adjust GPTConfig parameters |\n\n## Metrics to Monitor\n\n| Metric | Typical Signal | Notes |\n|--------|----------------|-------|\n| Validation loss | Steady decrease | Absolute value depends on dataset\u002Ftokenizer |\n| Grad norm | Moderate, stable range | Large spikes indicate instability |\n| Training stability | Smooth curves | Frequent spikes suggest LR\u002Fbatch issues |\n| Throughput | Consistent tokens\u002Fsec | Use for comparing configs |\n\n## External Resources\n\n- nanoGPT: https:\u002F\u002Fgithub.com\u002Fkarpathy\u002FnanoGPT\n- build-nanogpt: https:\u002F\u002Fgithub.com\u002Fkarpathy\u002Fbuild-nanogpt\n- modded-nanogpt: https:\u002F\u002Fgithub.com\u002FKellerJordan\u002Fmodded-nanogpt\n- FineWeb-Edu token shards: https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Fkarpathy\u002Ffineweb-edu-100B-gpt2-token-shards\n","# NanoGPT Training\n\n## Overview\n\nTraining GPT-2 scale models (~124M parameters) efficiently on a single GPU. It provides:\n\n- **GPT-124M Architecture**: Standard transformer with RoPE and modern optimizations\n- **Tokenized Datasets**: Loading pre-tokenized shards from HuggingFace Hub or local files\n- **Modern Optimizers**: Muon optimizer with Newton-Schulz orthogonalization\n- **Mixed Precision**: bfloat16 training on A100 for 2x speedup\n\nTraining options:\n- **Baseline GPT**: Standard residual connections\n- **Experimental residual variants**: Optional alternative residual schemes for stability\u002Fefficiency\n\n## Quick Reference\n\n| Topic | Reference |\n|-------|-----------|\n| Model Architecture | [GPT Architecture](references\u002Fgpt-architecture.md) |\n| Data Loading | [Tokenized Data](references\u002Ftokenized-data.md) |\n| Optimizers | [Optimizers](references\u002Foptimizers.md) |\n| Training Loop | [Training Loop](references\u002Ftraining-loop.md) |\n| Hyperparameters | [Hyperparameters](references\u002Fhyperparameters.md) |\n\n## Installation\n\n```bash\npip install torch einops numpy huggingface_hub\n```\n\n## Minimal Example\n\n```python\nimport modal\n\napp = modal.App(\"gpt-training\")\n\nimage = modal.Image.debian_slim(python_version=\"3.11\").pip_install(\n    \"torch\", \"einops\", \"numpy\", \"huggingface_hub\"\n)\n\n@app.function(gpu=\"A100\", image=image, timeout=3600)\ndef train():\n    import torch\n    from dataclasses import dataclass\n\n    @dataclass\n    class GPTConfig:\n        block_size: int = 1024\n        vocab_size: int = 50257\n        n_layer: int = 12\n        n_head: int = 12\n        n_embd: int = 768\n        dropout: float = 0.0\n        bias: bool = False\n\n    # Download data, build model, train\n    # ... (see references for full implementation)\n\n    return {\"final_loss\": final_loss}\n\n@app.local_entrypoint()\ndef main():\n    results = train.remote()\n    print(results)\n```\n\n## Common Imports\n\n```python\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.cuda.amp import autocast, GradScaler\nfrom dataclasses import dataclass\nfrom einops import rearrange, repeat, reduce\nimport numpy as np\nimport math\n```\n\n## When to Use What\n\n| Scenario | Approach |\n|----------|----------|\n| Standard GPT training | Use baseline model with standard residuals |\n| Stability experiments | Try alternative residual variants or extra streams |\n| Small experiments | Use T4\u002FA10G GPU |\n| Full training | Use A100 with bfloat16 |\n| Custom data | Modify the dataset loader class |\n| Different model size | Adjust GPTConfig parameters |\n\n## Metrics to Monitor\n\n| Metric | Typical Signal | Notes |\n|--------|----------------|-------|\n| Validation loss | Steady decrease | Absolute value depends on dataset\u002Ftokenizer |\n| Grad norm | Moderate, stable range | Large spikes indicate instability |\n| Training stability | Smooth curves | Frequent spikes suggest LR\u002Fbatch issues |\n| Throughput | Consistent tokens\u002Fsec | Use for comparing configs |\n\n## External Resources\n\n- nanoGPT: https:\u002F\u002Fgithub.com\u002Fkarpathy\u002FnanoGPT\n- build-nanogpt: https:\u002F\u002Fgithub.com\u002Fkarpathy\u002Fbuild-nanogpt\n- modded-nanogpt: https:\u002F\u002Fgithub.com\u002FKellerJordan\u002Fmodded-nanogpt\n- FineWeb-Edu token shards: https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Fkarpathy\u002Ffineweb-edu-100B-gpt2-token-shards\n",{"name":5,"description":7},"main","7eeb258ada72b7f58119b0e7a2c00ea89d9beedd","tasks-no-skills\u002Fmhc-layer-impl\u002Fenvironment\u002Fskills\u002Fnanogpt-training","https:\u002F\u002Fgithub.com\u002FWaaaalnut\u002Fskillsbench","https:\u002F\u002Fgithub.com\u002FWaaaalnut","student of Zhejiang University","2026-04-21 23:16:23",{"source":8,"skillName":5,"commands":51,"prompt":61,"zipAvailable":62},[52,55,58],{"tool":53,"command":54},"npx","npx skills add Waaaalnut\u002Fskillsbench --skill nanogpt-training",{"tool":56,"command":57},"bunx","bunx skills add Waaaalnut\u002Fskillsbench --skill nanogpt-training",{"tool":59,"command":60},"pnpm","pnpm dlx skills add Waaaalnut\u002Fskillsbench --skill nanogpt-training","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FWaaaalnut\u002Fskillsbench\u002Fblob\u002Fmain\u002Ftasks-no-skills\u002Fmhc-layer-impl\u002Fenvironment\u002Fskills\u002Fnanogpt-training\u002FSKILL.md",true,[64,70,74,78,82,86],{"path":65,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":68,"sourceHtmlUrl":69},"references\u002Ffineweb-data.md","REFERENCE","参考文档",3609,"https:\u002F\u002Fgithub.com\u002FWaaaalnut\u002Fskillsbench\u002Fblob\u002Fmain\u002Ftasks-no-skills\u002Fmhc-layer-impl\u002Fenvironment\u002Fskills\u002Fnanogpt-training\u002Freferences\u002Ffineweb-data.md",{"path":71,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":72,"sourceHtmlUrl":73},"references\u002Fgpt-architecture.md",4630,"https:\u002F\u002Fgithub.com\u002FWaaaalnut\u002Fskillsbench\u002Fblob\u002Fmain\u002Ftasks-no-skills\u002Fmhc-layer-impl\u002Fenvironment\u002Fskills\u002Fnanogpt-training\u002Freferences\u002Fgpt-architecture.md",{"path":75,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":76,"sourceHtmlUrl":77},"references\u002Fhyperparameters.md",2760,"https:\u002F\u002Fgithub.com\u002FWaaaalnut\u002Fskillsbench\u002Fblob\u002Fmain\u002Ftasks-no-skills\u002Fmhc-layer-impl\u002Fenvironment\u002Fskills\u002Fnanogpt-training\u002Freferences\u002Fhyperparameters.md",{"path":79,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":80,"sourceHtmlUrl":81},"references\u002Foptimizers.md",3077,"https:\u002F\u002Fgithub.com\u002FWaaaalnut\u002Fskillsbench\u002Fblob\u002Fmain\u002Ftasks-no-skills\u002Fmhc-layer-impl\u002Fenvironment\u002Fskills\u002Fnanogpt-training\u002Freferences\u002Foptimizers.md",{"path":83,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":84,"sourceHtmlUrl":85},"references\u002Ftokenized-data.md",4129,"https:\u002F\u002Fgithub.com\u002FWaaaalnut\u002Fskillsbench\u002Fblob\u002Fmain\u002Ftasks-no-skills\u002Fmhc-layer-impl\u002Fenvironment\u002Fskills\u002Fnanogpt-training\u002Freferences\u002Ftokenized-data.md",{"path":87,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":88,"sourceHtmlUrl":89},"references\u002Ftraining-loop.md",3648,"https:\u002F\u002Fgithub.com\u002FWaaaalnut\u002Fskillsbench\u002Fblob\u002Fmain\u002Ftasks-no-skills\u002Fmhc-layer-impl\u002Fenvironment\u002Fskills\u002Fnanogpt-training\u002Freferences\u002Ftraining-loop.md",[91,105,120,148,170,189],{"skillNo":92,"name":93,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":94,"categories":95,"keywords":101,"publishedAt":38},"01M6XEWMJ6VY3ZC53MPDCPG3GH","data-reconciliation","https:\u002F\u002Fgithub.com\u002FWaaaalnut\u002Fskillsbench\u002Fblob\u002Fmain\u002Ftasks\u002Fxlsx-recover-data\u002Fenvironment\u002Fskills\u002Fdata-reconciliation\u002FSKILL.md",[96],{"categoryNo":97,"categoryName":98,"taxonomyType":99,"taxonomyTypeDesc":100},"01JBSEEDCAT000000000000033","数据分析师","PROFESSION","适用职业",[102,103,104],"data reconciliation","reconciliation","data",{"skillNo":106,"name":107,"nameZh":6,"description":108,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":109,"categories":110,"keywords":111,"publishedAt":38},"01M6XEW8MNPMF1KRXV6ATGFRQF","speech-to-text","Transcribe video to timestamped text using Whisper tiny model (pre-installed).","https:\u002F\u002Fgithub.com\u002FWaaaalnut\u002Fskillsbench\u002Fblob\u002Fmain\u002Ftasks\u002Fvideo-tutorial-indexer\u002Fenvironment\u002Fskills\u002Fspeech-to-text\u002FSKILL.md",[],[112,113,114,115,116,117,118,119],"speech","text","speech to text","transcribe video","whisper","model","Video","transcribe",{"skillNo":121,"name":122,"nameZh":6,"description":123,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":124,"categories":125,"keywords":131,"publishedAt":38},"01M6XEW3FQZP0T1RKQQJFXVY9T","video-processor","Process videos by removing segments and concatenating remaining parts. Use when you need to remove detected pauses\u002Fopenings from videos, create highlight reels, or batch process segment removals using ffmpeg filter_complex.","https:\u002F\u002Fgithub.com\u002FWaaaalnut\u002Fskillsbench\u002Fblob\u002Fmain\u002Ftasks\u002Fvideo-silence-remover\u002Fenvironment\u002Fskills\u002Fvideo-processor\u002FSKILL.md",[126],{"categoryNo":127,"categoryName":128,"taxonomyType":129,"taxonomyTypeDesc":130},"01JBSEEDCAT000000000000016","图像与多媒体","SCENARIO","使用场景",[118,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147],"processor","video processor","batch","create","reels","when","parts","segments","process","videos","you","segment","filter","ffmpeg","use","remove",{"skillNo":149,"name":150,"nameZh":6,"description":151,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":152,"categories":153,"keywords":155,"publishedAt":38},"01M6XEVXJ5KZ0NTY3XZKRY4AS1","silence-detector","Detect initial silence segments in audio\u002Fvideo using energy-based analysis. Use when you need to find low-energy periods at the start of recordings (title slides, setup time, pre-roll silence).","https:\u002F\u002Fgithub.com\u002FWaaaalnut\u002Fskillsbench\u002Fblob\u002Fmain\u002Ftasks\u002Fvideo-silence-remover\u002Fenvironment\u002Fskills\u002Fsilence-detector\u002FSKILL.md",[154],{"categoryNo":127,"categoryName":128,"taxonomyType":129,"taxonomyTypeDesc":130},[156,157,158,159,137,142,160,161,162,163,164,139,165,166,118,167,168,169,146],"detector","silence detector","title","slides","time","energy","at","recordings","detect","find","Audio","analysis","start","setup",{"skillNo":171,"name":172,"nameZh":6,"description":173,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":174,"categories":175,"keywords":176,"publishedAt":38},"01M6XEVT3NGRN1FT7JPMAPXS6M","segment-combiner","Combine multiple segment detection results into a unified list. Use when you need to merge segments from different detectors, prepare removal lists for video processing, or consolidate detection outputs.","https:\u002F\u002Fgithub.com\u002FWaaaalnut\u002Fskillsbench\u002Fblob\u002Fmain\u002Ftasks\u002Fvideo-silence-remover\u002Fenvironment\u002Fskills\u002Fsegment-combiner\u002FSKILL.md",[],[143,177,118,178,179,180,181,182,137,183,184,139,142,185,186,146,187,188],"segment combiner","detection","prepare","merge","video processing","for","outputs","combine","processing","consolidate","results","list",{"skillNo":190,"name":191,"nameZh":6,"description":192,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":193,"categories":194,"keywords":198,"publishedAt":38},"01M6XEVMPVASGS2KX86QK4FMT7","report-generator","Generate compression reports for video processing. Use when you need to create structured JSON reports with duration statistics, compression ratios, and segment details after video processing.","https:\u002F\u002Fgithub.com\u002FWaaaalnut\u002Fskillsbench\u002Fblob\u002Fmain\u002Ftasks\u002Fvideo-silence-remover\u002Fenvironment\u002Fskills\u002Freport-generator\u002FSKILL.md",[195],{"categoryNo":196,"categoryName":197,"taxonomyType":99,"taxonomyTypeDesc":100},"01JBSEEDCAT000000000000034","行政与办公人员",[199,200,201,143,137,182,142,202,185,203,204,205,146,181,206,118,207,208,135],"generator","report generator","report","with","statistics","structured","reports","json","compression","generate"]