[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6KQREENHGJXAM4QK9DN6BD5":3,"skill-same:01M6KQREENHGJXAM4QK9DN6BD5":95},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":57,"commitSha":58,"skillDirectory":59,"forksCount":13,"repositoryHtmlUrl":60,"repositoryHomepageUrl":6,"ownerHtmlUrl":61,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":62,"install":63,"resources":76},"01M6KQREENHGJXAM4QK9DN6BD5","distributed-llm-pretraining-torchtitan",null,"Provides PyTorch-native distributed LLM pretraining using torchtitan with 4D parallelism (FSDP2, TP, PP, CP). Use when pretraining Llama 3.1, DeepSeek V3, or custom models at scale from 8 to 512+ GPUs with Float8, torch.compile, and distributed checkpointing.","endman100\u002Fskill-router-nexus","endman100","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F31997019?v=4","01M6KPDX17BYDWW4RRB0TNGGMX","01M6KPDX64Y2QBEWVFG5W5PWPF",0,"MIT","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Training\u002Ftorchtitan\u002Ftorchtitan\u002FSKILL.md",[],[18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"distributed","Pretraining","torchtitan","distributed llm pretraining torchtitan","FSDP2","when","checkpointing","orchestra","at","pp","native","architecture","pytorch","compile","Model Architecture","training","context","parallel","model","pipeline","Distributed Training","2026-11-10 10:12:30",false,"---\nname: distributed-llm-pretraining-torchtitan\ndescription: Provides PyTorch-native distributed LLM pretraining using torchtitan with 4D parallelism (FSDP2, TP, PP, CP). Use when pretraining Llama 3.1, DeepSeek V3, or custom models at scale from 8 to 512+ GPUs with Float8, torch.compile, and distributed checkpointing.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Model Architecture, Distributed Training, TorchTitan, FSDP2, Tensor Parallel, Pipeline Parallel, Context Parallel, Float8, Llama, Pretraining]\ndependencies: [torch>=2.6.0, torchtitan>=0.2.0, torchao>=0.5.0]\n---\n\n# TorchTitan - PyTorch Native Distributed LLM Pretraining\n\n## Quick start\n\nTorchTitan is PyTorch's official platform for large-scale LLM pretraining with composable 4D parallelism (FSDP2, TP, PP, CP), achieving 65%+ speedups over baselines on H100 GPUs.\n\n**Installation**:\n```bash\n# From PyPI (stable)\npip install torchtitan\n\n# From source (latest features, requires PyTorch nightly)\ngit clone https:\u002F\u002Fgithub.com\u002Fpytorch\u002Ftorchtitan\ncd torchtitan\npip install -r requirements.txt\n```\n\n**Download tokenizer**:\n```bash\n# Get HF token from https:\u002F\u002Fhuggingface.co\u002Fsettings\u002Ftokens\npython scripts\u002Fdownload_hf_assets.py --repo_id meta-llama\u002FLlama-3.1-8B --assets tokenizer --hf_token=...\n```\n\n**Start training on 8 GPUs**:\n```bash\nCONFIG_FILE=\".\u002Ftorchtitan\u002Fmodels\u002Fllama3\u002Ftrain_configs\u002Fllama3_8b.toml\" .\u002Frun_train.sh\n```\n\n## Common workflows\n\n### Workflow 1: Pretrain Llama 3.1 8B on single node\n\nCopy this checklist:\n\n```\nSingle Node Pretraining:\n- [ ] Step 1: Download tokenizer\n- [ ] Step 2: Configure training\n- [ ] Step 3: Launch training\n- [ ] Step 4: Monitor and checkpoint\n```\n\n**Step 1: Download tokenizer**\n\n```bash\npython scripts\u002Fdownload_hf_assets.py \\\n  --repo_id meta-llama\u002FLlama-3.1-8B \\\n  --assets tokenizer \\\n  --hf_token=YOUR_HF_TOKEN\n```\n\n**Step 2: Configure training**\n\nEdit or create a TOML config file:\n\n```toml\n# llama3_8b_custom.toml\n[job]\ndump_folder = \".\u002Foutputs\"\ndescription = \"Llama 3.1 8B training\"\n\n[model]\nname = \"llama3\"\nflavor = \"8B\"\nhf_assets_path = \".\u002Fassets\u002Fhf\u002FLlama-3.1-8B\"\n\n[optimizer]\nname = \"AdamW\"\nlr = 3e-4\n\n[lr_scheduler]\nwarmup_steps = 200\n\n[training]\nlocal_batch_size = 2\nseq_len = 8192\nmax_norm = 1.0\nsteps = 1000\ndataset = \"c4\"\n\n[parallelism]\ndata_parallel_shard_degree = -1  # Use all GPUs for FSDP\n\n[activation_checkpoint]\nmode = \"selective\"\nselective_ac_option = \"op\"\n\n[checkpoint]\nenable = true\nfolder = \"checkpoint\"\ninterval = 500\n```\n\n**Step 3: Launch training**\n\n```bash\n# 8 GPUs on single node\nCONFIG_FILE=\".\u002Fllama3_8b_custom.toml\" .\u002Frun_train.sh\n\n# Or explicitly with torchrun\ntorchrun --nproc_per_node=8 \\\n  -m torchtitan.train \\\n  --job.config_file .\u002Fllama3_8b_custom.toml\n```\n\n**Step 4: Monitor and checkpoint**\n\nTensorBoard logs are saved to `.\u002Foutputs\u002Ftb\u002F`:\n```bash\ntensorboard --logdir .\u002Foutputs\u002Ftb\n```\n\n### Workflow 2: Multi-node training with SLURM\n\n```\nMulti-Node Training:\n- [ ] Step 1: Configure parallelism for scale\n- [ ] Step 2: Set up SLURM script\n- [ ] Step 3: Submit job\n- [ ] Step 4: Resume from checkpoint\n```\n\n**Step 1: Configure parallelism for scale**\n\nFor 70B model on 256 GPUs (32 nodes):\n```toml\n[parallelism]\ndata_parallel_shard_degree = 32  # FSDP across 32 ranks\ntensor_parallel_degree = 8        # TP within node\npipeline_parallel_degree = 1      # No PP for 70B\ncontext_parallel_degree = 1       # Increase for long sequences\n```\n\n**Step 2: Set up SLURM script**\n\n```bash\n#!\u002Fbin\u002Fbash\n#SBATCH --job-name=llama70b\n#SBATCH --nodes=32\n#SBATCH --ntasks-per-node=8\n#SBATCH --gpus-per-node=8\n\nsrun torchrun \\\n  --nnodes=32 \\\n  --nproc_per_node=8 \\\n  --rdzv_backend=c10d \\\n  --rdzv_endpoint=$MASTER_ADDR:$MASTER_PORT \\\n  -m torchtitan.train \\\n  --job.config_file .\u002Fllama3_70b.toml\n```\n\n**Step 3: Submit job**\n\n```bash\nsbatch multinode_trainer.slurm\n```\n\n**Step 4: Resume from checkpoint**\n\nTraining auto-resumes if checkpoint exists in configured folder.\n\n### Workflow 3: Enable Float8 training for H100s\n\nFloat8 provides 30-50% speedup on H100 GPUs.\n\n```\nFloat8 Training:\n- [ ] Step 1: Install torchao\n- [ ] Step 2: Configure Float8\n- [ ] Step 3: Launch with compile\n```\n\n**Step 1: Install torchao**\n\n```bash\nUSE_CPP=0 pip install git+https:\u002F\u002Fgithub.com\u002Fpytorch\u002Fao.git\n```\n\n**Step 2: Configure Float8**\n\nAdd to your TOML config:\n```toml\n[model]\nconverters = [\"quantize.linear.float8\"]\n\n[quantize.linear.float8]\nenable_fsdp_float8_all_gather = true\nprecompute_float8_dynamic_scale_for_fsdp = true\nfilter_fqns = [\"output\"]  # Exclude output layer\n\n[compile]\nenable = true\ncomponents = [\"model\", \"loss\"]\n```\n\n**Step 3: Launch with compile**\n\n```bash\nCONFIG_FILE=\".\u002Fllama3_8b.toml\" .\u002Frun_train.sh \\\n  --model.converters=\"quantize.linear.float8\" \\\n  --quantize.linear.float8.enable_fsdp_float8_all_gather \\\n  --compile.enable\n```\n\n### Workflow 4: 4D parallelism for 405B models\n\n```\n4D Parallelism (FSDP + TP + PP + CP):\n- [ ] Step 1: Create seed checkpoint\n- [ ] Step 2: Configure 4D parallelism\n- [ ] Step 3: Launch on 512 GPUs\n```\n\n**Step 1: Create seed checkpoint**\n\nRequired for consistent initialization across PP stages:\n```bash\nNGPU=1 CONFIG_FILE=.\u002Fllama3_405b.toml .\u002Frun_train.sh \\\n  --checkpoint.enable \\\n  --checkpoint.create_seed_checkpoint \\\n  --parallelism.data_parallel_shard_degree 1 \\\n  --parallelism.tensor_parallel_degree 1 \\\n  --parallelism.pipeline_parallel_degree 1\n```\n\n**Step 2: Configure 4D parallelism**\n\n```toml\n[parallelism]\ndata_parallel_shard_degree = 8   # FSDP\ntensor_parallel_degree = 8       # TP within node\npipeline_parallel_degree = 8     # PP across nodes\ncontext_parallel_degree = 1      # CP for long sequences\n\n[training]\nlocal_batch_size = 32\nseq_len = 8192\n```\n\n**Step 3: Launch on 512 GPUs**\n\n```bash\n# 64 nodes x 8 GPUs = 512 GPUs\nsrun torchrun --nnodes=64 --nproc_per_node=8 \\\n  -m torchtitan.train \\\n  --job.config_file .\u002Fllama3_405b.toml\n```\n\n## When to use vs alternatives\n\n**Use TorchTitan when:**\n- Pretraining LLMs from scratch (8B to 405B+)\n- Need PyTorch-native solution without third-party dependencies\n- Require composable 4D parallelism (FSDP2, TP, PP, CP)\n- Training on H100s with Float8 support\n- Want interoperable checkpoints with torchtune\u002FHuggingFace\n\n**Use alternatives instead:**\n- **Megatron-LM**: Maximum performance for NVIDIA-only deployments\n- **DeepSpeed**: Broader ZeRO optimization ecosystem, inference support\n- **Axolotl\u002FTRL**: Fine-tuning rather than pretraining\n- **LitGPT**: Educational, smaller-scale training\n\n## Common issues\n\n**Issue: Out of memory on large models**\n\nEnable activation checkpointing and reduce batch size:\n```toml\n[activation_checkpoint]\nmode = \"full\"  # Instead of \"selective\"\n\n[training]\nlocal_batch_size = 1\n```\n\nOr use gradient accumulation:\n```toml\n[training]\nlocal_batch_size = 1\nglobal_batch_size = 32  # Accumulates gradients\n```\n\n**Issue: TP causes high memory with async collectives**\n\nSet environment variable:\n```bash\nexport TORCH_NCCL_AVOID_RECORD_STREAMS=1\n```\n\n**Issue: Float8 training not faster**\n\nFloat8 only benefits large GEMMs. Filter small layers:\n```toml\n[quantize.linear.float8]\nfilter_fqns = [\"attention.wk\", \"attention.wv\", \"output\", \"auto_filter_small_kn\"]\n```\n\n**Issue: Checkpoint loading fails after parallelism change**\n\nUse DCP's resharding capability:\n```bash\n# Convert sharded checkpoint to single file\npython -m torch.distributed.checkpoint.format_utils \\\n  dcp_to_torch checkpoint\u002Fstep-1000 checkpoint.pt\n```\n\n**Issue: Pipeline parallelism initialization**\n\nCreate seed checkpoint first (see Workflow 4, Step 1).\n\n## Supported models\n\n| Model | Sizes | Status |\n|-------|-------|--------|\n| Llama 3.1 | 8B, 70B, 405B | Production |\n| Llama 4 | Various | Experimental |\n| DeepSeek V3 | 16B, 236B, 671B (MoE) | Experimental |\n| GPT-OSS | 20B, 120B (MoE) | Experimental |\n| Qwen 3 | Various | Experimental |\n| Flux | Diffusion | Experimental |\n\n## Performance benchmarks (H100)\n\n| Model | GPUs | Parallelism | TPS\u002FGPU | Techniques |\n|-------|------|-------------|---------|------------|\n| Llama 8B | 8 | FSDP | 5,762 | Baseline |\n| Llama 8B | 8 | FSDP+compile+FP8 | 8,532 | +48% |\n| Llama 70B | 256 | FSDP+TP+AsyncTP | 876 | 2D parallel |\n| Llama 405B | 512 | FSDP+TP+PP | 128 | 3D parallel |\n\n## Advanced topics\n\n**FSDP2 configuration**: See [references\u002Ffsdp.md](references\u002Ffsdp.md) for detailed FSDP2 vs FSDP1 comparison and ZeRO equivalents.\n\n**Float8 training**: See [references\u002Ffloat8.md](references\u002Ffloat8.md) for tensorwise vs rowwise scaling recipes.\n\n**Checkpointing**: See [references\u002Fcheckpoint.md](references\u002Fcheckpoint.md) for HuggingFace conversion and async checkpointing.\n\n**Adding custom models**: See [references\u002Fcustom-models.md](references\u002Fcustom-models.md) for TrainSpec protocol.\n\n## Resources\n\n- GitHub: https:\u002F\u002Fgithub.com\u002Fpytorch\u002Ftorchtitan\n- Paper: https:\u002F\u002Farxiv.org\u002Fabs\u002F2410.06511\n- ICLR 2025: https:\u002F\u002Ficlr.cc\u002Fvirtual\u002F2025\u002Fposter\u002F29620\n- PyTorch Forum: https:\u002F\u002Fdiscuss.pytorch.org\u002Fc\u002Fdistributed\u002Ftorchtitan\u002F44\n\n","# TorchTitan - PyTorch Native Distributed LLM Pretraining\n\n## Quick start\n\nTorchTitan is PyTorch's official platform for large-scale LLM pretraining with composable 4D parallelism (FSDP2, TP, PP, CP), achieving 65%+ speedups over baselines on H100 GPUs.\n\n**Installation**:\n```bash\n# From PyPI (stable)\npip install torchtitan\n\n# From source (latest features, requires PyTorch nightly)\ngit clone https:\u002F\u002Fgithub.com\u002Fpytorch\u002Ftorchtitan\ncd torchtitan\npip install -r requirements.txt\n```\n\n**Download tokenizer**:\n```bash\n# Get HF token from https:\u002F\u002Fhuggingface.co\u002Fsettings\u002Ftokens\npython scripts\u002Fdownload_hf_assets.py --repo_id meta-llama\u002FLlama-3.1-8B --assets tokenizer --hf_token=...\n```\n\n**Start training on 8 GPUs**:\n```bash\nCONFIG_FILE=\".\u002Ftorchtitan\u002Fmodels\u002Fllama3\u002Ftrain_configs\u002Fllama3_8b.toml\" .\u002Frun_train.sh\n```\n\n## Common workflows\n\n### Workflow 1: Pretrain Llama 3.1 8B on single node\n\nCopy this checklist:\n\n```\nSingle Node Pretraining:\n- [ ] Step 1: Download tokenizer\n- [ ] Step 2: Configure training\n- [ ] Step 3: Launch training\n- [ ] Step 4: Monitor and checkpoint\n```\n\n**Step 1: Download tokenizer**\n\n```bash\npython scripts\u002Fdownload_hf_assets.py \\\n  --repo_id meta-llama\u002FLlama-3.1-8B \\\n  --assets tokenizer \\\n  --hf_token=YOUR_HF_TOKEN\n```\n\n**Step 2: Configure training**\n\nEdit or create a TOML config file:\n\n```toml\n# llama3_8b_custom.toml\n[job]\ndump_folder = \".\u002Foutputs\"\ndescription = \"Llama 3.1 8B training\"\n\n[model]\nname = \"llama3\"\nflavor = \"8B\"\nhf_assets_path = \".\u002Fassets\u002Fhf\u002FLlama-3.1-8B\"\n\n[optimizer]\nname = \"AdamW\"\nlr = 3e-4\n\n[lr_scheduler]\nwarmup_steps = 200\n\n[training]\nlocal_batch_size = 2\nseq_len = 8192\nmax_norm = 1.0\nsteps = 1000\ndataset = \"c4\"\n\n[parallelism]\ndata_parallel_shard_degree = -1  # Use all GPUs for FSDP\n\n[activation_checkpoint]\nmode = \"selective\"\nselective_ac_option = \"op\"\n\n[checkpoint]\nenable = true\nfolder = \"checkpoint\"\ninterval = 500\n```\n\n**Step 3: Launch training**\n\n```bash\n# 8 GPUs on single node\nCONFIG_FILE=\".\u002Fllama3_8b_custom.toml\" .\u002Frun_train.sh\n\n# Or explicitly with torchrun\ntorchrun --nproc_per_node=8 \\\n  -m torchtitan.train \\\n  --job.config_file .\u002Fllama3_8b_custom.toml\n```\n\n**Step 4: Monitor and checkpoint**\n\nTensorBoard logs are saved to `.\u002Foutputs\u002Ftb\u002F`:\n```bash\ntensorboard --logdir .\u002Foutputs\u002Ftb\n```\n\n### Workflow 2: Multi-node training with SLURM\n\n```\nMulti-Node Training:\n- [ ] Step 1: Configure parallelism for scale\n- [ ] Step 2: Set up SLURM script\n- [ ] Step 3: Submit job\n- [ ] Step 4: Resume from checkpoint\n```\n\n**Step 1: Configure parallelism for scale**\n\nFor 70B model on 256 GPUs (32 nodes):\n```toml\n[parallelism]\ndata_parallel_shard_degree = 32  # FSDP across 32 ranks\ntensor_parallel_degree = 8        # TP within node\npipeline_parallel_degree = 1      # No PP for 70B\ncontext_parallel_degree = 1       # Increase for long sequences\n```\n\n**Step 2: Set up SLURM script**\n\n```bash\n#!\u002Fbin\u002Fbash\n#SBATCH --job-name=llama70b\n#SBATCH --nodes=32\n#SBATCH --ntasks-per-node=8\n#SBATCH --gpus-per-node=8\n\nsrun torchrun \\\n  --nnodes=32 \\\n  --nproc_per_node=8 \\\n  --rdzv_backend=c10d \\\n  --rdzv_endpoint=$MASTER_ADDR:$MASTER_PORT \\\n  -m torchtitan.train \\\n  --job.config_file .\u002Fllama3_70b.toml\n```\n\n**Step 3: Submit job**\n\n```bash\nsbatch multinode_trainer.slurm\n```\n\n**Step 4: Resume from checkpoint**\n\nTraining auto-resumes if checkpoint exists in configured folder.\n\n### Workflow 3: Enable Float8 training for H100s\n\nFloat8 provides 30-50% speedup on H100 GPUs.\n\n```\nFloat8 Training:\n- [ ] Step 1: Install torchao\n- [ ] Step 2: Configure Float8\n- [ ] Step 3: Launch with compile\n```\n\n**Step 1: Install torchao**\n\n```bash\nUSE_CPP=0 pip install git+https:\u002F\u002Fgithub.com\u002Fpytorch\u002Fao.git\n```\n\n**Step 2: Configure Float8**\n\nAdd to your TOML config:\n```toml\n[model]\nconverters = [\"quantize.linear.float8\"]\n\n[quantize.linear.float8]\nenable_fsdp_float8_all_gather = true\nprecompute_float8_dynamic_scale_for_fsdp = true\nfilter_fqns = [\"output\"]  # Exclude output layer\n\n[compile]\nenable = true\ncomponents = [\"model\", \"loss\"]\n```\n\n**Step 3: Launch with compile**\n\n```bash\nCONFIG_FILE=\".\u002Fllama3_8b.toml\" .\u002Frun_train.sh \\\n  --model.converters=\"quantize.linear.float8\" \\\n  --quantize.linear.float8.enable_fsdp_float8_all_gather \\\n  --compile.enable\n```\n\n### Workflow 4: 4D parallelism for 405B models\n\n```\n4D Parallelism (FSDP + TP + PP + CP):\n- [ ] Step 1: Create seed checkpoint\n- [ ] Step 2: Configure 4D parallelism\n- [ ] Step 3: Launch on 512 GPUs\n```\n\n**Step 1: Create seed checkpoint**\n\nRequired for consistent initialization across PP stages:\n```bash\nNGPU=1 CONFIG_FILE=.\u002Fllama3_405b.toml .\u002Frun_train.sh \\\n  --checkpoint.enable \\\n  --checkpoint.create_seed_checkpoint \\\n  --parallelism.data_parallel_shard_degree 1 \\\n  --parallelism.tensor_parallel_degree 1 \\\n  --parallelism.pipeline_parallel_degree 1\n```\n\n**Step 2: Configure 4D parallelism**\n\n```toml\n[parallelism]\ndata_parallel_shard_degree = 8   # FSDP\ntensor_parallel_degree = 8       # TP within node\npipeline_parallel_degree = 8     # PP across nodes\ncontext_parallel_degree = 1      # CP for long sequences\n\n[training]\nlocal_batch_size = 32\nseq_len = 8192\n```\n\n**Step 3: Launch on 512 GPUs**\n\n```bash\n# 64 nodes x 8 GPUs = 512 GPUs\nsrun torchrun --nnodes=64 --nproc_per_node=8 \\\n  -m torchtitan.train \\\n  --job.config_file .\u002Fllama3_405b.toml\n```\n\n## When to use vs alternatives\n\n**Use TorchTitan when:**\n- Pretraining LLMs from scratch (8B to 405B+)\n- Need PyTorch-native solution without third-party dependencies\n- Require composable 4D parallelism (FSDP2, TP, PP, CP)\n- Training on H100s with Float8 support\n- Want interoperable checkpoints with torchtune\u002FHuggingFace\n\n**Use alternatives instead:**\n- **Megatron-LM**: Maximum performance for NVIDIA-only deployments\n- **DeepSpeed**: Broader ZeRO optimization ecosystem, inference support\n- **Axolotl\u002FTRL**: Fine-tuning rather than pretraining\n- **LitGPT**: Educational, smaller-scale training\n\n## Common issues\n\n**Issue: Out of memory on large models**\n\nEnable activation checkpointing and reduce batch size:\n```toml\n[activation_checkpoint]\nmode = \"full\"  # Instead of \"selective\"\n\n[training]\nlocal_batch_size = 1\n```\n\nOr use gradient accumulation:\n```toml\n[training]\nlocal_batch_size = 1\nglobal_batch_size = 32  # Accumulates gradients\n```\n\n**Issue: TP causes high memory with async collectives**\n\nSet environment variable:\n```bash\nexport TORCH_NCCL_AVOID_RECORD_STREAMS=1\n```\n\n**Issue: Float8 training not faster**\n\nFloat8 only benefits large GEMMs. Filter small layers:\n```toml\n[quantize.linear.float8]\nfilter_fqns = [\"attention.wk\", \"attention.wv\", \"output\", \"auto_filter_small_kn\"]\n```\n\n**Issue: Checkpoint loading fails after parallelism change**\n\nUse DCP's resharding capability:\n```bash\n# Convert sharded checkpoint to single file\npython -m torch.distributed.checkpoint.format_utils \\\n  dcp_to_torch checkpoint\u002Fstep-1000 checkpoint.pt\n```\n\n**Issue: Pipeline parallelism initialization**\n\nCreate seed checkpoint first (see Workflow 4, Step 1).\n\n## Supported models\n\n| Model | Sizes | Status |\n|-------|-------|--------|\n| Llama 3.1 | 8B, 70B, 405B | Production |\n| Llama 4 | Various | Experimental |\n| DeepSeek V3 | 16B, 236B, 671B (MoE) | Experimental |\n| GPT-OSS | 20B, 120B (MoE) | Experimental |\n| Qwen 3 | Various | Experimental |\n| Flux | Diffusion | Experimental |\n\n## Performance benchmarks (H100)\n\n| Model | GPUs | Parallelism | TPS\u002FGPU | Techniques |\n|-------|------|-------------|---------|------------|\n| Llama 8B | 8 | FSDP | 5,762 | Baseline |\n| Llama 8B | 8 | FSDP+compile+FP8 | 8,532 | +48% |\n| Llama 70B | 256 | FSDP+TP+AsyncTP | 876 | 2D parallel |\n| Llama 405B | 512 | FSDP+TP+PP | 128 | 3D parallel |\n\n## Advanced topics\n\n**FSDP2 configuration**: See [references\u002Ffsdp.md](references\u002Ffsdp.md) for detailed FSDP2 vs FSDP1 comparison and ZeRO equivalents.\n\n**Float8 training**: See [references\u002Ffloat8.md](references\u002Ffloat8.md) for tensorwise vs rowwise scaling recipes.\n\n**Checkpointing**: See [references\u002Fcheckpoint.md](references\u002Fcheckpoint.md) for HuggingFace conversion and async checkpointing.\n\n**Adding custom models**: See [references\u002Fcustom-models.md](references\u002Fcustom-models.md) for TrainSpec protocol.\n\n## Resources\n\n- GitHub: https:\u002F\u002Fgithub.com\u002Fpytorch\u002Ftorchtitan\n- Paper: https:\u002F\u002Farxiv.org\u002Fabs\u002F2410.06511\n- ICLR 2025: https:\u002F\u002Ficlr.cc\u002Fvirtual\u002F2025\u002Fposter\u002F29620\n- PyTorch Forum: https:\u002F\u002Fdiscuss.pytorch.org\u002Fc\u002Fdistributed\u002Ftorchtitan\u002F44\n\n",{"name":5,"tags":44,"author":51,"license":14,"version":52,"description":7,"dependencies":53},[32,38,45,22,46,47,48,49,50,19],"TorchTitan","Tensor Parallel","Pipeline Parallel","Context Parallel","Float8","Llama","Orchestra Research","1.0.0",[54,55,56],"torch>=2.6.0","torchtitan>=0.2.0","torchao>=0.5.0","main","f19b254e8975d0bfddb394686706106930d7964d","AI-Training\u002Ftorchtitan\u002Ftorchtitan","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus","https:\u002F\u002Fgithub.com\u002Fendman100","2026-11-13 20:29:52",{"source":8,"skillName":5,"commands":64,"prompt":74,"zipAvailable":75},[65,68,71],{"tool":66,"command":67},"npx","npx skills add endman100\u002Fskill-router-nexus --skill distributed-llm-pretraining-torchtitan",{"tool":69,"command":70},"bunx","bunx skills add endman100\u002Fskill-router-nexus --skill distributed-llm-pretraining-torchtitan",{"tool":72,"command":73},"pnpm","pnpm dlx skills add endman100\u002Fskill-router-nexus --skill distributed-llm-pretraining-torchtitan","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Training\u002Ftorchtitan\u002Ftorchtitan\u002FSKILL.md",true,[77,83,87,91],{"path":78,"resourceType":79,"resourceTypeDesc":80,"sizeBytes":81,"sourceHtmlUrl":82},"references\u002Fcheckpoint.md","REFERENCE","参考文档",4216,"https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Training\u002Ftorchtitan\u002Ftorchtitan\u002Freferences\u002Fcheckpoint.md",{"path":84,"resourceType":79,"resourceTypeDesc":80,"sizeBytes":85,"sourceHtmlUrl":86},"references\u002Fcustom-models.md",7383,"https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Training\u002Ftorchtitan\u002Ftorchtitan\u002Freferences\u002Fcustom-models.md",{"path":88,"resourceType":79,"resourceTypeDesc":80,"sizeBytes":89,"sourceHtmlUrl":90},"references\u002Ffloat8.md",4055,"https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Training\u002Ftorchtitan\u002Ftorchtitan\u002Freferences\u002Ffloat8.md",{"path":92,"resourceType":79,"resourceTypeDesc":80,"sizeBytes":93,"sourceHtmlUrl":94},"references\u002Ffsdp.md",3888,"https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FAI-Training\u002Ftorchtitan\u002Ftorchtitan\u002Freferences\u002Ffsdp.md",[96,120,157,179,188,212],{"skillNo":97,"name":98,"nameZh":6,"description":99,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":100,"sourceHtmlUrl":101,"categories":102,"keywords":108,"publishedAt":39},"01M6MPCT5BXVER8HQZTW3QE9AS","create-master","基于佛教经典文献，生成特定高僧大德的 AI 教学角色","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FMetaphysics-and-Spirituality\u002FMaster-skill\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FMetaphysics-and-Spirituality\u002FMaster-skill\u002FSKILL.md",[103],{"categoryNo":104,"categoryName":105,"taxonomyType":106,"taxonomyTypeDesc":107},"01JBSEEDCAT000000000000039","教师与教育","PROFESSION","适用职业",[109,110,111,112,113,114,115,116,117,118,119],"create master","master","create","生成","true","webfetch","edit","write","read","grep","Bash",{"skillNo":121,"name":122,"nameZh":6,"description":123,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":124,"sourceHtmlUrl":125,"categories":126,"keywords":138,"publishedAt":39},"01M6M43AKE95Y64E1JRXEF5QQ3","writing-dna-skill","从至少 20 篇完整文章中蒸馏可复用的写作 DNA，分析语言、文章结构、选题逻辑、素材策略、认知框架和视觉风格，并生成 Writing-DNA.md；按该 DNA 写作时会读取全部蒸馏产物和 5 篇相关原文。用于中英文作者、账号、品牌或出版物的风格分析与一致性写作。 Distill reusable Writing DNA from at least 20 complete articles for Chinese or English authors, publications, brands, and accounts; use for language, structure, topic logic, source strategy, cognitive-frame, and visual-style analysis.","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fwriting-dna-skill\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fwriting-dna-skill\u002FSKILL.md",[127,132,135],{"categoryNo":128,"categoryName":129,"taxonomyType":130,"taxonomyTypeDesc":131},"01JBSEEDCAT000000000000018","写作与文案","SCENARIO","使用场景",{"categoryNo":133,"categoryName":134,"taxonomyType":130,"taxonomyTypeDesc":131},"01JBSEEDCAT000000000000023","Skill与智能体开发",{"categoryNo":136,"categoryName":137,"taxonomyType":106,"taxonomyTypeDesc":107},"01JBSEEDCAT000000000000036","内容创作者",[139,140,141,142,143,144,145,146,147,148,149,150,151,112,152,153,154,155,26,156],"writing dna skill","DNA","writing","strategy","english","logic","visual","structure","chinese","reusable","use","style","distill","analysis","frame","source","for","complete",{"skillNo":158,"name":159,"nameZh":6,"description":160,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":161,"sourceHtmlUrl":162,"categories":163,"keywords":166,"publishedAt":39},"01M6M428TTCGQ4ME3KY99VN037","stop-slop","Remove AI writing patterns from prose. Use when drafting, editing, or reviewing text to eliminate predictable AI tells.","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fstop-slop\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fstop-slop\u002FSKILL.md",[164,165],{"categoryNo":128,"categoryName":129,"taxonomyType":130,"taxonomyTypeDesc":131},{"categoryNo":136,"categoryName":137,"taxonomyType":106,"taxonomyTypeDesc":107},[167,168,169,170,23,171,172,173,174,149,175,176,177,155,178,141],"slop","stop slop","stop","drafting","drafts","https","patterns","prose","remove","editing","text","content",{"skillNo":180,"name":181,"nameZh":6,"description":182,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":183,"sourceHtmlUrl":184,"categories":185,"keywords":186,"publishedAt":39},"01M6M41X0B7EBX42TK435DMGGR","shuorenhua","检查和清理中英文文本里的 AI 套路，适用于“去 AI 味”“说人话”“自然一点”“别像模板”“先标问题”这类改写和审稿需求；按场景控制力度，同时保留事实、术语、语域和责任主体。","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fshuorenhua\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fshuorenhua\u002FSKILL.md",[],[181,187],"文本",{"skillNo":189,"name":190,"nameZh":6,"description":191,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":192,"sourceHtmlUrl":193,"categories":194,"keywords":195,"publishedAt":39},"01M6M3ZD12WBEW5KXKDM4N6DFG","naming","Name products, SaaS, brands, open source projects, bots, and apps. Use when the user needs to name something, find a brand name, or pick a product name. Metaphor-driven process that produces memorable, meaningful names and avoids AI slop.","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fnaming-main\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fnaming-main\u002FSKILL.md",[],[190,196,197,198,199,200,149,201,117,154,202,203,204,205,119,206,207,208,209,23,210,211,167],"repo","websearch","product name","apps","bots","view","brand","pick","gh","projects","open","saas","open source","product","describe","ai slop",{"skillNo":213,"name":214,"nameZh":6,"description":215,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":216,"sourceHtmlUrl":217,"categories":218,"keywords":220,"publishedAt":39},"01M6M3YVXAFAAWJ7F6JMPFREYE","humanizer-zh","去除文本中的 AI 生成痕迹。适用于编辑或审阅文本，使其听起来更自然、更像人类书写。\n基于维基百科的\"AI 写作特征\"综合指南。检测并修复以下模式：夸大的象征意义、\n宣传性语言、以 -ing 结尾的肤浅分析、模糊的归因、破折号过度使用、三段式法则、\nAI 词汇、否定式排比、过多的连接性短语。","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fhumanizer-zh\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fendman100\u002Fskill-router-nexus\u002Fblob\u002Fmain\u002FWriting-and-Content\u002Fhumanizer-zh\u002FSKILL.md",[219],{"categoryNo":128,"categoryName":129,"taxonomyType":130,"taxonomyTypeDesc":131},[221,222,223,117,116,115,141,112,187,224,167,169],"humanizer","zh","humanizer zh","翻译"]