[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M73ABW1ZP5C1EQM1F3SZ0S8H":3,"skill-same:01M73ABW1ZP5C1EQM1F3SZ0S8H":111},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":39,"commitSha":40,"skillDirectory":41,"forksCount":42,"repositoryHtmlUrl":43,"repositoryHomepageUrl":6,"ownerHtmlUrl":44,"ownerBlogUrl":45,"ownerDescription":6,"updatedAt":46,"install":47,"resources":60},"01M73ABW1ZP5C1EQM1F3SZ0S8H","pytorch-fsdp2",null,"Adds PyTorch FSDP2 (fully_shard) to training scripts with correct init, sharding, mixed precision\u002Foffload config, and distributed checkpointing. Use when models exceed single-GPU memory or when you need DTensor-based sharding with DeviceMesh.","qhjqhj00\u002Fresearch-skills-pool","qhjqhj00","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F29684092?v=4","01M72AE0ZGXVM5C57N512DGQG8","Tommy Chien","01M72AE107ESJMHT7QC5SXX144",3,"MIT","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--08-distributed-training__pytorch-fsdp2\u002FSKILL.md",[],[],"2026-11-11 15:11:14",false,"---\nname: pytorch-fsdp2\ndescription: Adds PyTorch FSDP2 (fully_shard) to training scripts with correct init, sharding, mixed precision\u002Foffload config, and distributed checkpointing. Use when models exceed single-GPU memory or when you need DTensor-based sharding with DeviceMesh.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [PyTorch, FSDP2, Fully Sharded Data Parallel, Distributed Training, DTensor, Device Mesh, Sharded Checkpointing, Mixed Precision, Offload, Torch Distributed]\ndependencies: [torch]\n---\n\n# Skill: Use PyTorch FSDP2 (`fully_shard`) correctly in a training script\n\nThis skill teaches a coding agent how to **add PyTorch FSDP2** to a training loop with correct initialization, sharding, mixed precision\u002Foffload configuration, and checkpointing.\n\n> FSDP2 in PyTorch is exposed primarily via `torch.distributed.fsdp.fully_shard` and the `FSDPModule` methods it adds in-place to modules. See: `references\u002Fpytorch_fully_shard_api.md`, `references\u002Fpytorch_fsdp2_tutorial.md`.\n\n---\n\n## When to use this skill\n\nUse FSDP2 when:\n- Your model **doesn’t fit** on one GPU (parameters + gradients + optimizer state).\n- You want an eager-mode sharding approach that is **DTensor-based per-parameter sharding** (more inspectable, simpler sharded state dicts) than FSDP1.  \n- You may later compose DP with **Tensor Parallel** using **DeviceMesh**.\n\nAvoid (or be careful) if:\n- You need strict backwards-compatible checkpoints across PyTorch versions (DCP warns against this).\n- You’re forced onto older PyTorch versions without the FSDP2 stack.\n\n## Alternatives (when FSDP2 is not the best fit)\n\n- **DistributedDataParallel (DDP)**: Use the standard data-parallel wrapper when you want classic distributed data parallel training.\n- **FullyShardedDataParallel (FSDP1)**: Use the original FSDP wrapper for parameter sharding across data-parallel workers.\n\nReference: `references\u002Fpytorch_ddp_notes.md`, `references\u002Fpytorch_fsdp1_api.md`.\n\n---\n\n## Contract the agent must follow\n\n1. **Launch with `torchrun`** and set the CUDA device per process (usually via `LOCAL_RANK`).  \n2. **Apply `fully_shard()` bottom-up**, i.e., shard submodules (e.g., Transformer blocks) before the root module.  \n3. **Call `model(input)`**, not `model.forward(input)`, so the FSDP2 hooks run (unless you explicitly `unshard()` or register the forward method).  \n4. **Create the optimizer after sharding** and make sure it is built on the **DTensor parameters** (post-`fully_shard`).  \n5. **Checkpoint using Distributed Checkpoint (DCP)** or the distributed-state-dict helpers, not naïve `torch.save(model.state_dict())` unless you deliberately gather to full tensors.\n\n(Each of these rules is directly described in the official API docs\u002Ftutorial; see references.)\n\n---\n\n## Step-by-step procedure\n\n### 0) Version & environment sanity\n- Prefer a recent stable PyTorch where the docs show FSDP2 and DCP updated recently.\n- Use `torchrun --nproc_per_node \u003Cgpus_per_node> ...` and ensure `RANK`, `WORLD_SIZE`, `LOCAL_RANK` are visible.\n\nReference: `references\u002Fpytorch_fsdp2_tutorial.md` (launch commands and setup), `references\u002Fpytorch_fully_shard_api.md` (user contract).\n\n---\n\n### 1) Initialize distributed and set device\nMinimal, correct pattern:\n- `dist.init_process_group(backend=\"nccl\")`\n- `torch.cuda.set_device(int(os.environ[\"LOCAL_RANK\"]))`\n- Optionally create a `DeviceMesh` to describe the data-parallel group(s)\n\nReference: `references\u002Fpytorch_device_mesh_tutorial.md` (why DeviceMesh exists & how it manages process groups).\n\n---\n\n### 2) Build model on meta device (recommended for very large models)\nFor big models, initialize on `meta`, apply sharding, then materialize weights on GPU:\n- `with torch.device(\"meta\"): model = ...`\n- apply `fully_shard(...)` on submodules, then `fully_shard(model)`\n- `model.to_empty(device=\"cuda\")`\n- `model.reset_parameters()` (or your init routine)\n\nReference: `references\u002Fpytorch_fsdp2_tutorial.md` (migration guide shows this flow explicitly).\n\n---\n\n### 3) Apply `fully_shard()` bottom-up (wrapping policy = “apply where needed”)\n**Do not** only call `fully_shard` on the topmost module.\n\nRecommended sharding pattern for transformer-like models:\n- iterate modules, `if isinstance(m, TransformerBlock): fully_shard(m, ...)`\n- then `fully_shard(model, ...)`\n\nWhy:\n- `fully_shard` forms “parameter groups” for collective efficiency and excludes params already grouped by earlier calls. Bottom-up gives better overlap and lower peak memory.\n\nReference: `references\u002Fpytorch_fully_shard_api.md` (bottom-up requirement and why).\n\n---\n\n### 4) Configure `reshard_after_forward` for memory\u002Fperf trade-offs\nDefault behavior:\n- `None` means `True` for non-root modules and `False` for root modules (good default).\n\nHeuristics:\n- If you’re memory-bound: keep defaults or force `True` on many blocks.\n- If you’re throughput-bound and can afford memory: consider keeping unsharded params longer (root often `False`).\n- Advanced: use an `int` to reshard to a smaller mesh after forward (e.g., intra-node) if it’s a meaningful divisor.\n\nReference: `references\u002Fpytorch_fully_shard_api.md` (full semantics).\n\n---\n\n### 5) Mixed precision & offload (optional but common)\nFSDP2 uses:\n- `mp_policy=MixedPrecisionPolicy(param_dtype=..., reduce_dtype=..., output_dtype=..., cast_forward_inputs=...)`\n- `offload_policy=CPUOffloadPolicy()` if you want CPU offload\n\nRules of thumb:\n- Start with BF16 parameters\u002Freductions on H100\u002FA100-class GPUs (if numerically stable for your model).\n- Keep `reduce_dtype` aligned with your gradient reduction expectations.\n- If you use CPU offload, budget for PCIe\u002FNVLink traffic and runtime overhead.\n\nReference: `references\u002Fpytorch_fully_shard_api.md` (MixedPrecisionPolicy \u002F OffloadPolicy classes).\n\n---\n\n### 6) Optimizer, gradient clipping, accumulation\n- Create the optimizer **after** sharding so it holds DTensor params.\n- If you need gradient accumulation \u002F no_sync:\n  - use the FSDP2 mechanism (`set_requires_gradient_sync`) instead of FSDP1’s `no_sync()`.\n\nGradient clipping:\n- Use the approach shown in the FSDP2 tutorial (“Gradient Clipping and Optimizer with DTensor”), because parameters\u002Fgradients are DTensors.\n\nReference: `references\u002Fpytorch_fsdp2_tutorial.md`.\n\n---\n\n### 7) Checkpointing: prefer DCP or distributed state dict helpers\nTwo recommended approaches:\n\n**A) Distributed Checkpoint (DCP) — best default**\n- DCP saves\u002Floads from multiple ranks in parallel and supports load-time resharding.\n- DCP produces **multiple files** (often at least one per rank) and operates “in place”.\n\n**B) Distributed state dict helpers**\n- `get_model_state_dict` \u002F `set_model_state_dict` with `StateDictOptions(full_state_dict=True, cpu_offload=True, broadcast_from_rank0=True, ...)`\n- For optimizer: `get_optimizer_state_dict` \u002F `set_optimizer_state_dict`\n\nAvoid:\n- Saving DTensor state dicts with plain `torch.save` unless you intentionally convert with `DTensor.full_tensor()` and manage memory carefully.\n\nReferences:\n- `references\u002Fpytorch_dcp_overview.md` (DCP behavior and caveats)\n- `references\u002Fpytorch_dcp_recipe.md` and `references\u002Fpytorch_dcp_async_recipe.md` (end-to-end usage)\n- `references\u002Fpytorch_fsdp2_tutorial.md` (DTensor vs DCP state-dict flows)\n- `references\u002Fpytorch_examples_fsdp2.md` (working checkpoint scripts)\n\n---\n\n## Workflow checklists (copy-paste friendly)\n\n### Workflow A: Retrofit FSDP2 into an existing training script\n- [ ] Launch with `torchrun` and initialize the process group.\n- [ ] Set the CUDA device from `LOCAL_RANK`; create a `DeviceMesh` if you need multi-dim parallelism.\n- [ ] Build the model (use `meta` if needed), apply `fully_shard` bottom-up, then `fully_shard(model)`.\n- [ ] Create the optimizer after sharding so it captures DTensor parameters.\n- [ ] Use `model(inputs)` so hooks run; use `set_requires_gradient_sync` for accumulation.\n- [ ] Add DCP save\u002Fload via `torch.distributed.checkpoint` helpers.\n\nReference: `references\u002Fpytorch_fsdp2_tutorial.md`, `references\u002Fpytorch_fully_shard_api.md`, `references\u002Fpytorch_device_mesh_tutorial.md`, `references\u002Fpytorch_dcp_recipe.md`.\n\n### Workflow B: Add DCP save\u002Fload (minimal pattern)\n- [ ] Wrap state in `Stateful` or assemble state via `get_state_dict`.\n- [ ] Call `dcp.save(...)` from all ranks to a shared path.\n- [ ] Call `dcp.load(...)` and restore with `set_state_dict`.\n- [ ] Validate any resharding assumptions when loading into a different mesh.\n\nReference: `references\u002Fpytorch_dcp_recipe.md`.\n\n## Debug checklist (what the agent should check first)\n\n1. **All ranks on distinct GPUs?**  \n   If not, verify `torch.cuda.set_device(LOCAL_RANK)` and your `torchrun` flags.\n2. **Did you accidentally call `forward()` directly?**  \n   Use `model(input)` or explicitly `unshard()` \u002F register forward.\n3. **Is `fully_shard()` applied bottom-up?**  \n   If only root is sharded, expect worse memory\u002Fperf and possible confusion.\n4. **Optimizer created at the right time?**  \n   Must be built on DTensor parameters *after* sharding.\n5. **Checkpointing path consistent?**  \n   - If using DCP, don’t mix with ad-hoc `torch.save` unless you understand conversions.\n   - Be mindful of PyTorch-version compatibility warnings for DCP.\n\n---\n\n## Common issues and fixes\n\n- **Forward hooks not running** → Call `model(inputs)` (or `unshard()` explicitly) instead of `model.forward(...)`.\n- **Optimizer sees non-DTensor params** → Create optimizer after all `fully_shard` calls.\n- **Only root module sharded** → Apply `fully_shard` bottom-up on submodules before the root.\n- **Memory spikes after forward** → Set `reshard_after_forward=True` for more modules.\n- **Gradient accumulation desync** → Use `set_requires_gradient_sync` instead of FSDP1’s `no_sync()`.\n\nReference: `references\u002Fpytorch_fully_shard_api.md`, `references\u002Fpytorch_fsdp2_tutorial.md`.\n\n---\n\n## Minimal reference implementation outline (agent-friendly)\n\nThe coding agent should implement a script with these labeled blocks:\n\n- `init_distributed()`: init process group, set device\n- `build_model_meta()`: model on meta, apply `fully_shard`, materialize weights\n- `build_optimizer()`: optimizer created after sharding\n- `train_step()`: forward\u002Fbackward\u002Fstep with `model(inputs)` and DTensor-aware patterns\n- `checkpoint_save\u002Fload()`: DCP or distributed state dict helpers\n\nConcrete examples live in `references\u002Fpytorch_examples_fsdp2.md` and the official tutorial reference.\n\n---\n\n## References\n- `references\u002Fpytorch_fsdp2_tutorial.md`\n- `references\u002Fpytorch_fully_shard_api.md`\n- `references\u002Fpytorch_ddp_notes.md`\n- `references\u002Fpytorch_fsdp1_api.md`\n- `references\u002Fpytorch_device_mesh_tutorial.md`\n- `references\u002Fpytorch_tp_tutorial.md`\n- `references\u002Fpytorch_dcp_overview.md`\n- `references\u002Fpytorch_dcp_recipe.md`\n- `references\u002Fpytorch_dcp_async_recipe.md`\n- `references\u002Fpytorch_examples_fsdp2.md`\n- `references\u002Ftorchtitan_fsdp_notes.md` (optional, production notes)\n- `references\u002Fray_train_fsdp2_example.md` (optional, integration example)\n","# Skill: Use PyTorch FSDP2 (`fully_shard`) correctly in a training script\n\nThis skill teaches a coding agent how to **add PyTorch FSDP2** to a training loop with correct initialization, sharding, mixed precision\u002Foffload configuration, and checkpointing.\n\n> FSDP2 in PyTorch is exposed primarily via `torch.distributed.fsdp.fully_shard` and the `FSDPModule` methods it adds in-place to modules. See: `references\u002Fpytorch_fully_shard_api.md`, `references\u002Fpytorch_fsdp2_tutorial.md`.\n\n---\n\n## When to use this skill\n\nUse FSDP2 when:\n- Your model **doesn’t fit** on one GPU (parameters + gradients + optimizer state).\n- You want an eager-mode sharding approach that is **DTensor-based per-parameter sharding** (more inspectable, simpler sharded state dicts) than FSDP1.  \n- You may later compose DP with **Tensor Parallel** using **DeviceMesh**.\n\nAvoid (or be careful) if:\n- You need strict backwards-compatible checkpoints across PyTorch versions (DCP warns against this).\n- You’re forced onto older PyTorch versions without the FSDP2 stack.\n\n## Alternatives (when FSDP2 is not the best fit)\n\n- **DistributedDataParallel (DDP)**: Use the standard data-parallel wrapper when you want classic distributed data parallel training.\n- **FullyShardedDataParallel (FSDP1)**: Use the original FSDP wrapper for parameter sharding across data-parallel workers.\n\nReference: `references\u002Fpytorch_ddp_notes.md`, `references\u002Fpytorch_fsdp1_api.md`.\n\n---\n\n## Contract the agent must follow\n\n1. **Launch with `torchrun`** and set the CUDA device per process (usually via `LOCAL_RANK`).  \n2. **Apply `fully_shard()` bottom-up**, i.e., shard submodules (e.g., Transformer blocks) before the root module.  \n3. **Call `model(input)`**, not `model.forward(input)`, so the FSDP2 hooks run (unless you explicitly `unshard()` or register the forward method).  \n4. **Create the optimizer after sharding** and make sure it is built on the **DTensor parameters** (post-`fully_shard`).  \n5. **Checkpoint using Distributed Checkpoint (DCP)** or the distributed-state-dict helpers, not naïve `torch.save(model.state_dict())` unless you deliberately gather to full tensors.\n\n(Each of these rules is directly described in the official API docs\u002Ftutorial; see references.)\n\n---\n\n## Step-by-step procedure\n\n### 0) Version & environment sanity\n- Prefer a recent stable PyTorch where the docs show FSDP2 and DCP updated recently.\n- Use `torchrun --nproc_per_node \u003Cgpus_per_node> ...` and ensure `RANK`, `WORLD_SIZE`, `LOCAL_RANK` are visible.\n\nReference: `references\u002Fpytorch_fsdp2_tutorial.md` (launch commands and setup), `references\u002Fpytorch_fully_shard_api.md` (user contract).\n\n---\n\n### 1) Initialize distributed and set device\nMinimal, correct pattern:\n- `dist.init_process_group(backend=\"nccl\")`\n- `torch.cuda.set_device(int(os.environ[\"LOCAL_RANK\"]))`\n- Optionally create a `DeviceMesh` to describe the data-parallel group(s)\n\nReference: `references\u002Fpytorch_device_mesh_tutorial.md` (why DeviceMesh exists & how it manages process groups).\n\n---\n\n### 2) Build model on meta device (recommended for very large models)\nFor big models, initialize on `meta`, apply sharding, then materialize weights on GPU:\n- `with torch.device(\"meta\"): model = ...`\n- apply `fully_shard(...)` on submodules, then `fully_shard(model)`\n- `model.to_empty(device=\"cuda\")`\n- `model.reset_parameters()` (or your init routine)\n\nReference: `references\u002Fpytorch_fsdp2_tutorial.md` (migration guide shows this flow explicitly).\n\n---\n\n### 3) Apply `fully_shard()` bottom-up (wrapping policy = “apply where needed”)\n**Do not** only call `fully_shard` on the topmost module.\n\nRecommended sharding pattern for transformer-like models:\n- iterate modules, `if isinstance(m, TransformerBlock): fully_shard(m, ...)`\n- then `fully_shard(model, ...)`\n\nWhy:\n- `fully_shard` forms “parameter groups” for collective efficiency and excludes params already grouped by earlier calls. Bottom-up gives better overlap and lower peak memory.\n\nReference: `references\u002Fpytorch_fully_shard_api.md` (bottom-up requirement and why).\n\n---\n\n### 4) Configure `reshard_after_forward` for memory\u002Fperf trade-offs\nDefault behavior:\n- `None` means `True` for non-root modules and `False` for root modules (good default).\n\nHeuristics:\n- If you’re memory-bound: keep defaults or force `True` on many blocks.\n- If you’re throughput-bound and can afford memory: consider keeping unsharded params longer (root often `False`).\n- Advanced: use an `int` to reshard to a smaller mesh after forward (e.g., intra-node) if it’s a meaningful divisor.\n\nReference: `references\u002Fpytorch_fully_shard_api.md` (full semantics).\n\n---\n\n### 5) Mixed precision & offload (optional but common)\nFSDP2 uses:\n- `mp_policy=MixedPrecisionPolicy(param_dtype=..., reduce_dtype=..., output_dtype=..., cast_forward_inputs=...)`\n- `offload_policy=CPUOffloadPolicy()` if you want CPU offload\n\nRules of thumb:\n- Start with BF16 parameters\u002Freductions on H100\u002FA100-class GPUs (if numerically stable for your model).\n- Keep `reduce_dtype` aligned with your gradient reduction expectations.\n- If you use CPU offload, budget for PCIe\u002FNVLink traffic and runtime overhead.\n\nReference: `references\u002Fpytorch_fully_shard_api.md` (MixedPrecisionPolicy \u002F OffloadPolicy classes).\n\n---\n\n### 6) Optimizer, gradient clipping, accumulation\n- Create the optimizer **after** sharding so it holds DTensor params.\n- If you need gradient accumulation \u002F no_sync:\n  - use the FSDP2 mechanism (`set_requires_gradient_sync`) instead of FSDP1’s `no_sync()`.\n\nGradient clipping:\n- Use the approach shown in the FSDP2 tutorial (“Gradient Clipping and Optimizer with DTensor”), because parameters\u002Fgradients are DTensors.\n\nReference: `references\u002Fpytorch_fsdp2_tutorial.md`.\n\n---\n\n### 7) Checkpointing: prefer DCP or distributed state dict helpers\nTwo recommended approaches:\n\n**A) Distributed Checkpoint (DCP) — best default**\n- DCP saves\u002Floads from multiple ranks in parallel and supports load-time resharding.\n- DCP produces **multiple files** (often at least one per rank) and operates “in place”.\n\n**B) Distributed state dict helpers**\n- `get_model_state_dict` \u002F `set_model_state_dict` with `StateDictOptions(full_state_dict=True, cpu_offload=True, broadcast_from_rank0=True, ...)`\n- For optimizer: `get_optimizer_state_dict` \u002F `set_optimizer_state_dict`\n\nAvoid:\n- Saving DTensor state dicts with plain `torch.save` unless you intentionally convert with `DTensor.full_tensor()` and manage memory carefully.\n\nReferences:\n- `references\u002Fpytorch_dcp_overview.md` (DCP behavior and caveats)\n- `references\u002Fpytorch_dcp_recipe.md` and `references\u002Fpytorch_dcp_async_recipe.md` (end-to-end usage)\n- `references\u002Fpytorch_fsdp2_tutorial.md` (DTensor vs DCP state-dict flows)\n- `references\u002Fpytorch_examples_fsdp2.md` (working checkpoint scripts)\n\n---\n\n## Workflow checklists (copy-paste friendly)\n\n### Workflow A: Retrofit FSDP2 into an existing training script\n- [ ] Launch with `torchrun` and initialize the process group.\n- [ ] Set the CUDA device from `LOCAL_RANK`; create a `DeviceMesh` if you need multi-dim parallelism.\n- [ ] Build the model (use `meta` if needed), apply `fully_shard` bottom-up, then `fully_shard(model)`.\n- [ ] Create the optimizer after sharding so it captures DTensor parameters.\n- [ ] Use `model(inputs)` so hooks run; use `set_requires_gradient_sync` for accumulation.\n- [ ] Add DCP save\u002Fload via `torch.distributed.checkpoint` helpers.\n\nReference: `references\u002Fpytorch_fsdp2_tutorial.md`, `references\u002Fpytorch_fully_shard_api.md`, `references\u002Fpytorch_device_mesh_tutorial.md`, `references\u002Fpytorch_dcp_recipe.md`.\n\n### Workflow B: Add DCP save\u002Fload (minimal pattern)\n- [ ] Wrap state in `Stateful` or assemble state via `get_state_dict`.\n- [ ] Call `dcp.save(...)` from all ranks to a shared path.\n- [ ] Call `dcp.load(...)` and restore with `set_state_dict`.\n- [ ] Validate any resharding assumptions when loading into a different mesh.\n\nReference: `references\u002Fpytorch_dcp_recipe.md`.\n\n## Debug checklist (what the agent should check first)\n\n1. **All ranks on distinct GPUs?**  \n   If not, verify `torch.cuda.set_device(LOCAL_RANK)` and your `torchrun` flags.\n2. **Did you accidentally call `forward()` directly?**  \n   Use `model(input)` or explicitly `unshard()` \u002F register forward.\n3. **Is `fully_shard()` applied bottom-up?**  \n   If only root is sharded, expect worse memory\u002Fperf and possible confusion.\n4. **Optimizer created at the right time?**  \n   Must be built on DTensor parameters *after* sharding.\n5. **Checkpointing path consistent?**  \n   - If using DCP, don’t mix with ad-hoc `torch.save` unless you understand conversions.\n   - Be mindful of PyTorch-version compatibility warnings for DCP.\n\n---\n\n## Common issues and fixes\n\n- **Forward hooks not running** → Call `model(inputs)` (or `unshard()` explicitly) instead of `model.forward(...)`.\n- **Optimizer sees non-DTensor params** → Create optimizer after all `fully_shard` calls.\n- **Only root module sharded** → Apply `fully_shard` bottom-up on submodules before the root.\n- **Memory spikes after forward** → Set `reshard_after_forward=True` for more modules.\n- **Gradient accumulation desync** → Use `set_requires_gradient_sync` instead of FSDP1’s `no_sync()`.\n\nReference: `references\u002Fpytorch_fully_shard_api.md`, `references\u002Fpytorch_fsdp2_tutorial.md`.\n\n---\n\n## Minimal reference implementation outline (agent-friendly)\n\nThe coding agent should implement a script with these labeled blocks:\n\n- `init_distributed()`: init process group, set device\n- `build_model_meta()`: model on meta, apply `fully_shard`, materialize weights\n- `build_optimizer()`: optimizer created after sharding\n- `train_step()`: forward\u002Fbackward\u002Fstep with `model(inputs)` and DTensor-aware patterns\n- `checkpoint_save\u002Fload()`: DCP or distributed state dict helpers\n\nConcrete examples live in `references\u002Fpytorch_examples_fsdp2.md` and the official tutorial reference.\n\n---\n\n## References\n- `references\u002Fpytorch_fsdp2_tutorial.md`\n- `references\u002Fpytorch_fully_shard_api.md`\n- `references\u002Fpytorch_ddp_notes.md`\n- `references\u002Fpytorch_fsdp1_api.md`\n- `references\u002Fpytorch_device_mesh_tutorial.md`\n- `references\u002Fpytorch_tp_tutorial.md`\n- `references\u002Fpytorch_dcp_overview.md`\n- `references\u002Fpytorch_dcp_recipe.md`\n- `references\u002Fpytorch_dcp_async_recipe.md`\n- `references\u002Fpytorch_examples_fsdp2.md`\n- `references\u002Ftorchtitan_fsdp_notes.md` (optional, production notes)\n- `references\u002Fray_train_fsdp2_example.md` (optional, integration example)\n",{"name":5,"tags":24,"author":35,"license":15,"version":36,"description":7,"dependencies":37},[25,26,27,28,29,30,31,32,33,34],"PyTorch","FSDP2","Fully Sharded Data Parallel","Distributed Training","DTensor","Device Mesh","Sharded Checkpointing","Mixed Precision","Offload","Torch Distributed","Orchestra Research","1.0.0",[38],"torch","master","c247302c836de6438d746b45670990eb71691166","skills_collected\u002FOrchestra-Research--AI-research-SKILLs--08-distributed-training__pytorch-fsdp2",1,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool","https:\u002F\u002Fgithub.com\u002Fqhjqhj00","chien.io","2026-05-11 15:07:59",{"source":8,"skillName":5,"commands":48,"prompt":58,"zipAvailable":59},[49,52,55],{"tool":50,"command":51},"npx","npx skills add qhjqhj00\u002Fresearch-skills-pool --skill pytorch-fsdp2",{"tool":53,"command":54},"bunx","bunx skills add qhjqhj00\u002Fresearch-skills-pool --skill pytorch-fsdp2",{"tool":56,"command":57},"pnpm","pnpm dlx skills add qhjqhj00\u002Fresearch-skills-pool --skill pytorch-fsdp2","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--08-distributed-training__pytorch-fsdp2\u002FSKILL.md",true,[61,67,71,75,79,83,87,91,95,99,103,107],{"path":62,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":65,"sourceHtmlUrl":66},"references\u002Fpytorch_dcp_async_recipe.md","REFERENCE","参考文档",808,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--08-distributed-training__pytorch-fsdp2\u002Freferences\u002Fpytorch_dcp_async_recipe.md",{"path":68,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":69,"sourceHtmlUrl":70},"references\u002Fpytorch_dcp_overview.md",1022,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--08-distributed-training__pytorch-fsdp2\u002Freferences\u002Fpytorch_dcp_overview.md",{"path":72,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":73,"sourceHtmlUrl":74},"references\u002Fpytorch_dcp_recipe.md",1043,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--08-distributed-training__pytorch-fsdp2\u002Freferences\u002Fpytorch_dcp_recipe.md",{"path":76,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":77,"sourceHtmlUrl":78},"references\u002Fpytorch_ddp_notes.md",461,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--08-distributed-training__pytorch-fsdp2\u002Freferences\u002Fpytorch_ddp_notes.md",{"path":80,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":81,"sourceHtmlUrl":82},"references\u002Fpytorch_device_mesh_tutorial.md",1222,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--08-distributed-training__pytorch-fsdp2\u002Freferences\u002Fpytorch_device_mesh_tutorial.md",{"path":84,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":85,"sourceHtmlUrl":86},"references\u002Fpytorch_examples_fsdp2.md",742,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--08-distributed-training__pytorch-fsdp2\u002Freferences\u002Fpytorch_examples_fsdp2.md",{"path":88,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":89,"sourceHtmlUrl":90},"references\u002Fpytorch_fsdp1_api.md",396,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--08-distributed-training__pytorch-fsdp2\u002Freferences\u002Fpytorch_fsdp1_api.md",{"path":92,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":93,"sourceHtmlUrl":94},"references\u002Fpytorch_fsdp2_tutorial.md",2495,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--08-distributed-training__pytorch-fsdp2\u002Freferences\u002Fpytorch_fsdp2_tutorial.md",{"path":96,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":97,"sourceHtmlUrl":98},"references\u002Fpytorch_fully_shard_api.md",2884,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--08-distributed-training__pytorch-fsdp2\u002Freferences\u002Fpytorch_fully_shard_api.md",{"path":100,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":101,"sourceHtmlUrl":102},"references\u002Fpytorch_tp_tutorial.md",994,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--08-distributed-training__pytorch-fsdp2\u002Freferences\u002Fpytorch_tp_tutorial.md",{"path":104,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":105,"sourceHtmlUrl":106},"references\u002Fray_train_fsdp2_example.md",592,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--08-distributed-training__pytorch-fsdp2\u002Freferences\u002Fray_train_fsdp2_example.md",{"path":108,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":109,"sourceHtmlUrl":110},"references\u002Ftorchtitan_fsdp_notes.md",666,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--08-distributed-training__pytorch-fsdp2\u002Freferences\u002Ftorchtitan_fsdp_notes.md",[112,119,126,133,140,147],{"skillNo":113,"name":114,"nameZh":6,"description":115,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":116,"categories":117,"keywords":118,"publishedAt":19},"01M73JJB6EQPFPPVQK2Y9N67Q0","transformer-lens-interpretability","Provides guidance for mechanistic interpretability research using TransformerLens to inspect and manipulate transformer internals via HookPoints and activation caching. Use when reverse-engineering model algorithms, studying attention patterns, or performing activation patching experiments.","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Ftop100-runs\u002F130-transformer-lens\u002FSKILL.md",[],[],{"skillNo":120,"name":121,"nameZh":6,"description":122,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":123,"categories":124,"keywords":125,"publishedAt":19},"01M73JHTANRRWEE9HP98TPEHAY","wiki-vitepress","Packages generated wiki Markdown into a VitePress static site with dark theme, dark-mode Mermaid diagrams with click-to-zoom, and production build output. Use when the user wants to create a browsable website from generated wiki pages.","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Ftop100-runs\u002F126-wiki-vitepress\u002FSKILL.md",[],[],{"skillNo":127,"name":128,"nameZh":6,"description":129,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":130,"categories":131,"keywords":132,"publishedAt":19},"01M73JHJBMVFW46F1Y8K3SBF43","phoenix-observability","Open-source AI observability platform for LLM tracing, evaluation, and monitoring. Use when debugging LLM applications with detailed traces, running evaluations on datasets, or monitoring production AI systems with real-time insights.","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Ftop100-runs\u002F124-phoenix\u002FSKILL.md",[],[],{"skillNo":134,"name":135,"nameZh":6,"description":136,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":137,"categories":138,"keywords":139,"publishedAt":19},"01M73JHFPA3A2806PMP42ZN86Q","frontend-ui-dark-ts","Build dark-themed React applications using Tailwind CSS with custom theming, glassmorphism effects, and Framer Motion animations. Use when creating dashboards, admin panels, or data-rich interfaces with a refined dark aesthetic.","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Ftop100-runs\u002F123-frontend-ui-dark-ts\u002FSKILL.md",[],[],{"skillNo":141,"name":142,"nameZh":6,"description":143,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":144,"categories":145,"keywords":146,"publishedAt":19},"01M73JFVBVK57NRJS49TT2VDCT","tensorboard","Visualize training metrics, debug models with histograms, compare experiments, visualize model graphs, and profile performance with TensorBoard - Google's ML visualization toolkit","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Ftop100-runs\u002F113-tensorboard\u002FSKILL.md",[],[],{"skillNo":148,"name":149,"nameZh":6,"description":150,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":151,"licenseFileUrl":6,"sourceHtmlUrl":152,"categories":153,"keywords":154,"publishedAt":19},"01M73JFF6PK6ZCN41C89J7HS2B","markdown-mermaid-writing","Comprehensive markdown and Mermaid diagram writing skill. Use when creating any scientific document, report, analysis, or visualization. Establishes text-based diagrams as the default documentation standard with full style guides (markdown + mermaid), 24 diagram type references, and 9 document templates.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Ftop100-runs\u002F110-markdown-mermaid-writing\u002FSKILL.md",[],[]]