[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6MAWVBTS6DDEZDCYE3FYEYW":3,"skill-same:01M6MAWVBTS6DDEZDCYE3FYEYW":130},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":23,"publishedAt":45,"linkOnly":46,"markdownContent":47,"markdownBody":48,"frontMatter":49,"refName":56,"commitSha":57,"skillDirectory":58,"forksCount":59,"repositoryHtmlUrl":60,"repositoryHomepageUrl":61,"ownerHtmlUrl":62,"ownerBlogUrl":63,"ownerDescription":64,"updatedAt":65,"install":66,"resources":79},"01M6MAWVBTS6DDEZDCYE3FYEYW","pytorch-fsdp2",null,"Adds PyTorch FSDP2 (fully_shard) to training scripts with correct init, sharding, mixed precision\u002Foffload config, and distributed checkpointing. Use when models exceed single-GPU memory or when you need DTensor-based sharding with DeviceMesh.","hashgraph-online\u002Fawesome-codex-plugins","hashgraph-online","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F160371962?v=4","01M6CNB6NRJVNNGN14TR01Q6D3","HOL (Hashgraph Online)","01M6JT1HSKKG9V68KQAQGBA7XP",1575,"MIT","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fpytorch-fsdp2\u002FSKILL.md",[18],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44],"pytorch fsdp2","pytorch","FSDP2","models","when","you","Torch Distributed","Sharded Checkpointing","Device Mesh","DTensor","Fully Sharded Data Parallel","scripts","mesh","use","parallel","shard","training","distributed","config","init","memory","2026-11-10 10:12:30",false,"---\nname: pytorch-fsdp2\ndescription: Adds PyTorch FSDP2 (fully_shard) to training scripts with correct init, sharding, mixed precision\u002Foffload config, and distributed checkpointing. Use when models exceed single-GPU memory or when you need DTensor-based sharding with DeviceMesh.\nlicense: MIT\nmetadata:\n  version: 1.0.0\n  category: ml-training\n  maintainer: Kalaris Labs\n  tags: PyTorch, FSDP2, Fully Sharded Data Parallel, Distributed Training, DTensor, Device Mesh, Sharded Checkpointing, Mixed Precision, Offload, Torch Distributed\n  dependencies: torch\n---\n\n# Skill: Use PyTorch FSDP2 (`fully_shard`) correctly in a training script\n\nThis skill teaches a coding agent how to **add PyTorch FSDP2** to a training loop with correct initialization, sharding, mixed precision\u002Foffload configuration, and checkpointing.\n\n> FSDP2 in PyTorch is exposed primarily via `torch.distributed.fsdp.fully_shard` and the `FSDPModule` methods it adds in-place to modules. See: `references\u002Fpytorch_fully_shard_api.md`, `references\u002Fpytorch_fsdp2_tutorial.md`.\n\n---\n\n## When to use this skill\n\nUse FSDP2 when:\n- Your model **doesn’t fit** on one GPU (parameters + gradients + optimizer state).\n- You want an eager-mode sharding approach that is **DTensor-based per-parameter sharding** (more inspectable, simpler sharded state dicts) than FSDP1.  \n- You may later compose DP with **Tensor Parallel** using **DeviceMesh**.\n\nAvoid (or be careful) if:\n- You need strict backwards-compatible checkpoints across PyTorch versions (DCP warns against this).\n- You’re forced onto older PyTorch versions without the FSDP2 stack.\n\n## Alternatives (when FSDP2 is not the best fit)\n\n- **DistributedDataParallel (DDP)**: Use the standard data-parallel wrapper when you want classic distributed data parallel training.\n- **FullyShardedDataParallel (FSDP1)**: Use the original FSDP wrapper for parameter sharding across data-parallel workers.\n\nReference: `references\u002Fpytorch_ddp_notes.md`, `references\u002Fpytorch_fsdp1_api.md`.\n\n---\n\n## Contract the agent must follow\n\n1. **Launch with `torchrun`** and set the CUDA device per process (usually via `LOCAL_RANK`).  \n2. **Apply `fully_shard()` bottom-up**, i.e., shard submodules (e.g., Transformer blocks) before the root module.  \n3. **Call `model(input)`**, not `model.forward(input)`, so the FSDP2 hooks run (unless you explicitly `unshard()` or register the forward method).  \n4. **Create the optimizer after sharding** and make sure it is built on the **DTensor parameters** (post-`fully_shard`).  \n5. **Checkpoint using Distributed Checkpoint (DCP)** or the distributed-state-dict helpers, not naïve `torch.save(model.state_dict())` unless you deliberately gather to full tensors.\n\n(Each of these rules is directly described in the official API docs\u002Ftutorial; see references.)\n\n---\n\n## Step-by-step procedure\n\n### 0) Version & environment sanity\n- Prefer a recent stable PyTorch where the docs show FSDP2 and DCP updated recently.\n- Use `torchrun --nproc_per_node \u003Cgpus_per_node> ...` and ensure `RANK`, `WORLD_SIZE`, `LOCAL_RANK` are visible.\n\nReference: `references\u002Fpytorch_fsdp2_tutorial.md` (launch commands and setup), `references\u002Fpytorch_fully_shard_api.md` (user contract).\n\n---\n\n### 1) Initialize distributed and set device\nMinimal, correct pattern:\n- `dist.init_process_group(backend=\"nccl\")`\n- `torch.cuda.set_device(int(os.environ[\"LOCAL_RANK\"]))`\n- Optionally create a `DeviceMesh` to describe the data-parallel group(s)\n\nReference: `references\u002Fpytorch_device_mesh_tutorial.md` (why DeviceMesh exists & how it manages process groups).\n\n---\n\n### 2) Build model on meta device (recommended for very large models)\nFor big models, initialize on `meta`, apply sharding, then materialize weights on GPU:\n- `with torch.device(\"meta\"): model = ...`\n- apply `fully_shard(...)` on submodules, then `fully_shard(model)`\n- `model.to_empty(device=\"cuda\")`\n- `model.reset_parameters()` (or your init routine)\n\nReference: `references\u002Fpytorch_fsdp2_tutorial.md` (migration guide shows this flow explicitly).\n\n---\n\n### 3) Apply `fully_shard()` bottom-up (wrapping policy = “apply where needed”)\n**Do not** only call `fully_shard` on the topmost module.\n\nRecommended sharding pattern for transformer-like models:\n- iterate modules, `if isinstance(m, TransformerBlock): fully_shard(m, ...)`\n- then `fully_shard(model, ...)`\n\nWhy:\n- `fully_shard` forms “parameter groups” for collective efficiency and excludes params already grouped by earlier calls. Bottom-up gives better overlap and lower peak memory.\n\nReference: `references\u002Fpytorch_fully_shard_api.md` (bottom-up requirement and why).\n\n---\n\n### 4) Configure `reshard_after_forward` for memory\u002Fperf trade-offs\nDefault behavior:\n- `None` means `True` for non-root modules and `False` for root modules (good default).\n\nHeuristics:\n- If you’re memory-bound: keep defaults or force `True` on many blocks.\n- If you’re throughput-bound and can afford memory: consider keeping unsharded params longer (root often `False`).\n- Advanced: use an `int` to reshard to a smaller mesh after forward (e.g., intra-node) if it’s a meaningful divisor.\n\nReference: `references\u002Fpytorch_fully_shard_api.md` (full semantics).\n\n---\n\n### 5) Mixed precision & offload (optional but common)\nFSDP2 uses:\n- `mp_policy=MixedPrecisionPolicy(param_dtype=..., reduce_dtype=..., output_dtype=..., cast_forward_inputs=...)`\n- `offload_policy=CPUOffloadPolicy()` if you want CPU offload\n\nRules of thumb:\n- Start with BF16 parameters\u002Freductions on H100\u002FA100-class GPUs (if numerically stable for your model).\n- Keep `reduce_dtype` aligned with your gradient reduction expectations.\n- If you use CPU offload, budget for PCIe\u002FNVLink traffic and runtime overhead.\n\nReference: `references\u002Fpytorch_fully_shard_api.md` (MixedPrecisionPolicy \u002F OffloadPolicy classes).\n\n---\n\n### 6) Optimizer, gradient clipping, accumulation\n- Create the optimizer **after** sharding so it holds DTensor params.\n- If you need gradient accumulation \u002F no_sync:\n  - use the FSDP2 mechanism (`set_requires_gradient_sync`) instead of FSDP1’s `no_sync()`.\n\nGradient clipping:\n- Use the approach shown in the FSDP2 tutorial (“Gradient Clipping and Optimizer with DTensor”), because parameters\u002Fgradients are DTensors.\n\nReference: `references\u002Fpytorch_fsdp2_tutorial.md`.\n\n---\n\n### 7) Checkpointing: prefer DCP or distributed state dict helpers\nTwo recommended approaches:\n\n**A) Distributed Checkpoint (DCP) — best default**\n- DCP saves\u002Floads from multiple ranks in parallel and supports load-time resharding.\n- DCP produces **multiple files** (often at least one per rank) and operates “in place”.\n\n**B) Distributed state dict helpers**\n- `get_model_state_dict` \u002F `set_model_state_dict` with `StateDictOptions(full_state_dict=True, cpu_offload=True, broadcast_from_rank0=True, ...)`\n- For optimizer: `get_optimizer_state_dict` \u002F `set_optimizer_state_dict`\n\nAvoid:\n- Saving DTensor state dicts with plain `torch.save` unless you intentionally convert with `DTensor.full_tensor()` and manage memory carefully.\n\nReferences:\n- `references\u002Fpytorch_dcp_overview.md` (DCP behavior and caveats)\n- `references\u002Fpytorch_dcp_recipe.md` and `references\u002Fpytorch_dcp_async_recipe.md` (end-to-end usage)\n- `references\u002Fpytorch_fsdp2_tutorial.md` (DTensor vs DCP state-dict flows)\n- `references\u002Fpytorch_examples_fsdp2.md` (working checkpoint scripts)\n\n---\n\n## Workflow checklists (copy-paste friendly)\n\n### Workflow A: Retrofit FSDP2 into an existing training script\n- [ ] Launch with `torchrun` and initialize the process group.\n- [ ] Set the CUDA device from `LOCAL_RANK`; create a `DeviceMesh` if you need multi-dim parallelism.\n- [ ] Build the model (use `meta` if needed), apply `fully_shard` bottom-up, then `fully_shard(model)`.\n- [ ] Create the optimizer after sharding so it captures DTensor parameters.\n- [ ] Use `model(inputs)` so hooks run; use `set_requires_gradient_sync` for accumulation.\n- [ ] Add DCP save\u002Fload via `torch.distributed.checkpoint` helpers.\n\nReference: `references\u002Fpytorch_fsdp2_tutorial.md`, `references\u002Fpytorch_fully_shard_api.md`, `references\u002Fpytorch_device_mesh_tutorial.md`, `references\u002Fpytorch_dcp_recipe.md`.\n\n### Workflow B: Add DCP save\u002Fload (minimal pattern)\n- [ ] Wrap state in `Stateful` or assemble state via `get_state_dict`.\n- [ ] Call `dcp.save(...)` from all ranks to a shared path.\n- [ ] Call `dcp.load(...)` and restore with `set_state_dict`.\n- [ ] Validate any resharding assumptions when loading into a different mesh.\n\nReference: `references\u002Fpytorch_dcp_recipe.md`.\n\n## Debug checklist (what the agent should check first)\n\n1. **All ranks on distinct GPUs?**  \n   If not, verify `torch.cuda.set_device(LOCAL_RANK)` and your `torchrun` flags.\n2. **Did you accidentally call `forward()` directly?**  \n   Use `model(input)` or explicitly `unshard()` \u002F register forward.\n3. **Is `fully_shard()` applied bottom-up?**  \n   If only root is sharded, expect worse memory\u002Fperf and possible confusion.\n4. **Optimizer created at the right time?**  \n   Must be built on DTensor parameters *after* sharding.\n5. **Checkpointing path consistent?**  \n   - If using DCP, don’t mix with ad-hoc `torch.save` unless you understand conversions.\n   - Be mindful of PyTorch-version compatibility warnings for DCP.\n\n---\n\n## Common issues and fixes\n\n- **Forward hooks not running** → Call `model(inputs)` (or `unshard()` explicitly) instead of `model.forward(...)`.\n- **Optimizer sees non-DTensor params** → Create optimizer after all `fully_shard` calls.\n- **Only root module sharded** → Apply `fully_shard` bottom-up on submodules before the root.\n- **Memory spikes after forward** → Set `reshard_after_forward=True` for more modules.\n- **Gradient accumulation desync** → Use `set_requires_gradient_sync` instead of FSDP1’s `no_sync()`.\n\nReference: `references\u002Fpytorch_fully_shard_api.md`, `references\u002Fpytorch_fsdp2_tutorial.md`.\n\n---\n\n## Minimal reference implementation outline (agent-friendly)\n\nThe coding agent should implement a script with these labeled blocks:\n\n- `init_distributed()`: init process group, set device\n- `build_model_meta()`: model on meta, apply `fully_shard`, materialize weights\n- `build_optimizer()`: optimizer created after sharding\n- `train_step()`: forward\u002Fbackward\u002Fstep with `model(inputs)` and DTensor-aware patterns\n- `checkpoint_save\u002Fload()`: DCP or distributed state dict helpers\n\nConcrete examples live in `references\u002Fpytorch_examples_fsdp2.md` and the official tutorial reference.\n\n---\n\n## References\n- `references\u002Fpytorch_fsdp2_tutorial.md`\n- `references\u002Fpytorch_fully_shard_api.md`\n- `references\u002Fpytorch_ddp_notes.md`\n- `references\u002Fpytorch_fsdp1_api.md`\n- `references\u002Fpytorch_device_mesh_tutorial.md`\n- `references\u002Fpytorch_tp_tutorial.md`\n- `references\u002Fpytorch_dcp_overview.md`\n- `references\u002Fpytorch_dcp_recipe.md`\n- `references\u002Fpytorch_dcp_async_recipe.md`\n- `references\u002Fpytorch_examples_fsdp2.md`\n- `references\u002Ftorchtitan_fsdp_notes.md` (optional, production notes)\n- `references\u002Fray_train_fsdp2_example.md` (optional, integration example)\n\n## Related skills\n\n- `huggingface-accelerate`: Wraps existing PyTorch training scripts with HuggingFace Accelerate (Accelerator class, accelerate config, accelerate launch) so the same c…\n- `distributed-llm-pretraining-torchtitan`: Provides PyTorch-native distributed LLM pretraining using torchtitan with 4D parallelism (FSDP2, TP, PP, CP).\n","# Skill: Use PyTorch FSDP2 (`fully_shard`) correctly in a training script\n\nThis skill teaches a coding agent how to **add PyTorch FSDP2** to a training loop with correct initialization, sharding, mixed precision\u002Foffload configuration, and checkpointing.\n\n> FSDP2 in PyTorch is exposed primarily via `torch.distributed.fsdp.fully_shard` and the `FSDPModule` methods it adds in-place to modules. See: `references\u002Fpytorch_fully_shard_api.md`, `references\u002Fpytorch_fsdp2_tutorial.md`.\n\n---\n\n## When to use this skill\n\nUse FSDP2 when:\n- Your model **doesn’t fit** on one GPU (parameters + gradients + optimizer state).\n- You want an eager-mode sharding approach that is **DTensor-based per-parameter sharding** (more inspectable, simpler sharded state dicts) than FSDP1.  \n- You may later compose DP with **Tensor Parallel** using **DeviceMesh**.\n\nAvoid (or be careful) if:\n- You need strict backwards-compatible checkpoints across PyTorch versions (DCP warns against this).\n- You’re forced onto older PyTorch versions without the FSDP2 stack.\n\n## Alternatives (when FSDP2 is not the best fit)\n\n- **DistributedDataParallel (DDP)**: Use the standard data-parallel wrapper when you want classic distributed data parallel training.\n- **FullyShardedDataParallel (FSDP1)**: Use the original FSDP wrapper for parameter sharding across data-parallel workers.\n\nReference: `references\u002Fpytorch_ddp_notes.md`, `references\u002Fpytorch_fsdp1_api.md`.\n\n---\n\n## Contract the agent must follow\n\n1. **Launch with `torchrun`** and set the CUDA device per process (usually via `LOCAL_RANK`).  \n2. **Apply `fully_shard()` bottom-up**, i.e., shard submodules (e.g., Transformer blocks) before the root module.  \n3. **Call `model(input)`**, not `model.forward(input)`, so the FSDP2 hooks run (unless you explicitly `unshard()` or register the forward method).  \n4. **Create the optimizer after sharding** and make sure it is built on the **DTensor parameters** (post-`fully_shard`).  \n5. **Checkpoint using Distributed Checkpoint (DCP)** or the distributed-state-dict helpers, not naïve `torch.save(model.state_dict())` unless you deliberately gather to full tensors.\n\n(Each of these rules is directly described in the official API docs\u002Ftutorial; see references.)\n\n---\n\n## Step-by-step procedure\n\n### 0) Version & environment sanity\n- Prefer a recent stable PyTorch where the docs show FSDP2 and DCP updated recently.\n- Use `torchrun --nproc_per_node \u003Cgpus_per_node> ...` and ensure `RANK`, `WORLD_SIZE`, `LOCAL_RANK` are visible.\n\nReference: `references\u002Fpytorch_fsdp2_tutorial.md` (launch commands and setup), `references\u002Fpytorch_fully_shard_api.md` (user contract).\n\n---\n\n### 1) Initialize distributed and set device\nMinimal, correct pattern:\n- `dist.init_process_group(backend=\"nccl\")`\n- `torch.cuda.set_device(int(os.environ[\"LOCAL_RANK\"]))`\n- Optionally create a `DeviceMesh` to describe the data-parallel group(s)\n\nReference: `references\u002Fpytorch_device_mesh_tutorial.md` (why DeviceMesh exists & how it manages process groups).\n\n---\n\n### 2) Build model on meta device (recommended for very large models)\nFor big models, initialize on `meta`, apply sharding, then materialize weights on GPU:\n- `with torch.device(\"meta\"): model = ...`\n- apply `fully_shard(...)` on submodules, then `fully_shard(model)`\n- `model.to_empty(device=\"cuda\")`\n- `model.reset_parameters()` (or your init routine)\n\nReference: `references\u002Fpytorch_fsdp2_tutorial.md` (migration guide shows this flow explicitly).\n\n---\n\n### 3) Apply `fully_shard()` bottom-up (wrapping policy = “apply where needed”)\n**Do not** only call `fully_shard` on the topmost module.\n\nRecommended sharding pattern for transformer-like models:\n- iterate modules, `if isinstance(m, TransformerBlock): fully_shard(m, ...)`\n- then `fully_shard(model, ...)`\n\nWhy:\n- `fully_shard` forms “parameter groups” for collective efficiency and excludes params already grouped by earlier calls. Bottom-up gives better overlap and lower peak memory.\n\nReference: `references\u002Fpytorch_fully_shard_api.md` (bottom-up requirement and why).\n\n---\n\n### 4) Configure `reshard_after_forward` for memory\u002Fperf trade-offs\nDefault behavior:\n- `None` means `True` for non-root modules and `False` for root modules (good default).\n\nHeuristics:\n- If you’re memory-bound: keep defaults or force `True` on many blocks.\n- If you’re throughput-bound and can afford memory: consider keeping unsharded params longer (root often `False`).\n- Advanced: use an `int` to reshard to a smaller mesh after forward (e.g., intra-node) if it’s a meaningful divisor.\n\nReference: `references\u002Fpytorch_fully_shard_api.md` (full semantics).\n\n---\n\n### 5) Mixed precision & offload (optional but common)\nFSDP2 uses:\n- `mp_policy=MixedPrecisionPolicy(param_dtype=..., reduce_dtype=..., output_dtype=..., cast_forward_inputs=...)`\n- `offload_policy=CPUOffloadPolicy()` if you want CPU offload\n\nRules of thumb:\n- Start with BF16 parameters\u002Freductions on H100\u002FA100-class GPUs (if numerically stable for your model).\n- Keep `reduce_dtype` aligned with your gradient reduction expectations.\n- If you use CPU offload, budget for PCIe\u002FNVLink traffic and runtime overhead.\n\nReference: `references\u002Fpytorch_fully_shard_api.md` (MixedPrecisionPolicy \u002F OffloadPolicy classes).\n\n---\n\n### 6) Optimizer, gradient clipping, accumulation\n- Create the optimizer **after** sharding so it holds DTensor params.\n- If you need gradient accumulation \u002F no_sync:\n  - use the FSDP2 mechanism (`set_requires_gradient_sync`) instead of FSDP1’s `no_sync()`.\n\nGradient clipping:\n- Use the approach shown in the FSDP2 tutorial (“Gradient Clipping and Optimizer with DTensor”), because parameters\u002Fgradients are DTensors.\n\nReference: `references\u002Fpytorch_fsdp2_tutorial.md`.\n\n---\n\n### 7) Checkpointing: prefer DCP or distributed state dict helpers\nTwo recommended approaches:\n\n**A) Distributed Checkpoint (DCP) — best default**\n- DCP saves\u002Floads from multiple ranks in parallel and supports load-time resharding.\n- DCP produces **multiple files** (often at least one per rank) and operates “in place”.\n\n**B) Distributed state dict helpers**\n- `get_model_state_dict` \u002F `set_model_state_dict` with `StateDictOptions(full_state_dict=True, cpu_offload=True, broadcast_from_rank0=True, ...)`\n- For optimizer: `get_optimizer_state_dict` \u002F `set_optimizer_state_dict`\n\nAvoid:\n- Saving DTensor state dicts with plain `torch.save` unless you intentionally convert with `DTensor.full_tensor()` and manage memory carefully.\n\nReferences:\n- `references\u002Fpytorch_dcp_overview.md` (DCP behavior and caveats)\n- `references\u002Fpytorch_dcp_recipe.md` and `references\u002Fpytorch_dcp_async_recipe.md` (end-to-end usage)\n- `references\u002Fpytorch_fsdp2_tutorial.md` (DTensor vs DCP state-dict flows)\n- `references\u002Fpytorch_examples_fsdp2.md` (working checkpoint scripts)\n\n---\n\n## Workflow checklists (copy-paste friendly)\n\n### Workflow A: Retrofit FSDP2 into an existing training script\n- [ ] Launch with `torchrun` and initialize the process group.\n- [ ] Set the CUDA device from `LOCAL_RANK`; create a `DeviceMesh` if you need multi-dim parallelism.\n- [ ] Build the model (use `meta` if needed), apply `fully_shard` bottom-up, then `fully_shard(model)`.\n- [ ] Create the optimizer after sharding so it captures DTensor parameters.\n- [ ] Use `model(inputs)` so hooks run; use `set_requires_gradient_sync` for accumulation.\n- [ ] Add DCP save\u002Fload via `torch.distributed.checkpoint` helpers.\n\nReference: `references\u002Fpytorch_fsdp2_tutorial.md`, `references\u002Fpytorch_fully_shard_api.md`, `references\u002Fpytorch_device_mesh_tutorial.md`, `references\u002Fpytorch_dcp_recipe.md`.\n\n### Workflow B: Add DCP save\u002Fload (minimal pattern)\n- [ ] Wrap state in `Stateful` or assemble state via `get_state_dict`.\n- [ ] Call `dcp.save(...)` from all ranks to a shared path.\n- [ ] Call `dcp.load(...)` and restore with `set_state_dict`.\n- [ ] Validate any resharding assumptions when loading into a different mesh.\n\nReference: `references\u002Fpytorch_dcp_recipe.md`.\n\n## Debug checklist (what the agent should check first)\n\n1. **All ranks on distinct GPUs?**  \n   If not, verify `torch.cuda.set_device(LOCAL_RANK)` and your `torchrun` flags.\n2. **Did you accidentally call `forward()` directly?**  \n   Use `model(input)` or explicitly `unshard()` \u002F register forward.\n3. **Is `fully_shard()` applied bottom-up?**  \n   If only root is sharded, expect worse memory\u002Fperf and possible confusion.\n4. **Optimizer created at the right time?**  \n   Must be built on DTensor parameters *after* sharding.\n5. **Checkpointing path consistent?**  \n   - If using DCP, don’t mix with ad-hoc `torch.save` unless you understand conversions.\n   - Be mindful of PyTorch-version compatibility warnings for DCP.\n\n---\n\n## Common issues and fixes\n\n- **Forward hooks not running** → Call `model(inputs)` (or `unshard()` explicitly) instead of `model.forward(...)`.\n- **Optimizer sees non-DTensor params** → Create optimizer after all `fully_shard` calls.\n- **Only root module sharded** → Apply `fully_shard` bottom-up on submodules before the root.\n- **Memory spikes after forward** → Set `reshard_after_forward=True` for more modules.\n- **Gradient accumulation desync** → Use `set_requires_gradient_sync` instead of FSDP1’s `no_sync()`.\n\nReference: `references\u002Fpytorch_fully_shard_api.md`, `references\u002Fpytorch_fsdp2_tutorial.md`.\n\n---\n\n## Minimal reference implementation outline (agent-friendly)\n\nThe coding agent should implement a script with these labeled blocks:\n\n- `init_distributed()`: init process group, set device\n- `build_model_meta()`: model on meta, apply `fully_shard`, materialize weights\n- `build_optimizer()`: optimizer created after sharding\n- `train_step()`: forward\u002Fbackward\u002Fstep with `model(inputs)` and DTensor-aware patterns\n- `checkpoint_save\u002Fload()`: DCP or distributed state dict helpers\n\nConcrete examples live in `references\u002Fpytorch_examples_fsdp2.md` and the official tutorial reference.\n\n---\n\n## References\n- `references\u002Fpytorch_fsdp2_tutorial.md`\n- `references\u002Fpytorch_fully_shard_api.md`\n- `references\u002Fpytorch_ddp_notes.md`\n- `references\u002Fpytorch_fsdp1_api.md`\n- `references\u002Fpytorch_device_mesh_tutorial.md`\n- `references\u002Fpytorch_tp_tutorial.md`\n- `references\u002Fpytorch_dcp_overview.md`\n- `references\u002Fpytorch_dcp_recipe.md`\n- `references\u002Fpytorch_dcp_async_recipe.md`\n- `references\u002Fpytorch_examples_fsdp2.md`\n- `references\u002Ftorchtitan_fsdp_notes.md` (optional, production notes)\n- `references\u002Fray_train_fsdp2_example.md` (optional, integration example)\n\n## Related skills\n\n- `huggingface-accelerate`: Wraps existing PyTorch training scripts with HuggingFace Accelerate (Accelerator class, accelerate config, accelerate launch) so the same c…\n- `distributed-llm-pretraining-torchtitan`: Provides PyTorch-native distributed LLM pretraining using torchtitan with 4D parallelism (FSDP2, TP, PP, CP).\n",{"name":5,"license":15,"metadata":50,"description":7},{"tags":51,"version":52,"category":53,"maintainer":54,"dependencies":55},"PyTorch, FSDP2, Fully Sharded Data Parallel, Distributed Training, DTensor, Device Mesh, Sharded Checkpointing, Mixed Precision, Offload, Torch Distributed","1.0.0","ml-training","Kalaris Labs","torch","main","2b63f856ce19c2a797929cacabd9e0fc936812fa","plugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fpytorch-fsdp2",491,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins","https:\u002F\u002Fhol.org\u002Fplugins\u002Fbest-codex-plugins","https:\u002F\u002Fgithub.com\u002Fhashgraph-online","https:\u002F\u002Fhol.org","Open-source standards and developer tools for interoperable AI agents: discovery, identity, communication, payments, privacy and security.","2026-11-14 14:06:13",{"source":8,"skillName":5,"commands":67,"prompt":77,"zipAvailable":78},[68,71,74],{"tool":69,"command":70},"npx","npx skills add hashgraph-online\u002Fawesome-codex-plugins --skill pytorch-fsdp2",{"tool":72,"command":73},"bunx","bunx skills add hashgraph-online\u002Fawesome-codex-plugins --skill pytorch-fsdp2",{"tool":75,"command":76},"pnpm","pnpm dlx skills add hashgraph-online\u002Fawesome-codex-plugins --skill pytorch-fsdp2","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fpytorch-fsdp2\u002FSKILL.md",true,[80,86,90,94,98,102,106,110,114,118,122,126],{"path":81,"resourceType":82,"resourceTypeDesc":83,"sizeBytes":84,"sourceHtmlUrl":85},"references\u002Fpytorch_dcp_async_recipe.md","REFERENCE","参考文档",808,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fpytorch-fsdp2\u002Freferences\u002Fpytorch_dcp_async_recipe.md",{"path":87,"resourceType":82,"resourceTypeDesc":83,"sizeBytes":88,"sourceHtmlUrl":89},"references\u002Fpytorch_dcp_overview.md",1022,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fpytorch-fsdp2\u002Freferences\u002Fpytorch_dcp_overview.md",{"path":91,"resourceType":82,"resourceTypeDesc":83,"sizeBytes":92,"sourceHtmlUrl":93},"references\u002Fpytorch_dcp_recipe.md",1043,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fpytorch-fsdp2\u002Freferences\u002Fpytorch_dcp_recipe.md",{"path":95,"resourceType":82,"resourceTypeDesc":83,"sizeBytes":96,"sourceHtmlUrl":97},"references\u002Fpytorch_ddp_notes.md",461,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fpytorch-fsdp2\u002Freferences\u002Fpytorch_ddp_notes.md",{"path":99,"resourceType":82,"resourceTypeDesc":83,"sizeBytes":100,"sourceHtmlUrl":101},"references\u002Fpytorch_device_mesh_tutorial.md",1222,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fpytorch-fsdp2\u002Freferences\u002Fpytorch_device_mesh_tutorial.md",{"path":103,"resourceType":82,"resourceTypeDesc":83,"sizeBytes":104,"sourceHtmlUrl":105},"references\u002Fpytorch_examples_fsdp2.md",742,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fpytorch-fsdp2\u002Freferences\u002Fpytorch_examples_fsdp2.md",{"path":107,"resourceType":82,"resourceTypeDesc":83,"sizeBytes":108,"sourceHtmlUrl":109},"references\u002Fpytorch_fsdp1_api.md",396,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fpytorch-fsdp2\u002Freferences\u002Fpytorch_fsdp1_api.md",{"path":111,"resourceType":82,"resourceTypeDesc":83,"sizeBytes":112,"sourceHtmlUrl":113},"references\u002Fpytorch_fsdp2_tutorial.md",2495,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fpytorch-fsdp2\u002Freferences\u002Fpytorch_fsdp2_tutorial.md",{"path":115,"resourceType":82,"resourceTypeDesc":83,"sizeBytes":116,"sourceHtmlUrl":117},"references\u002Fpytorch_fully_shard_api.md",2884,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fpytorch-fsdp2\u002Freferences\u002Fpytorch_fully_shard_api.md",{"path":119,"resourceType":82,"resourceTypeDesc":83,"sizeBytes":120,"sourceHtmlUrl":121},"references\u002Fpytorch_tp_tutorial.md",994,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fpytorch-fsdp2\u002Freferences\u002Fpytorch_tp_tutorial.md",{"path":123,"resourceType":82,"resourceTypeDesc":83,"sizeBytes":124,"sourceHtmlUrl":125},"references\u002Fray_train_fsdp2_example.md",592,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fpytorch-fsdp2\u002Freferences\u002Fray_train_fsdp2_example.md",{"path":127,"resourceType":82,"resourceTypeDesc":83,"sizeBytes":128,"sourceHtmlUrl":129},"references\u002Ftorchtitan_fsdp_notes.md",666,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fpytorch-fsdp2\u002Freferences\u002Ftorchtitan_fsdp_notes.md",[131,140,148,156,164,172],{"skillNo":132,"name":133,"nameZh":6,"description":134,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":135,"sourceHtmlUrl":136,"categories":137,"keywords":138,"publishedAt":139},"01M7DV332YQMGD8WTRJ9DZQXTB","worktable-review-with-annotations","Create, inspect, reply to, update, or resolve situated Worktable annotations. Use when feedback belongs to a Doc section, block, text range, or HTML Doc, or when the user asks to address existing review feedback and coordinate a requested artifact edit.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-review-with-annotations\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-review-with-annotations\u002FSKILL.md",[],[],"2026-11-15 14:19:17",{"skillNo":141,"name":142,"nameZh":6,"description":143,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":144,"sourceHtmlUrl":145,"categories":146,"keywords":147,"publishedAt":139},"01M7DV2YGHCZVBEB1AHYB6CGHE","worktable-find-and-synthesize-context","Find, assess, and synthesize context from Worktable Docs and Records. Use when the user asks what a workspace contains, wants workspace-backed research, needs current facts reconciled across artifacts, or requests a cited synthesis without changing workspace state.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-find-and-synthesize-context\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-find-and-synthesize-context\u002FSKILL.md",[],[],{"skillNo":149,"name":150,"nameZh":6,"description":151,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":152,"sourceHtmlUrl":153,"categories":154,"keywords":155,"publishedAt":139},"01M7DV2TQN1H11ME2SAF9MXN7V","worktable-create-or-update-docs","Create, edit, rename, or maintain narrative Worktable Docs. Use for plans, decisions, briefs, notes, research syntheses, and other durable prose when the user requests a new artifact or a safe update to an existing one.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-update-docs\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-update-docs\u002FSKILL.md",[],[],{"skillNo":157,"name":158,"nameZh":6,"description":159,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":160,"sourceHtmlUrl":161,"categories":162,"keywords":163,"publishedAt":139},"01M7DV2NE452NEA8TXATKSNAJH","worktable-create-or-manage-records","Organize repeated structured information as durable Worktable Records. Use for research datasets, inventories, directories, pipelines, trackers, and other collections whose items need identity, fields, filtering, links, validation, or individual updates.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-records\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-records\u002FSKILL.md",[],[],{"skillNo":165,"name":166,"nameZh":6,"description":167,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":168,"sourceHtmlUrl":169,"categories":170,"keywords":171,"publishedAt":139},"01M7DV2HENRBR2KSCVM064WH0Y","worktable-create-or-manage-interactive-html","Create or improve durable visual and interactive artifacts in Worktable. Use for dashboards, trackers, calculators, mini apps, slide presentations, and other requests that benefit from a polished HTML Doc.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-interactive-html\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-interactive-html\u002FSKILL.md",[],[],{"skillNo":173,"name":174,"nameZh":6,"description":175,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":176,"sourceHtmlUrl":177,"categories":178,"keywords":179,"publishedAt":139},"01M7DV2CADR5R7RQFFYPNAGR9Y","worktable-collaborate-threads","Read, answer, and hand off messages through durable Worktable threads with users or other authenticated participants. Use when the user asks to check addressed messages, reply to a participant, continue a thread, request another agent's help, wait briefly for a response, or watch for comments and messages sent to the agent.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-collaborate-threads\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-collaborate-threads\u002FSKILL.md",[],[]]