[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6MAX201CVX2YGSB5AQDB226":3,"skill-same:01M6MAX201CVX2YGSB5AQDB226":89},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":40,"linkOnly":41,"markdownContent":42,"markdownBody":43,"frontMatter":44,"refName":51,"commitSha":52,"skillDirectory":53,"forksCount":54,"repositoryHtmlUrl":55,"repositoryHomepageUrl":56,"ownerHtmlUrl":57,"ownerBlogUrl":58,"ownerDescription":59,"updatedAt":60,"install":61,"resources":74},"01M6MAX201CVX2YGSB5AQDB226","pytorch-lightning-distributed",null,"High-level PyTorch framework with Trainer class, automatic distributed training (DDP\u002FFSDP\u002FDeepSpeed), callbacks system, and minimal boilerplate. Scales from laptop to supercomputer with same code. Use when you want clean training loops with built-in best practices.","hashgraph-online\u002Fawesome-codex-plugins","hashgraph-online","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F160371962?v=4","01M6CNB6NRJVNNGN14TR01Q6D3","HOL (Hashgraph Online)","01M6JT1HSKKG9V68KQAQGBA7XP",1575,"MIT","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fpytorch-lightning-distributed\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39],"pytorch","distributed","pytorch lightning distributed","lightning","pytorch lightning","when","automatic","you","loops","Scalable","Callbacks","High-Level API","Training Framework","DDP","FSDP","clean","api","Transformers","framework","code","training","2026-11-10 10:12:30",false,"---\nname: pytorch-lightning-distributed\ndescription: High-level PyTorch framework with Trainer class, automatic distributed training (DDP\u002FFSDP\u002FDeepSpeed), callbacks system, and minimal boilerplate. Scales from laptop to supercomputer with same code. Use when you want clean training loops with built-in best practices.\nlicense: MIT\nmetadata:\n  version: 1.0.0\n  category: ml-training\n  maintainer: Kalaris Labs\n  tags: PyTorch Lightning, Training Framework, Distributed Training, DDP, FSDP, DeepSpeed, High-Level API, Callbacks, Best Practices, Scalable\n  dependencies: lightning, torch, transformers\n---\n\n# PyTorch Lightning - High-Level Training Framework\n\n## Quick start\n\nPyTorch Lightning organizes PyTorch code to eliminate boilerplate while maintaining flexibility.\n\n**Installation**:\n```bash\npip install lightning\n```\n\n**Convert PyTorch to Lightning** (3 steps):\n\n```python\nimport lightning as L\nimport torch\nfrom torch import nn\nfrom torch.utils.data import DataLoader, Dataset\n\n# Step 1: Define LightningModule (organize your PyTorch code)\nclass LitModel(L.LightningModule):\n    def __init__(self, hidden_size=128):\n        super().__init__()\n        self.model = nn.Sequential(\n            nn.Linear(28 * 28, hidden_size),\n            nn.ReLU(),\n            nn.Linear(hidden_size, 10)\n        )\n\n    def training_step(self, batch, batch_idx):\n        x, y = batch\n        y_hat = self.model(x)\n        loss = nn.functional.cross_entropy(y_hat, y)\n        self.log('train_loss', loss)  # Auto-logged to TensorBoard\n        return loss\n\n    def configure_optimizers(self):\n        return torch.optim.Adam(self.parameters(), lr=1e-3)\n\n# Step 2: Create data\ntrain_loader = DataLoader(train_dataset, batch_size=32)\n\n# Step 3: Train with Trainer (handles everything else!)\ntrainer = L.Trainer(max_epochs=10, accelerator='gpu', devices=2)\nmodel = LitModel()\ntrainer.fit(model, train_loader)\n```\n\n**That's it!** Trainer handles:\n- GPU\u002FTPU\u002FCPU switching\n- Distributed training (DDP, FSDP, DeepSpeed)\n- Mixed precision (FP16, BF16)\n- Gradient accumulation\n- Checkpointing\n- Logging\n- Progress bars\n\n## Common workflows\n\n### Workflow 1: From PyTorch to Lightning\n\n**Original PyTorch code**:\n```python\nmodel = MyModel()\noptimizer = torch.optim.Adam(model.parameters())\nmodel.to('cuda')\n\nfor epoch in range(max_epochs):\n    for batch in train_loader:\n        batch = batch.to('cuda')\n        optimizer.zero_grad()\n        loss = model(batch)\n        loss.backward()\n        optimizer.step()\n```\n\n**Lightning version**:\n```python\nclass LitModel(L.LightningModule):\n    def __init__(self):\n        super().__init__()\n        self.model = MyModel()\n\n    def training_step(self, batch, batch_idx):\n        loss = self.model(batch)  # No .to('cuda') needed!\n        return loss\n\n    def configure_optimizers(self):\n        return torch.optim.Adam(self.parameters())\n\n# Train\ntrainer = L.Trainer(max_epochs=10, accelerator='gpu')\ntrainer.fit(LitModel(), train_loader)\n```\n\n**Benefits**: 40+ lines → 15 lines, no device management, automatic distributed\n\n### Workflow 2: Validation and testing\n\n```python\nclass LitModel(L.LightningModule):\n    def __init__(self):\n        super().__init__()\n        self.model = MyModel()\n\n    def training_step(self, batch, batch_idx):\n        x, y = batch\n        y_hat = self.model(x)\n        loss = nn.functional.cross_entropy(y_hat, y)\n        self.log('train_loss', loss)\n        return loss\n\n    def validation_step(self, batch, batch_idx):\n        x, y = batch\n        y_hat = self.model(x)\n        val_loss = nn.functional.cross_entropy(y_hat, y)\n        acc = (y_hat.argmax(dim=1) == y).float().mean()\n        self.log('val_loss', val_loss)\n        self.log('val_acc', acc)\n\n    def test_step(self, batch, batch_idx):\n        x, y = batch\n        y_hat = self.model(x)\n        test_loss = nn.functional.cross_entropy(y_hat, y)\n        self.log('test_loss', test_loss)\n\n    def configure_optimizers(self):\n        return torch.optim.Adam(self.parameters(), lr=1e-3)\n\n# Train with validation\ntrainer = L.Trainer(max_epochs=10)\ntrainer.fit(model, train_loader, val_loader)\n\n# Test\ntrainer.test(model, test_loader)\n```\n\n**Automatic features**:\n- Validation runs every epoch by default\n- Metrics logged to TensorBoard\n- Best model checkpointing based on val_loss\n\n### Workflow 3: Distributed training (DDP)\n\n```python\n# Same code as single GPU!\nmodel = LitModel()\n\n# 8 GPUs with DDP (automatic!)\ntrainer = L.Trainer(\n    accelerator='gpu',\n    devices=8,\n    strategy='ddp'  # Or 'fsdp', 'deepspeed'\n)\n\ntrainer.fit(model, train_loader)\n```\n\n**Launch**:\n```bash\n# Single command, Lightning handles the rest\npython train.py\n```\n\n**No changes needed**:\n- Automatic data distribution\n- Gradient synchronization\n- Multi-node support (just set `num_nodes=2`)\n\n### Workflow 4: Callbacks for monitoring\n\n```python\nfrom lightning.pytorch.callbacks import ModelCheckpoint, EarlyStopping, LearningRateMonitor\n\n# Create callbacks\ncheckpoint = ModelCheckpoint(\n    monitor='val_loss',\n    mode='min',\n    save_top_k=3,\n    filename='model-{epoch:02d}-{val_loss:.2f}'\n)\n\nearly_stop = EarlyStopping(\n    monitor='val_loss',\n    patience=5,\n    mode='min'\n)\n\nlr_monitor = LearningRateMonitor(logging_interval='epoch')\n\n# Add to Trainer\ntrainer = L.Trainer(\n    max_epochs=100,\n    callbacks=[checkpoint, early_stop, lr_monitor]\n)\n\ntrainer.fit(model, train_loader, val_loader)\n```\n\n**Result**:\n- Auto-saves best 3 models\n- Stops early if no improvement for 5 epochs\n- Logs learning rate to TensorBoard\n\n### Workflow 5: Learning rate scheduling\n\n```python\nclass LitModel(L.LightningModule):\n    # ... (training_step, etc.)\n\n    def configure_optimizers(self):\n        optimizer = torch.optim.Adam(self.parameters(), lr=1e-3)\n\n        # Cosine annealing\n        scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(\n            optimizer,\n            T_max=100,\n            eta_min=1e-5\n        )\n\n        return {\n            'optimizer': optimizer,\n            'lr_scheduler': {\n                'scheduler': scheduler,\n                'interval': 'epoch',  # Update per epoch\n                'frequency': 1\n            }\n        }\n\n# Learning rate auto-logged!\ntrainer = L.Trainer(max_epochs=100)\ntrainer.fit(model, train_loader)\n```\n\n## When to use vs alternatives\n\n**Use PyTorch Lightning when**:\n- Want clean, organized code\n- Need production-ready training loops\n- Switching between single GPU, multi-GPU, TPU\n- Want built-in callbacks and logging\n- Team collaboration (standardized structure)\n\n**Key advantages**:\n- **Organized**: Separates research code from engineering\n- **Automatic**: DDP, FSDP, DeepSpeed with 1 line\n- **Callbacks**: Modular training extensions\n- **Reproducible**: Less boilerplate = fewer bugs\n- **Tested**: 1M+ downloads\u002Fmonth, battle-tested\n\n**Use alternatives instead**:\n- **Accelerate**: Minimal changes to existing code, more flexibility\n- **Ray Train**: Multi-node orchestration, hyperparameter tuning\n- **Raw PyTorch**: Maximum control, learning purposes\n- **Keras**: TensorFlow ecosystem\n\n## Common issues\n\n**Issue: Loss not decreasing**\n\nCheck data and model setup:\n```python\n# Add to training_step\ndef training_step(self, batch, batch_idx):\n    if batch_idx == 0:\n        print(f\"Batch shape: {batch[0].shape}\")\n        print(f\"Labels: {batch[1]}\")\n    loss = ...\n    return loss\n```\n\n**Issue: Out of memory**\n\nReduce batch size or use gradient accumulation:\n```python\ntrainer = L.Trainer(\n    accumulate_grad_batches=4,  # Effective batch = batch_size × 4\n    precision='bf16'  # Or 'fp16', reduces memory 50%\n)\n```\n\n**Issue: Validation not running**\n\nEnsure you pass val_loader:\n```python\n# WRONG\ntrainer.fit(model, train_loader)\n\n# CORRECT\ntrainer.fit(model, train_loader, val_loader)\n```\n\n**Issue: DDP spawns multiple processes unexpectedly**\n\nLightning auto-detects GPUs. Explicitly set devices:\n```python\n# Test on CPU first\ntrainer = L.Trainer(accelerator='cpu', devices=1)\n\n# Then GPU\ntrainer = L.Trainer(accelerator='gpu', devices=1)\n```\n\n## Advanced topics\n\n**Callbacks**: See [references\u002Fcallbacks.md](references\u002Fcallbacks.md) for EarlyStopping, ModelCheckpoint, custom callbacks, and callback hooks.\n\n**Distributed strategies**: See [references\u002Fdistributed.md](references\u002Fdistributed.md) for DDP, FSDP, DeepSpeed ZeRO integration, multi-node setup.\n\n**Hyperparameter tuning**: See [references\u002Fhyperparameter-tuning.md](references\u002Fhyperparameter-tuning.md) for integration with Optuna, Ray Tune, and WandB sweeps.\n\n## Hardware requirements\n\n- **CPU**: Works (good for debugging)\n- **Single GPU**: Works\n- **Multi-GPU**: DDP (default), FSDP, or DeepSpeed\n- **Multi-node**: DDP, FSDP, DeepSpeed\n- **TPU**: Supported (8 cores)\n- **Apple MPS**: Supported\n\n**Precision options**:\n- FP32 (default)\n- FP16 (V100, older GPUs)\n- BF16 (A100\u002FH100, recommended)\n- FP8 (H100)\n\n## Resources\n\n- Docs: https:\u002F\u002Flightning.ai\u002Fdocs\u002Fpytorch\u002Fstable\u002F\n- GitHub: https:\u002F\u002Fgithub.com\u002FLightning-AI\u002Fpytorch-lightning ⭐ 29,000+\n- Version: 2.5.5+\n- Examples: https:\u002F\u002Fgithub.com\u002FLightning-AI\u002Fpytorch-lightning\u002Ftree\u002Fmaster\u002Fexamples\n- Discord: https:\u002F\u002Fdiscord.gg\u002Flightning-ai\n- Used by: Kaggle winners, research labs, production teams\n\n## Agent operating procedure\n\n1. **Check the environment.** Check GPU type, memory and driver\u002FCUDA versions (`nvidia-smi`), framework versions, and dataset location and size.\n2. **Pin down the inputs.** Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.\n3. **Run a small version first.** Do a smoke run: tiny model or subset, few steps, and confirm loss decreases and checkpoints save.\n4. **Execute the full task** using the instructions and references above.\n5. **Validate the result.** Track metrics on held-out data, compare against a baseline, and record seeds, configs and hardware.\n6. **Report.** State what was run (versions, commands, parameters), what was checked, and what is still uncertain.\n\n| If this happens | Do this |\n|---|---|\n| CUDA out-of-memory | Reduce batch size, enable gradient accumulation\u002Fcheckpointing or mixed precision, or shard the model. |\n| Loss is NaN or diverges | Lower the learning rate, check data for invalid values, and enable gradient clipping. |\n| A function, flag or endpoint in these instructions is missing in the installed version | Check the installed version's own documentation (`help()`, `--help`, official docs), adapt, and tell the user. Never invent an API. |\n| A required input, identifier or parameter is ambiguous | Ask the user, or state the assumption explicitly before running. |\n\n**Integrity rules**\n\n- Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.\n- Never claim training results without logs; estimate compute cost before launching large jobs.\n- Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.\n- Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.\n\n## Related skills\n\n- `pytorch-lightning`: Organizes PyTorch training code with the lightning package (PyTorch Lightning): LightningModule, LightningDataModule, Trainer, callbacks su…\n- `ray-train`: Distributed training orchestration across clusters.\n- `huggingface-accelerate`: Wraps existing PyTorch training scripts with HuggingFace Accelerate (Accelerator class, accelerate config, accelerate launch) so the same c…\n","# PyTorch Lightning - High-Level Training Framework\n\n## Quick start\n\nPyTorch Lightning organizes PyTorch code to eliminate boilerplate while maintaining flexibility.\n\n**Installation**:\n```bash\npip install lightning\n```\n\n**Convert PyTorch to Lightning** (3 steps):\n\n```python\nimport lightning as L\nimport torch\nfrom torch import nn\nfrom torch.utils.data import DataLoader, Dataset\n\n# Step 1: Define LightningModule (organize your PyTorch code)\nclass LitModel(L.LightningModule):\n    def __init__(self, hidden_size=128):\n        super().__init__()\n        self.model = nn.Sequential(\n            nn.Linear(28 * 28, hidden_size),\n            nn.ReLU(),\n            nn.Linear(hidden_size, 10)\n        )\n\n    def training_step(self, batch, batch_idx):\n        x, y = batch\n        y_hat = self.model(x)\n        loss = nn.functional.cross_entropy(y_hat, y)\n        self.log('train_loss', loss)  # Auto-logged to TensorBoard\n        return loss\n\n    def configure_optimizers(self):\n        return torch.optim.Adam(self.parameters(), lr=1e-3)\n\n# Step 2: Create data\ntrain_loader = DataLoader(train_dataset, batch_size=32)\n\n# Step 3: Train with Trainer (handles everything else!)\ntrainer = L.Trainer(max_epochs=10, accelerator='gpu', devices=2)\nmodel = LitModel()\ntrainer.fit(model, train_loader)\n```\n\n**That's it!** Trainer handles:\n- GPU\u002FTPU\u002FCPU switching\n- Distributed training (DDP, FSDP, DeepSpeed)\n- Mixed precision (FP16, BF16)\n- Gradient accumulation\n- Checkpointing\n- Logging\n- Progress bars\n\n## Common workflows\n\n### Workflow 1: From PyTorch to Lightning\n\n**Original PyTorch code**:\n```python\nmodel = MyModel()\noptimizer = torch.optim.Adam(model.parameters())\nmodel.to('cuda')\n\nfor epoch in range(max_epochs):\n    for batch in train_loader:\n        batch = batch.to('cuda')\n        optimizer.zero_grad()\n        loss = model(batch)\n        loss.backward()\n        optimizer.step()\n```\n\n**Lightning version**:\n```python\nclass LitModel(L.LightningModule):\n    def __init__(self):\n        super().__init__()\n        self.model = MyModel()\n\n    def training_step(self, batch, batch_idx):\n        loss = self.model(batch)  # No .to('cuda') needed!\n        return loss\n\n    def configure_optimizers(self):\n        return torch.optim.Adam(self.parameters())\n\n# Train\ntrainer = L.Trainer(max_epochs=10, accelerator='gpu')\ntrainer.fit(LitModel(), train_loader)\n```\n\n**Benefits**: 40+ lines → 15 lines, no device management, automatic distributed\n\n### Workflow 2: Validation and testing\n\n```python\nclass LitModel(L.LightningModule):\n    def __init__(self):\n        super().__init__()\n        self.model = MyModel()\n\n    def training_step(self, batch, batch_idx):\n        x, y = batch\n        y_hat = self.model(x)\n        loss = nn.functional.cross_entropy(y_hat, y)\n        self.log('train_loss', loss)\n        return loss\n\n    def validation_step(self, batch, batch_idx):\n        x, y = batch\n        y_hat = self.model(x)\n        val_loss = nn.functional.cross_entropy(y_hat, y)\n        acc = (y_hat.argmax(dim=1) == y).float().mean()\n        self.log('val_loss', val_loss)\n        self.log('val_acc', acc)\n\n    def test_step(self, batch, batch_idx):\n        x, y = batch\n        y_hat = self.model(x)\n        test_loss = nn.functional.cross_entropy(y_hat, y)\n        self.log('test_loss', test_loss)\n\n    def configure_optimizers(self):\n        return torch.optim.Adam(self.parameters(), lr=1e-3)\n\n# Train with validation\ntrainer = L.Trainer(max_epochs=10)\ntrainer.fit(model, train_loader, val_loader)\n\n# Test\ntrainer.test(model, test_loader)\n```\n\n**Automatic features**:\n- Validation runs every epoch by default\n- Metrics logged to TensorBoard\n- Best model checkpointing based on val_loss\n\n### Workflow 3: Distributed training (DDP)\n\n```python\n# Same code as single GPU!\nmodel = LitModel()\n\n# 8 GPUs with DDP (automatic!)\ntrainer = L.Trainer(\n    accelerator='gpu',\n    devices=8,\n    strategy='ddp'  # Or 'fsdp', 'deepspeed'\n)\n\ntrainer.fit(model, train_loader)\n```\n\n**Launch**:\n```bash\n# Single command, Lightning handles the rest\npython train.py\n```\n\n**No changes needed**:\n- Automatic data distribution\n- Gradient synchronization\n- Multi-node support (just set `num_nodes=2`)\n\n### Workflow 4: Callbacks for monitoring\n\n```python\nfrom lightning.pytorch.callbacks import ModelCheckpoint, EarlyStopping, LearningRateMonitor\n\n# Create callbacks\ncheckpoint = ModelCheckpoint(\n    monitor='val_loss',\n    mode='min',\n    save_top_k=3,\n    filename='model-{epoch:02d}-{val_loss:.2f}'\n)\n\nearly_stop = EarlyStopping(\n    monitor='val_loss',\n    patience=5,\n    mode='min'\n)\n\nlr_monitor = LearningRateMonitor(logging_interval='epoch')\n\n# Add to Trainer\ntrainer = L.Trainer(\n    max_epochs=100,\n    callbacks=[checkpoint, early_stop, lr_monitor]\n)\n\ntrainer.fit(model, train_loader, val_loader)\n```\n\n**Result**:\n- Auto-saves best 3 models\n- Stops early if no improvement for 5 epochs\n- Logs learning rate to TensorBoard\n\n### Workflow 5: Learning rate scheduling\n\n```python\nclass LitModel(L.LightningModule):\n    # ... (training_step, etc.)\n\n    def configure_optimizers(self):\n        optimizer = torch.optim.Adam(self.parameters(), lr=1e-3)\n\n        # Cosine annealing\n        scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(\n            optimizer,\n            T_max=100,\n            eta_min=1e-5\n        )\n\n        return {\n            'optimizer': optimizer,\n            'lr_scheduler': {\n                'scheduler': scheduler,\n                'interval': 'epoch',  # Update per epoch\n                'frequency': 1\n            }\n        }\n\n# Learning rate auto-logged!\ntrainer = L.Trainer(max_epochs=100)\ntrainer.fit(model, train_loader)\n```\n\n## When to use vs alternatives\n\n**Use PyTorch Lightning when**:\n- Want clean, organized code\n- Need production-ready training loops\n- Switching between single GPU, multi-GPU, TPU\n- Want built-in callbacks and logging\n- Team collaboration (standardized structure)\n\n**Key advantages**:\n- **Organized**: Separates research code from engineering\n- **Automatic**: DDP, FSDP, DeepSpeed with 1 line\n- **Callbacks**: Modular training extensions\n- **Reproducible**: Less boilerplate = fewer bugs\n- **Tested**: 1M+ downloads\u002Fmonth, battle-tested\n\n**Use alternatives instead**:\n- **Accelerate**: Minimal changes to existing code, more flexibility\n- **Ray Train**: Multi-node orchestration, hyperparameter tuning\n- **Raw PyTorch**: Maximum control, learning purposes\n- **Keras**: TensorFlow ecosystem\n\n## Common issues\n\n**Issue: Loss not decreasing**\n\nCheck data and model setup:\n```python\n# Add to training_step\ndef training_step(self, batch, batch_idx):\n    if batch_idx == 0:\n        print(f\"Batch shape: {batch[0].shape}\")\n        print(f\"Labels: {batch[1]}\")\n    loss = ...\n    return loss\n```\n\n**Issue: Out of memory**\n\nReduce batch size or use gradient accumulation:\n```python\ntrainer = L.Trainer(\n    accumulate_grad_batches=4,  # Effective batch = batch_size × 4\n    precision='bf16'  # Or 'fp16', reduces memory 50%\n)\n```\n\n**Issue: Validation not running**\n\nEnsure you pass val_loader:\n```python\n# WRONG\ntrainer.fit(model, train_loader)\n\n# CORRECT\ntrainer.fit(model, train_loader, val_loader)\n```\n\n**Issue: DDP spawns multiple processes unexpectedly**\n\nLightning auto-detects GPUs. Explicitly set devices:\n```python\n# Test on CPU first\ntrainer = L.Trainer(accelerator='cpu', devices=1)\n\n# Then GPU\ntrainer = L.Trainer(accelerator='gpu', devices=1)\n```\n\n## Advanced topics\n\n**Callbacks**: See [references\u002Fcallbacks.md](references\u002Fcallbacks.md) for EarlyStopping, ModelCheckpoint, custom callbacks, and callback hooks.\n\n**Distributed strategies**: See [references\u002Fdistributed.md](references\u002Fdistributed.md) for DDP, FSDP, DeepSpeed ZeRO integration, multi-node setup.\n\n**Hyperparameter tuning**: See [references\u002Fhyperparameter-tuning.md](references\u002Fhyperparameter-tuning.md) for integration with Optuna, Ray Tune, and WandB sweeps.\n\n## Hardware requirements\n\n- **CPU**: Works (good for debugging)\n- **Single GPU**: Works\n- **Multi-GPU**: DDP (default), FSDP, or DeepSpeed\n- **Multi-node**: DDP, FSDP, DeepSpeed\n- **TPU**: Supported (8 cores)\n- **Apple MPS**: Supported\n\n**Precision options**:\n- FP32 (default)\n- FP16 (V100, older GPUs)\n- BF16 (A100\u002FH100, recommended)\n- FP8 (H100)\n\n## Resources\n\n- Docs: https:\u002F\u002Flightning.ai\u002Fdocs\u002Fpytorch\u002Fstable\u002F\n- GitHub: https:\u002F\u002Fgithub.com\u002FLightning-AI\u002Fpytorch-lightning ⭐ 29,000+\n- Version: 2.5.5+\n- Examples: https:\u002F\u002Fgithub.com\u002FLightning-AI\u002Fpytorch-lightning\u002Ftree\u002Fmaster\u002Fexamples\n- Discord: https:\u002F\u002Fdiscord.gg\u002Flightning-ai\n- Used by: Kaggle winners, research labs, production teams\n\n## Agent operating procedure\n\n1. **Check the environment.** Check GPU type, memory and driver\u002FCUDA versions (`nvidia-smi`), framework versions, and dataset location and size.\n2. **Pin down the inputs.** Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.\n3. **Run a small version first.** Do a smoke run: tiny model or subset, few steps, and confirm loss decreases and checkpoints save.\n4. **Execute the full task** using the instructions and references above.\n5. **Validate the result.** Track metrics on held-out data, compare against a baseline, and record seeds, configs and hardware.\n6. **Report.** State what was run (versions, commands, parameters), what was checked, and what is still uncertain.\n\n| If this happens | Do this |\n|---|---|\n| CUDA out-of-memory | Reduce batch size, enable gradient accumulation\u002Fcheckpointing or mixed precision, or shard the model. |\n| Loss is NaN or diverges | Lower the learning rate, check data for invalid values, and enable gradient clipping. |\n| A function, flag or endpoint in these instructions is missing in the installed version | Check the installed version's own documentation (`help()`, `--help`, official docs), adapt, and tell the user. Never invent an API. |\n| A required input, identifier or parameter is ambiguous | Ask the user, or state the assumption explicitly before running. |\n\n**Integrity rules**\n\n- Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.\n- Never claim training results without logs; estimate compute cost before launching large jobs.\n- Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.\n- Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.\n\n## Related skills\n\n- `pytorch-lightning`: Organizes PyTorch training code with the lightning package (PyTorch Lightning): LightningModule, LightningDataModule, Trainer, callbacks su…\n- `ray-train`: Distributed training orchestration across clusters.\n- `huggingface-accelerate`: Wraps existing PyTorch training scripts with HuggingFace Accelerate (Accelerator class, accelerate config, accelerate launch) so the same c…\n",{"name":5,"license":15,"metadata":45,"description":7},{"tags":46,"version":47,"category":48,"maintainer":49,"dependencies":50},"PyTorch Lightning, Training Framework, Distributed Training, DDP, FSDP, DeepSpeed, High-Level API, Callbacks, Best Practices, Scalable","1.0.0","ml-training","Kalaris Labs","lightning, torch, transformers","main","2b63f856ce19c2a797929cacabd9e0fc936812fa","plugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fpytorch-lightning-distributed",491,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins","https:\u002F\u002Fhol.org\u002Fplugins\u002Fbest-codex-plugins","https:\u002F\u002Fgithub.com\u002Fhashgraph-online","https:\u002F\u002Fhol.org","Open-source standards and developer tools for interoperable AI agents: discovery, identity, communication, payments, privacy and security.","2026-11-14 14:06:13",{"source":8,"skillName":5,"commands":62,"prompt":72,"zipAvailable":73},[63,66,69],{"tool":64,"command":65},"npx","npx skills add hashgraph-online\u002Fawesome-codex-plugins --skill pytorch-lightning-distributed",{"tool":67,"command":68},"bunx","bunx skills add hashgraph-online\u002Fawesome-codex-plugins --skill pytorch-lightning-distributed",{"tool":70,"command":71},"pnpm","pnpm dlx skills add hashgraph-online\u002Fawesome-codex-plugins --skill pytorch-lightning-distributed","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fpytorch-lightning-distributed\u002FSKILL.md",true,[75,81,85],{"path":76,"resourceType":77,"resourceTypeDesc":78,"sizeBytes":79,"sourceHtmlUrl":80},"references\u002Fcallbacks.md","REFERENCE","参考文档",12031,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fpytorch-lightning-distributed\u002Freferences\u002Fcallbacks.md",{"path":82,"resourceType":77,"resourceTypeDesc":78,"sizeBytes":83,"sourceHtmlUrl":84},"references\u002Fdistributed.md",10818,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fpytorch-lightning-distributed\u002Freferences\u002Fdistributed.md",{"path":86,"resourceType":77,"resourceTypeDesc":78,"sizeBytes":87,"sourceHtmlUrl":88},"references\u002Fhyperparameter-tuning.md",12517,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fpytorch-lightning-distributed\u002Freferences\u002Fhyperparameter-tuning.md",[90,99,107,115,123,131],{"skillNo":91,"name":92,"nameZh":6,"description":93,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":94,"sourceHtmlUrl":95,"categories":96,"keywords":97,"publishedAt":98},"01M7DV332YQMGD8WTRJ9DZQXTB","worktable-review-with-annotations","Create, inspect, reply to, update, or resolve situated Worktable annotations. Use when feedback belongs to a Doc section, block, text range, or HTML Doc, or when the user asks to address existing review feedback and coordinate a requested artifact edit.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-review-with-annotations\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-review-with-annotations\u002FSKILL.md",[],[],"2026-11-15 14:19:17",{"skillNo":100,"name":101,"nameZh":6,"description":102,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":103,"sourceHtmlUrl":104,"categories":105,"keywords":106,"publishedAt":98},"01M7DV2YGHCZVBEB1AHYB6CGHE","worktable-find-and-synthesize-context","Find, assess, and synthesize context from Worktable Docs and Records. Use when the user asks what a workspace contains, wants workspace-backed research, needs current facts reconciled across artifacts, or requests a cited synthesis without changing workspace state.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-find-and-synthesize-context\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-find-and-synthesize-context\u002FSKILL.md",[],[],{"skillNo":108,"name":109,"nameZh":6,"description":110,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":111,"sourceHtmlUrl":112,"categories":113,"keywords":114,"publishedAt":98},"01M7DV2TQN1H11ME2SAF9MXN7V","worktable-create-or-update-docs","Create, edit, rename, or maintain narrative Worktable Docs. Use for plans, decisions, briefs, notes, research syntheses, and other durable prose when the user requests a new artifact or a safe update to an existing one.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-update-docs\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-update-docs\u002FSKILL.md",[],[],{"skillNo":116,"name":117,"nameZh":6,"description":118,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":119,"sourceHtmlUrl":120,"categories":121,"keywords":122,"publishedAt":98},"01M7DV2NE452NEA8TXATKSNAJH","worktable-create-or-manage-records","Organize repeated structured information as durable Worktable Records. Use for research datasets, inventories, directories, pipelines, trackers, and other collections whose items need identity, fields, filtering, links, validation, or individual updates.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-records\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-records\u002FSKILL.md",[],[],{"skillNo":124,"name":125,"nameZh":6,"description":126,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":127,"sourceHtmlUrl":128,"categories":129,"keywords":130,"publishedAt":98},"01M7DV2HENRBR2KSCVM064WH0Y","worktable-create-or-manage-interactive-html","Create or improve durable visual and interactive artifacts in Worktable. Use for dashboards, trackers, calculators, mini apps, slide presentations, and other requests that benefit from a polished HTML Doc.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-interactive-html\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-interactive-html\u002FSKILL.md",[],[],{"skillNo":132,"name":133,"nameZh":6,"description":134,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":135,"sourceHtmlUrl":136,"categories":137,"keywords":138,"publishedAt":98},"01M7DV2CADR5R7RQFFYPNAGR9Y","worktable-collaborate-threads","Read, answer, and hand off messages through durable Worktable threads with users or other authenticated participants. Use when the user asks to check addressed messages, reply to a participant, continue a thread, request another agent's help, wait briefly for a response, or watch for comments and messages sent to the agent.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-collaborate-threads\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-collaborate-threads\u002FSKILL.md",[],[]]