[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6MAYQH19AEP31Q19H4ZPB4V":3,"skill-same:01M6MAYQH19AEP31Q19H4ZPB4V":83},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":42,"linkOnly":43,"markdownContent":44,"markdownBody":45,"frontMatter":46,"refName":53,"commitSha":54,"skillDirectory":55,"forksCount":56,"repositoryHtmlUrl":57,"repositoryHomepageUrl":58,"ownerHtmlUrl":59,"ownerBlogUrl":60,"ownerDescription":61,"updatedAt":62,"install":63,"resources":76},"01M6MAYQH19AEP31Q19H4ZPB4V","ray-train",null,"Distributed training orchestration across clusters. Scales PyTorch\u002FTensorFlow\u002FHuggingFace from laptop to 1000s of nodes. Built-in hyperparameter tuning with Ray Tune, fault tolerance, elastic scaling. Use when training massive models across multiple machines or running distributed hyperparameter sweeps.","hashgraph-online\u002Fawesome-codex-plugins","hashgraph-online","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F160371962?v=4","01M6CNB6NRJVNNGN14TR01Q6D3","HOL (Hashgraph Online)","01M6JT1HSKKG9V68KQAQGBA7XP",1575,"MIT","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fray-train\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41],"Ray","Ray Train","train","Clusters","tensorflow","running","when","tolerance","Multi-Node","Elastic Scaling","Hyperparameter Tuning","fault tolerance","scaling","across","use","node","tuning","elastic","Transformers","Distributed Training","training","pytorch","orchestration","2026-11-10 10:12:30",false,"---\nname: ray-train\ndescription: Distributed training orchestration across clusters. Scales PyTorch\u002FTensorFlow\u002FHuggingFace from laptop to 1000s of nodes. Built-in hyperparameter tuning with Ray Tune, fault tolerance, elastic scaling. Use when training massive models across multiple machines or running distributed hyperparameter sweeps.\nlicense: MIT\nmetadata:\n  version: 1.0.0\n  category: ml-training\n  maintainer: Kalaris Labs\n  tags: Ray Train, Distributed Training, Orchestration, Ray, Hyperparameter Tuning, Fault Tolerance, Elastic Scaling, Multi-Node, PyTorch, TensorFlow\n  dependencies: ray[train], torch, transformers\n---\n\n# Ray Train - Distributed Training Orchestration\n\n## Quick start\n\nRay Train scales machine learning training from single GPU to multi-node clusters with minimal code changes.\n\n**Installation**:\n```bash\npip install -U \"ray[train]\"\n```\n\n**Basic PyTorch training** (single node):\n\n```python\nimport ray\nfrom ray import train\nfrom ray.train import ScalingConfig\nfrom ray.train.torch import TorchTrainer\nimport torch\nimport torch.nn as nn\n\n# Define training function\ndef train_func(config):\n    # Your normal PyTorch code\n    model = nn.Linear(10, 1)\n    optimizer = torch.optim.SGD(model.parameters(), lr=0.01)\n\n    # Prepare for distributed (Ray handles device placement)\n    model = train.torch.prepare_model(model)\n\n    for epoch in range(10):\n        # Your training loop\n        output = model(torch.randn(32, 10))\n        loss = output.sum()\n        loss.backward()\n        optimizer.step()\n        optimizer.zero_grad()\n\n        # Report metrics (logged automatically)\n        train.report({\"loss\": loss.item(), \"epoch\": epoch})\n\n# Run distributed training\ntrainer = TorchTrainer(\n    train_func,\n    scaling_config=ScalingConfig(\n        num_workers=4,  # 4 GPUs\u002Fworkers\n        use_gpu=True\n    )\n)\n\nresult = trainer.fit()\nprint(f\"Final loss: {result.metrics['loss']}\")\n```\n\n**That's it!** Ray handles:\n- Distributed coordination\n- GPU allocation\n- Fault tolerance\n- Checkpointing\n- Metric aggregation\n\n## Common workflows\n\n### Workflow 1: Scale existing PyTorch code\n\n**Original single-GPU code**:\n```python\nmodel = MyModel().cuda()\noptimizer = torch.optim.Adam(model.parameters())\n\nfor epoch in range(epochs):\n    for batch in dataloader:\n        loss = model(batch)\n        loss.backward()\n        optimizer.step()\n```\n\n**Ray Train version** (scales to multi-GPU\u002Fmulti-node):\n```python\nfrom ray.train.torch import TorchTrainer\nfrom ray import train\n\ndef train_func(config):\n    model = MyModel()\n    optimizer = torch.optim.Adam(model.parameters())\n\n    # Prepare for distributed (automatic device placement)\n    model = train.torch.prepare_model(model)\n    dataloader = train.torch.prepare_data_loader(dataloader)\n\n    for epoch in range(epochs):\n        for batch in dataloader:\n            loss = model(batch)\n            loss.backward()\n            optimizer.step()\n\n            # Report metrics\n            train.report({\"loss\": loss.item()})\n\n# Scale to 8 GPUs\ntrainer = TorchTrainer(\n    train_func,\n    scaling_config=ScalingConfig(num_workers=8, use_gpu=True)\n)\ntrainer.fit()\n```\n\n**Benefits**: Same code runs on 1 GPU or 1000 GPUs\n\n### Workflow 2: HuggingFace Transformers integration\n\n```python\nfrom ray.train.huggingface import TransformersTrainer\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments\n\ndef train_func(config):\n    # Load model and tokenizer\n    model = AutoModelForCausalLM.from_pretrained(\"gpt2\")\n    tokenizer = AutoTokenizer.from_pretrained(\"gpt2\")\n\n    # Training arguments (HuggingFace API)\n    training_args = TrainingArguments(\n        output_dir=\".\u002Foutput\",\n        num_train_epochs=3,\n        per_device_train_batch_size=8,\n        learning_rate=2e-5,\n    )\n\n    # Ray automatically handles distributed training\n    from transformers import Trainer\n    trainer = Trainer(\n        model=model,\n        args=training_args,\n        train_dataset=train_dataset,\n    )\n\n    trainer.train()\n\n# Scale to multi-node (2 nodes × 8 GPUs = 16 workers)\ntrainer = TransformersTrainer(\n    train_func,\n    scaling_config=ScalingConfig(\n        num_workers=16,\n        use_gpu=True,\n        resources_per_worker={\"GPU\": 1}\n    )\n)\n\nresult = trainer.fit()\n```\n\n### Workflow 3: Hyperparameter tuning with Ray Tune\n\n```python\nfrom ray import tune\nfrom ray.train.torch import TorchTrainer\nfrom ray.tune.schedulers import ASHAScheduler\n\ndef train_func(config):\n    # Use hyperparameters from config\n    lr = config[\"lr\"]\n    batch_size = config[\"batch_size\"]\n\n    model = MyModel()\n    optimizer = torch.optim.Adam(model.parameters(), lr=lr)\n\n    model = train.torch.prepare_model(model)\n\n    for epoch in range(10):\n        # Training loop\n        loss = train_epoch(model, optimizer, batch_size)\n        train.report({\"loss\": loss, \"epoch\": epoch})\n\n# Define search space\nparam_space = {\n    \"lr\": tune.loguniform(1e-5, 1e-2),\n    \"batch_size\": tune.choice([16, 32, 64, 128])\n}\n\n# Run 20 trials with early stopping\ntuner = tune.Tuner(\n    TorchTrainer(\n        train_func,\n        scaling_config=ScalingConfig(num_workers=4, use_gpu=True)\n    ),\n    param_space=param_space,\n    tune_config=tune.TuneConfig(\n        num_samples=20,\n        scheduler=ASHAScheduler(metric=\"loss\", mode=\"min\")\n    )\n)\n\nresults = tuner.fit()\nbest = results.get_best_result(metric=\"loss\", mode=\"min\")\nprint(f\"Best hyperparameters: {best.config}\")\n```\n\n**Result**: Distributed hyperparameter search across cluster\n\n### Workflow 4: Checkpointing and fault tolerance\n\n```python\nfrom ray import train\nfrom ray.train import Checkpoint\n\ndef train_func(config):\n    model = MyModel()\n    optimizer = torch.optim.Adam(model.parameters())\n\n    # Try to resume from checkpoint\n    checkpoint = train.get_checkpoint()\n    if checkpoint:\n        with checkpoint.as_directory() as checkpoint_dir:\n            state = torch.load(f\"{checkpoint_dir}\u002Fmodel.pt\")\n            model.load_state_dict(state[\"model\"])\n            optimizer.load_state_dict(state[\"optimizer\"])\n            start_epoch = state[\"epoch\"]\n    else:\n        start_epoch = 0\n\n    model = train.torch.prepare_model(model)\n\n    for epoch in range(start_epoch, 100):\n        loss = train_epoch(model, optimizer)\n\n        # Save checkpoint every 10 epochs\n        if epoch % 10 == 0:\n            checkpoint = Checkpoint.from_directory(\n                train.get_context().get_trial_dir()\n            )\n            torch.save({\n                \"model\": model.state_dict(),\n                \"optimizer\": optimizer.state_dict(),\n                \"epoch\": epoch\n            }, checkpoint.path \u002F \"model.pt\")\n\n            train.report({\"loss\": loss}, checkpoint=checkpoint)\n\ntrainer = TorchTrainer(\n    train_func,\n    scaling_config=ScalingConfig(num_workers=8, use_gpu=True)\n)\n\n# Automatically resumes from checkpoint if training fails\nresult = trainer.fit()\n```\n\n### Workflow 5: Multi-node training\n\n```python\nfrom ray.train import ScalingConfig\n\n# Connect to Ray cluster\nray.init(address=\"auto\")  # Or ray.init(\"ray:\u002F\u002Fhead-node:10001\")\n\n# Train across 4 nodes × 8 GPUs = 32 workers\ntrainer = TorchTrainer(\n    train_func,\n    scaling_config=ScalingConfig(\n        num_workers=32,\n        use_gpu=True,\n        resources_per_worker={\"GPU\": 1, \"CPU\": 4},\n        placement_strategy=\"SPREAD\"  # Spread across nodes\n    )\n)\n\nresult = trainer.fit()\n```\n\n**Launch Ray cluster**:\n```bash\n# On head node\nray start --head --port=6379\n\n# On worker nodes\nray start --address=\u003Chead-node-ip>:6379\n```\n\n## When to use vs alternatives\n\n**Use Ray Train when**:\n- Training across multiple machines (multi-node)\n- Need hyperparameter tuning at scale\n- Want fault tolerance (auto-restart failed workers)\n- Elastic scaling (add\u002Fremove nodes during training)\n- Unified framework (same code for PyTorch\u002FTF\u002FHF)\n\n**Key advantages**:\n- **Multi-node orchestration**: Easiest multi-node setup\n- **Ray Tune integration**: Best-in-class hyperparameter tuning\n- **Fault tolerance**: Automatic recovery from failures\n- **Elastic**: Add\u002Fremove nodes without restarting\n- **Framework agnostic**: PyTorch, TensorFlow, HuggingFace, XGBoost\n\n**Use alternatives instead**:\n- **Accelerate**: Single-node multi-GPU, simpler\n- **PyTorch Lightning**: High-level abstractions, callbacks\n- **DeepSpeed**: Maximum performance, complex setup\n- **Raw DDP**: Maximum control, minimal overhead\n\n## Common issues\n\n**Issue: Ray cluster not connecting**\n\nCheck ray status:\n```bash\nray status\n\n# Should show:\n# - Nodes: 4\n# - GPUs: 32\n# - Workers: Ready\n```\n\nIf not connected:\n```bash\n# Restart head node\nray stop\nray start --head --port=6379 --dashboard-host=0.0.0.0\n\n# Restart worker nodes\nray stop\nray start --address=\u003Chead-ip>:6379\n```\n\n**Issue: Out of memory**\n\nReduce workers or use gradient accumulation:\n```python\nscaling_config=ScalingConfig(\n    num_workers=4,  # Reduce from 8\n    use_gpu=True\n)\n\n# In train_func, accumulate gradients\nfor i, batch in enumerate(dataloader):\n    loss = model(batch) \u002F accumulation_steps\n    loss.backward()\n\n    if (i + 1) % accumulation_steps == 0:\n        optimizer.step()\n        optimizer.zero_grad()\n```\n\n**Issue: Slow training**\n\nCheck if data loading is bottleneck:\n```python\nimport time\n\ndef train_func(config):\n    for epoch in range(epochs):\n        start = time.time()\n        for batch in dataloader:\n            data_time = time.time() - start\n            # Train...\n            start = time.time()\n            print(f\"Data loading: {data_time:.3f}s\")\n```\n\nIf data loading is slow, increase workers:\n```python\ndataloader = DataLoader(dataset, num_workers=8)\n```\n\n## Advanced topics\n\n**Multi-node setup**: See [references\u002Fmulti-node.md](references\u002Fmulti-node.md) for Ray cluster deployment on AWS, GCP, Kubernetes, and SLURM.\n\n**Hyperparameter tuning**: See references\u002Fhyperparameter-tuning.md for Ray Tune integration, search algorithms (Optuna, HyperOpt), and population-based training.\n\n**Custom training loops**: See references\u002Fcustom-loops.md for advanced Ray Train usage, custom backends, and integration with other frameworks.\n\n## Hardware requirements\n\n- **Single node**: 1+ GPUs (or CPUs)\n- **Multi-node**: 2+ machines with network connectivity\n- **Cloud**: AWS, GCP, Azure (Ray autoscaling)\n- **On-prem**: Kubernetes, SLURM clusters\n\n**Supported accelerators**:\n- NVIDIA GPUs (CUDA)\n- AMD GPUs (ROCm)\n- TPUs (Google Cloud)\n- CPUs\n\n## Resources\n\n- Docs: https:\u002F\u002Fdocs.ray.io\u002Fen\u002Flatest\u002Ftrain\u002Ftrain.html\n- GitHub: https:\u002F\u002Fgithub.com\u002Fray-project\u002Fray ⭐ 36,000+\n- Version: 2.40.0+\n- Examples: https:\u002F\u002Fdocs.ray.io\u002Fen\u002Flatest\u002Ftrain\u002Fexamples.html\n- Slack: https:\u002F\u002Fforms.gle\u002F9TSdDYUgxYs8SA9e8\n- Used by: OpenAI, Uber, Spotify, Shopify, Instacart\n\n## Agent operating procedure\n\n1. **Check the environment.** Check GPU type, memory and driver\u002FCUDA versions (`nvidia-smi`), framework versions, and dataset location and size.\n2. **Pin down the inputs.** Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.\n3. **Run a small version first.** Do a smoke run: tiny model or subset, few steps, and confirm loss decreases and checkpoints save.\n4. **Execute the full task** using the instructions and references above.\n5. **Validate the result.** Track metrics on held-out data, compare against a baseline, and record seeds, configs and hardware.\n6. **Report.** State what was run (versions, commands, parameters), what was checked, and what is still uncertain.\n\n| If this happens | Do this |\n|---|---|\n| CUDA out-of-memory | Reduce batch size, enable gradient accumulation\u002Fcheckpointing or mixed precision, or shard the model. |\n| Loss is NaN or diverges | Lower the learning rate, check data for invalid values, and enable gradient clipping. |\n| A function, flag or endpoint in these instructions is missing in the installed version | Check the installed version's own documentation (`help()`, `--help`, official docs), adapt, and tell the user. Never invent an API. |\n| A required input, identifier or parameter is ambiguous | Ask the user, or state the assumption explicitly before running. |\n\n**Integrity rules**\n\n- Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.\n- Never claim training results without logs; estimate compute cost before launching large jobs.\n- Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.\n- Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.\n\n## Related skills\n\n- `ray-data`: Scalable data processing for ML workloads.\n- `pytorch-lightning-distributed`: High-level PyTorch framework with Trainer class, automatic distributed training (DDP\u002FFSDP\u002FDeepSpeed), callbacks system, and minimal boilerp…\n- `weights-and-biases`: Logs and tracks machine learning experiments with Weights & Biases (W&B, wandb): metrics, hyperparameters, checkpoints, sweeps, artifacts w…\n","# Ray Train - Distributed Training Orchestration\n\n## Quick start\n\nRay Train scales machine learning training from single GPU to multi-node clusters with minimal code changes.\n\n**Installation**:\n```bash\npip install -U \"ray[train]\"\n```\n\n**Basic PyTorch training** (single node):\n\n```python\nimport ray\nfrom ray import train\nfrom ray.train import ScalingConfig\nfrom ray.train.torch import TorchTrainer\nimport torch\nimport torch.nn as nn\n\n# Define training function\ndef train_func(config):\n    # Your normal PyTorch code\n    model = nn.Linear(10, 1)\n    optimizer = torch.optim.SGD(model.parameters(), lr=0.01)\n\n    # Prepare for distributed (Ray handles device placement)\n    model = train.torch.prepare_model(model)\n\n    for epoch in range(10):\n        # Your training loop\n        output = model(torch.randn(32, 10))\n        loss = output.sum()\n        loss.backward()\n        optimizer.step()\n        optimizer.zero_grad()\n\n        # Report metrics (logged automatically)\n        train.report({\"loss\": loss.item(), \"epoch\": epoch})\n\n# Run distributed training\ntrainer = TorchTrainer(\n    train_func,\n    scaling_config=ScalingConfig(\n        num_workers=4,  # 4 GPUs\u002Fworkers\n        use_gpu=True\n    )\n)\n\nresult = trainer.fit()\nprint(f\"Final loss: {result.metrics['loss']}\")\n```\n\n**That's it!** Ray handles:\n- Distributed coordination\n- GPU allocation\n- Fault tolerance\n- Checkpointing\n- Metric aggregation\n\n## Common workflows\n\n### Workflow 1: Scale existing PyTorch code\n\n**Original single-GPU code**:\n```python\nmodel = MyModel().cuda()\noptimizer = torch.optim.Adam(model.parameters())\n\nfor epoch in range(epochs):\n    for batch in dataloader:\n        loss = model(batch)\n        loss.backward()\n        optimizer.step()\n```\n\n**Ray Train version** (scales to multi-GPU\u002Fmulti-node):\n```python\nfrom ray.train.torch import TorchTrainer\nfrom ray import train\n\ndef train_func(config):\n    model = MyModel()\n    optimizer = torch.optim.Adam(model.parameters())\n\n    # Prepare for distributed (automatic device placement)\n    model = train.torch.prepare_model(model)\n    dataloader = train.torch.prepare_data_loader(dataloader)\n\n    for epoch in range(epochs):\n        for batch in dataloader:\n            loss = model(batch)\n            loss.backward()\n            optimizer.step()\n\n            # Report metrics\n            train.report({\"loss\": loss.item()})\n\n# Scale to 8 GPUs\ntrainer = TorchTrainer(\n    train_func,\n    scaling_config=ScalingConfig(num_workers=8, use_gpu=True)\n)\ntrainer.fit()\n```\n\n**Benefits**: Same code runs on 1 GPU or 1000 GPUs\n\n### Workflow 2: HuggingFace Transformers integration\n\n```python\nfrom ray.train.huggingface import TransformersTrainer\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments\n\ndef train_func(config):\n    # Load model and tokenizer\n    model = AutoModelForCausalLM.from_pretrained(\"gpt2\")\n    tokenizer = AutoTokenizer.from_pretrained(\"gpt2\")\n\n    # Training arguments (HuggingFace API)\n    training_args = TrainingArguments(\n        output_dir=\".\u002Foutput\",\n        num_train_epochs=3,\n        per_device_train_batch_size=8,\n        learning_rate=2e-5,\n    )\n\n    # Ray automatically handles distributed training\n    from transformers import Trainer\n    trainer = Trainer(\n        model=model,\n        args=training_args,\n        train_dataset=train_dataset,\n    )\n\n    trainer.train()\n\n# Scale to multi-node (2 nodes × 8 GPUs = 16 workers)\ntrainer = TransformersTrainer(\n    train_func,\n    scaling_config=ScalingConfig(\n        num_workers=16,\n        use_gpu=True,\n        resources_per_worker={\"GPU\": 1}\n    )\n)\n\nresult = trainer.fit()\n```\n\n### Workflow 3: Hyperparameter tuning with Ray Tune\n\n```python\nfrom ray import tune\nfrom ray.train.torch import TorchTrainer\nfrom ray.tune.schedulers import ASHAScheduler\n\ndef train_func(config):\n    # Use hyperparameters from config\n    lr = config[\"lr\"]\n    batch_size = config[\"batch_size\"]\n\n    model = MyModel()\n    optimizer = torch.optim.Adam(model.parameters(), lr=lr)\n\n    model = train.torch.prepare_model(model)\n\n    for epoch in range(10):\n        # Training loop\n        loss = train_epoch(model, optimizer, batch_size)\n        train.report({\"loss\": loss, \"epoch\": epoch})\n\n# Define search space\nparam_space = {\n    \"lr\": tune.loguniform(1e-5, 1e-2),\n    \"batch_size\": tune.choice([16, 32, 64, 128])\n}\n\n# Run 20 trials with early stopping\ntuner = tune.Tuner(\n    TorchTrainer(\n        train_func,\n        scaling_config=ScalingConfig(num_workers=4, use_gpu=True)\n    ),\n    param_space=param_space,\n    tune_config=tune.TuneConfig(\n        num_samples=20,\n        scheduler=ASHAScheduler(metric=\"loss\", mode=\"min\")\n    )\n)\n\nresults = tuner.fit()\nbest = results.get_best_result(metric=\"loss\", mode=\"min\")\nprint(f\"Best hyperparameters: {best.config}\")\n```\n\n**Result**: Distributed hyperparameter search across cluster\n\n### Workflow 4: Checkpointing and fault tolerance\n\n```python\nfrom ray import train\nfrom ray.train import Checkpoint\n\ndef train_func(config):\n    model = MyModel()\n    optimizer = torch.optim.Adam(model.parameters())\n\n    # Try to resume from checkpoint\n    checkpoint = train.get_checkpoint()\n    if checkpoint:\n        with checkpoint.as_directory() as checkpoint_dir:\n            state = torch.load(f\"{checkpoint_dir}\u002Fmodel.pt\")\n            model.load_state_dict(state[\"model\"])\n            optimizer.load_state_dict(state[\"optimizer\"])\n            start_epoch = state[\"epoch\"]\n    else:\n        start_epoch = 0\n\n    model = train.torch.prepare_model(model)\n\n    for epoch in range(start_epoch, 100):\n        loss = train_epoch(model, optimizer)\n\n        # Save checkpoint every 10 epochs\n        if epoch % 10 == 0:\n            checkpoint = Checkpoint.from_directory(\n                train.get_context().get_trial_dir()\n            )\n            torch.save({\n                \"model\": model.state_dict(),\n                \"optimizer\": optimizer.state_dict(),\n                \"epoch\": epoch\n            }, checkpoint.path \u002F \"model.pt\")\n\n            train.report({\"loss\": loss}, checkpoint=checkpoint)\n\ntrainer = TorchTrainer(\n    train_func,\n    scaling_config=ScalingConfig(num_workers=8, use_gpu=True)\n)\n\n# Automatically resumes from checkpoint if training fails\nresult = trainer.fit()\n```\n\n### Workflow 5: Multi-node training\n\n```python\nfrom ray.train import ScalingConfig\n\n# Connect to Ray cluster\nray.init(address=\"auto\")  # Or ray.init(\"ray:\u002F\u002Fhead-node:10001\")\n\n# Train across 4 nodes × 8 GPUs = 32 workers\ntrainer = TorchTrainer(\n    train_func,\n    scaling_config=ScalingConfig(\n        num_workers=32,\n        use_gpu=True,\n        resources_per_worker={\"GPU\": 1, \"CPU\": 4},\n        placement_strategy=\"SPREAD\"  # Spread across nodes\n    )\n)\n\nresult = trainer.fit()\n```\n\n**Launch Ray cluster**:\n```bash\n# On head node\nray start --head --port=6379\n\n# On worker nodes\nray start --address=\u003Chead-node-ip>:6379\n```\n\n## When to use vs alternatives\n\n**Use Ray Train when**:\n- Training across multiple machines (multi-node)\n- Need hyperparameter tuning at scale\n- Want fault tolerance (auto-restart failed workers)\n- Elastic scaling (add\u002Fremove nodes during training)\n- Unified framework (same code for PyTorch\u002FTF\u002FHF)\n\n**Key advantages**:\n- **Multi-node orchestration**: Easiest multi-node setup\n- **Ray Tune integration**: Best-in-class hyperparameter tuning\n- **Fault tolerance**: Automatic recovery from failures\n- **Elastic**: Add\u002Fremove nodes without restarting\n- **Framework agnostic**: PyTorch, TensorFlow, HuggingFace, XGBoost\n\n**Use alternatives instead**:\n- **Accelerate**: Single-node multi-GPU, simpler\n- **PyTorch Lightning**: High-level abstractions, callbacks\n- **DeepSpeed**: Maximum performance, complex setup\n- **Raw DDP**: Maximum control, minimal overhead\n\n## Common issues\n\n**Issue: Ray cluster not connecting**\n\nCheck ray status:\n```bash\nray status\n\n# Should show:\n# - Nodes: 4\n# - GPUs: 32\n# - Workers: Ready\n```\n\nIf not connected:\n```bash\n# Restart head node\nray stop\nray start --head --port=6379 --dashboard-host=0.0.0.0\n\n# Restart worker nodes\nray stop\nray start --address=\u003Chead-ip>:6379\n```\n\n**Issue: Out of memory**\n\nReduce workers or use gradient accumulation:\n```python\nscaling_config=ScalingConfig(\n    num_workers=4,  # Reduce from 8\n    use_gpu=True\n)\n\n# In train_func, accumulate gradients\nfor i, batch in enumerate(dataloader):\n    loss = model(batch) \u002F accumulation_steps\n    loss.backward()\n\n    if (i + 1) % accumulation_steps == 0:\n        optimizer.step()\n        optimizer.zero_grad()\n```\n\n**Issue: Slow training**\n\nCheck if data loading is bottleneck:\n```python\nimport time\n\ndef train_func(config):\n    for epoch in range(epochs):\n        start = time.time()\n        for batch in dataloader:\n            data_time = time.time() - start\n            # Train...\n            start = time.time()\n            print(f\"Data loading: {data_time:.3f}s\")\n```\n\nIf data loading is slow, increase workers:\n```python\ndataloader = DataLoader(dataset, num_workers=8)\n```\n\n## Advanced topics\n\n**Multi-node setup**: See [references\u002Fmulti-node.md](references\u002Fmulti-node.md) for Ray cluster deployment on AWS, GCP, Kubernetes, and SLURM.\n\n**Hyperparameter tuning**: See references\u002Fhyperparameter-tuning.md for Ray Tune integration, search algorithms (Optuna, HyperOpt), and population-based training.\n\n**Custom training loops**: See references\u002Fcustom-loops.md for advanced Ray Train usage, custom backends, and integration with other frameworks.\n\n## Hardware requirements\n\n- **Single node**: 1+ GPUs (or CPUs)\n- **Multi-node**: 2+ machines with network connectivity\n- **Cloud**: AWS, GCP, Azure (Ray autoscaling)\n- **On-prem**: Kubernetes, SLURM clusters\n\n**Supported accelerators**:\n- NVIDIA GPUs (CUDA)\n- AMD GPUs (ROCm)\n- TPUs (Google Cloud)\n- CPUs\n\n## Resources\n\n- Docs: https:\u002F\u002Fdocs.ray.io\u002Fen\u002Flatest\u002Ftrain\u002Ftrain.html\n- GitHub: https:\u002F\u002Fgithub.com\u002Fray-project\u002Fray ⭐ 36,000+\n- Version: 2.40.0+\n- Examples: https:\u002F\u002Fdocs.ray.io\u002Fen\u002Flatest\u002Ftrain\u002Fexamples.html\n- Slack: https:\u002F\u002Fforms.gle\u002F9TSdDYUgxYs8SA9e8\n- Used by: OpenAI, Uber, Spotify, Shopify, Instacart\n\n## Agent operating procedure\n\n1. **Check the environment.** Check GPU type, memory and driver\u002FCUDA versions (`nvidia-smi`), framework versions, and dataset location and size.\n2. **Pin down the inputs.** Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.\n3. **Run a small version first.** Do a smoke run: tiny model or subset, few steps, and confirm loss decreases and checkpoints save.\n4. **Execute the full task** using the instructions and references above.\n5. **Validate the result.** Track metrics on held-out data, compare against a baseline, and record seeds, configs and hardware.\n6. **Report.** State what was run (versions, commands, parameters), what was checked, and what is still uncertain.\n\n| If this happens | Do this |\n|---|---|\n| CUDA out-of-memory | Reduce batch size, enable gradient accumulation\u002Fcheckpointing or mixed precision, or shard the model. |\n| Loss is NaN or diverges | Lower the learning rate, check data for invalid values, and enable gradient clipping. |\n| A function, flag or endpoint in these instructions is missing in the installed version | Check the installed version's own documentation (`help()`, `--help`, official docs), adapt, and tell the user. Never invent an API. |\n| A required input, identifier or parameter is ambiguous | Ask the user, or state the assumption explicitly before running. |\n\n**Integrity rules**\n\n- Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.\n- Never claim training results without logs; estimate compute cost before launching large jobs.\n- Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.\n- Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.\n\n## Related skills\n\n- `ray-data`: Scalable data processing for ML workloads.\n- `pytorch-lightning-distributed`: High-level PyTorch framework with Trainer class, automatic distributed training (DDP\u002FFSDP\u002FDeepSpeed), callbacks system, and minimal boilerp…\n- `weights-and-biases`: Logs and tracks machine learning experiments with Weights & Biases (W&B, wandb): metrics, hyperparameters, checkpoints, sweeps, artifacts w…\n",{"name":5,"license":15,"metadata":47,"description":7},{"tags":48,"version":49,"category":50,"maintainer":51,"dependencies":52},"Ray Train, Distributed Training, Orchestration, Ray, Hyperparameter Tuning, Fault Tolerance, Elastic Scaling, Multi-Node, PyTorch, TensorFlow","1.0.0","ml-training","Kalaris Labs","ray[train], torch, transformers","main","2b63f856ce19c2a797929cacabd9e0fc936812fa","plugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fray-train",491,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins","https:\u002F\u002Fhol.org\u002Fplugins\u002Fbest-codex-plugins","https:\u002F\u002Fgithub.com\u002Fhashgraph-online","https:\u002F\u002Fhol.org","Open-source standards and developer tools for interoperable AI agents: discovery, identity, communication, payments, privacy and security.","2026-11-14 14:06:13",{"source":8,"skillName":5,"commands":64,"prompt":74,"zipAvailable":75},[65,68,71],{"tool":66,"command":67},"npx","npx skills add hashgraph-online\u002Fawesome-codex-plugins --skill ray-train",{"tool":69,"command":70},"bunx","bunx skills add hashgraph-online\u002Fawesome-codex-plugins --skill ray-train",{"tool":72,"command":73},"pnpm","pnpm dlx skills add hashgraph-online\u002Fawesome-codex-plugins --skill ray-train","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fray-train\u002FSKILL.md",true,[77],{"path":78,"resourceType":79,"resourceTypeDesc":80,"sizeBytes":81,"sourceHtmlUrl":82},"references\u002Fmulti-node.md","REFERENCE","参考文档",13526,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fray-train\u002Freferences\u002Fmulti-node.md",[84,93,101,109,117,125],{"skillNo":85,"name":86,"nameZh":6,"description":87,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":88,"sourceHtmlUrl":89,"categories":90,"keywords":91,"publishedAt":92},"01M7DV332YQMGD8WTRJ9DZQXTB","worktable-review-with-annotations","Create, inspect, reply to, update, or resolve situated Worktable annotations. Use when feedback belongs to a Doc section, block, text range, or HTML Doc, or when the user asks to address existing review feedback and coordinate a requested artifact edit.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-review-with-annotations\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-review-with-annotations\u002FSKILL.md",[],[],"2026-11-15 14:19:17",{"skillNo":94,"name":95,"nameZh":6,"description":96,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":97,"sourceHtmlUrl":98,"categories":99,"keywords":100,"publishedAt":92},"01M7DV2YGHCZVBEB1AHYB6CGHE","worktable-find-and-synthesize-context","Find, assess, and synthesize context from Worktable Docs and Records. Use when the user asks what a workspace contains, wants workspace-backed research, needs current facts reconciled across artifacts, or requests a cited synthesis without changing workspace state.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-find-and-synthesize-context\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-find-and-synthesize-context\u002FSKILL.md",[],[],{"skillNo":102,"name":103,"nameZh":6,"description":104,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":105,"sourceHtmlUrl":106,"categories":107,"keywords":108,"publishedAt":92},"01M7DV2TQN1H11ME2SAF9MXN7V","worktable-create-or-update-docs","Create, edit, rename, or maintain narrative Worktable Docs. Use for plans, decisions, briefs, notes, research syntheses, and other durable prose when the user requests a new artifact or a safe update to an existing one.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-update-docs\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-update-docs\u002FSKILL.md",[],[],{"skillNo":110,"name":111,"nameZh":6,"description":112,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":113,"sourceHtmlUrl":114,"categories":115,"keywords":116,"publishedAt":92},"01M7DV2NE452NEA8TXATKSNAJH","worktable-create-or-manage-records","Organize repeated structured information as durable Worktable Records. Use for research datasets, inventories, directories, pipelines, trackers, and other collections whose items need identity, fields, filtering, links, validation, or individual updates.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-records\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-records\u002FSKILL.md",[],[],{"skillNo":118,"name":119,"nameZh":6,"description":120,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":121,"sourceHtmlUrl":122,"categories":123,"keywords":124,"publishedAt":92},"01M7DV2HENRBR2KSCVM064WH0Y","worktable-create-or-manage-interactive-html","Create or improve durable visual and interactive artifacts in Worktable. Use for dashboards, trackers, calculators, mini apps, slide presentations, and other requests that benefit from a polished HTML Doc.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-interactive-html\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-interactive-html\u002FSKILL.md",[],[],{"skillNo":126,"name":127,"nameZh":6,"description":128,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":129,"sourceHtmlUrl":130,"categories":131,"keywords":132,"publishedAt":92},"01M7DV2CADR5R7RQFFYPNAGR9Y","worktable-collaborate-threads","Read, answer, and hand off messages through durable Worktable threads with users or other authenticated participants. Use when the user asks to check addressed messages, reply to a participant, continue a thread, request another agent's help, wait briefly for a response, or watch for comments and messages sent to the agent.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-collaborate-threads\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-collaborate-threads\u002FSKILL.md",[],[]]