[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M73AGQGY60WCCGBS18M2MPN6":3,"skill-same:01M73AGQGY60WCCGBS18M2MPN6":88},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":44,"commitSha":45,"skillDirectory":46,"forksCount":47,"repositoryHtmlUrl":48,"repositoryHomepageUrl":6,"ownerHtmlUrl":49,"ownerBlogUrl":50,"ownerDescription":6,"updatedAt":51,"install":52,"resources":65},"01M73AGQGY60WCCGBS18M2MPN6","fine-tuning-serving-openpi",null,"Fine-tune and serve Physical Intelligence OpenPI models (pi0, pi0-fast, pi0.5) using JAX or PyTorch backends for robot policy inference across ALOHA, DROID, and LIBERO environments. Use when adapting pi0 models to custom datasets, converting JAX checkpoints to PyTorch, running policy inference servers, or debugging norm stats and GPU memory issues.","qhjqhj00\u002Fresearch-skills-pool","qhjqhj00","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F29684092?v=4","01M72AE0ZGXVM5C57N512DGQG8","Tommy Chien","01M72AE107ESJMHT7QC5SXX144",3,"MIT","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--18-multimodal__openpi\u002FSKILL.md",[],[],"2026-11-11 15:11:14",false,"---\nname: fine-tuning-serving-openpi\ndescription: Fine-tune and serve Physical Intelligence OpenPI models (pi0, pi0-fast, pi0.5) using JAX or PyTorch backends for robot policy inference across ALOHA, DROID, and LIBERO environments. Use when adapting pi0 models to custom datasets, converting JAX checkpoints to PyTorch, running policy inference servers, or debugging norm stats and GPU memory issues.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [OpenPI, Physical Intelligence, VLA, Robotics, JAX, PyTorch, Fine-Tuning, Policy Serving, ALOHA, DROID, LIBERO, pi0]\ndependencies: [uv>=0.4.0, jax>=0.4.30, torch>=2.1.0, transformers>=4.53.2]\n---\n\n# OpenPI Fine-Tuning and Serving\n\nEnd-to-end workflows for fine-tuning and serving Physical Intelligence's OpenPI models (pi0, pi0-fast, pi0.5) on robot manipulation tasks from the public `openpi` repository. Covers blank-machine setup, JAX training, PyTorch training, checkpoint conversion, and policy inference serving.\n\n## Quick start\n\nClone the public repo, install the workspace, then serve a pretrained policy:\n\n```bash\ngit clone --recurse-submodules https:\u002F\u002Fgithub.com\u002FPhysical-Intelligence\u002Fopenpi.git\ncd openpi\nGIT_LFS_SKIP_SMUDGE=1 uv sync\nGIT_LFS_SKIP_SMUDGE=1 uv pip install -e .\nuv run scripts\u002Fserve_policy.py --env DROID\n```\n\n```python\nfrom openpi_client import websocket_client_policy\n\nclient = websocket_client_policy.WebsocketClientPolicy(host=\"localhost\", port=8000)\nresult = client.infer(observation)\nactions = result[\"actions\"]  # numpy array of shape (chunk_size, action_dim)\n```\n\n## Core concepts\n\n**Model family**: OpenPI implements three model variants from Physical Intelligence:\n\n| Model | Architecture | Speed | Quality | Typical use |\n|-------|-------------|-------|---------|-------------|\n| pi0 | Flow-matching VLA | Baseline | Highest | Research, complex tasks |\n| pi0-fast | Autoregressive action tokens | 2-5x faster | Good | Real-time control |\n| pi0.5 | pi0 + improved vision encoder | Baseline | Best | Latest default |\n\n**Key design choices**:\n- **Dual backend**: JAX (primary, official training) and PyTorch (community, deployment-friendly)\n- **Config-driven**: All training\u002Fserving parameters defined in `src\u002Fopenpi\u002Ftraining\u002Fconfig.py`\n- **Norm stats**: Every config requires precomputed normalization statistics before training\n- **WebSocket serving**: Policy servers expose a WebSocket API for low-latency inference\n\n**Training loop invariant**: After every config or dataset change, always re-run this cycle:\n1. Compute norm stats → 2. Train → 3. Serve checkpoint → 4. Validate inference\n\n## Compute requirements\n\n| Task | GPU | VRAM | Notes |\n|------|-----|------|-------|\n| Serve pi0.5 (inference) | 1x A100\u002FH100 | ~24 GB | Single GPU sufficient |\n| Fine-tune pi0.5 (JAX) | 1x A100 80GB | ~60 GB | Use `fsdp_devices` for multi-GPU |\n| Fine-tune pi0 (JAX) | 1x A100 80GB | ~40 GB | Smaller model footprint |\n| Fine-tune (PyTorch DDP) | 1-8x A100 | ~40 GB\u002FGPU | torchrun launcher |\n| Compute norm stats | CPU or 1x GPU | ~8 GB | Fast, can run on login node |\n\n## Workflow 0: Blank-machine setup\n\nCopy this checklist and track progress:\n\n```text\nSetup Progress:\n- [ ] Step 1: Clone the public openpi repo with submodules\n- [ ] Step 2: Install uv and sync the workspace\n- [ ] Step 3: Install the editable package\n- [ ] Step 4: Verify core imports and serving entrypoint\n```\n\n**Step 1: Clone repo**\n\n```bash\ngit clone --recurse-submodules https:\u002F\u002Fgithub.com\u002FPhysical-Intelligence\u002Fopenpi.git\ncd openpi\n```\n\nIf you already cloned without submodules:\n\n```bash\ngit submodule update --init --recursive\n```\n\n**Step 2: Sync dependencies**\n\n```bash\nGIT_LFS_SKIP_SMUDGE=1 uv sync\n```\n\n**Step 3: Install editable package**\n\n```bash\nGIT_LFS_SKIP_SMUDGE=1 uv pip install -e .\n```\n\n**Step 4: Verify installation**\n\n```bash\nuv run python -c \"from openpi.training import config as _config; print(_config.get_config('pi05_droid').name)\"\nuv run scripts\u002Fserve_policy.py --help\n```\n\n## When to use vs alternatives\n\n**Use this skill when:**\n- Fine-tuning pi0, pi0-fast, or pi0.5 on LeRobot or RLDS datasets\n- Serving OpenPI policies for ALOHA, DROID, or LIBERO evaluation\n- Converting JAX checkpoints to PyTorch format\n- Debugging OpenPI training issues (norm stats, memory, config)\n\n**Use `fine-tuning-openvla-oft` instead when:**\n- Fine-tuning OpenVLA with continuous action heads and LoRA\n- Reproducing OpenVLA-OFT paper results on LIBERO or ALOHA\n\n**Use `evaluating-cosmos-policy` instead when:**\n- Evaluating NVIDIA Cosmos Policy on simulation benchmarks\n\n---\n\n## Workflow 1: JAX fine-tuning on LeRobot data\n\nCopy this checklist and track progress:\n\n```text\nJAX Fine-Tuning Progress:\n- [ ] Step 1: Select and copy closest training config\n- [ ] Step 2: Update dataset mapping and base checkpoint\n- [ ] Step 3: Compute normalization statistics\n- [ ] Step 4: Launch JAX training\n- [ ] Step 5: Serve checkpoint and run inference sanity check\n```\n\n**Step 1: Select config**\n\nCopy the closest config from `src\u002Fopenpi\u002Ftraining\u002Fconfig.py`:\n\n| Config | Use case |\n|--------|----------|\n| `pi05_libero` | pi0.5 LIBERO fine-tuning |\n| `pi0_libero` | pi0 full fine-tuning on LIBERO |\n| `pi0_fast_libero` | pi0-fast on LIBERO |\n| `pi0_aloha_pen_uncap` | ALOHA custom data |\n| `pi05_droid_finetune` | Small custom DROID dataset (LeRobot format) |\n| `pi05_full_droid_finetune` | Full DROID RLDS large-scale training |\n\n**Step 2: Update dataset and transforms**\n\n```python\n# In src\u002Fopenpi\u002Ftraining\u002Fconfig.py, modify your config:\nTrainConfig(\n    name=\"my_custom_config\",\n    model_type=\"pi05\",\n    data=LeRobotDataConfig(\n        repo_id=\"your-org\u002Fyour-dataset\",\n        # Adjust transforms to match your data format\n    ),\n    weight_loader=Pi05WeightLoader(),  # Match model type\n)\n```\n\nSet `repo_id` for your dataset and ensure `weight_loader` matches the model type (pi0 vs pi0.5).\n\n**Step 3: Compute normalization statistics**\n\n```bash\nuv run scripts\u002Fcompute_norm_stats.py --config-name \u003Cconfig_name>\n```\n\nThis must run before every training launch when config, dataset, or transforms change.\n\n**Step 4: Launch JAX training**\n\n```bash\nXLA_PYTHON_CLIENT_MEM_FRACTION=0.9 uv run scripts\u002Ftrain.py \u003Cconfig_name> \\\n  --exp-name=\u003Crun_name> \\\n  --overwrite\n```\n\nFor full DROID RLDS training, add the `rlds` dependency group:\n\n```bash\nuv run --group rlds scripts\u002Fcompute_norm_stats.py \\\n  --config-name pi05_full_droid_finetune \\\n  --max-frames 10000000\n\nXLA_PYTHON_CLIENT_MEM_FRACTION=0.9 uv run --group rlds scripts\u002Ftrain.py \\\n  pi05_full_droid_finetune \\\n  --exp-name=\u003Crun_name> --overwrite\n```\n\n**Step 5: Serve and validate**\n\n```bash\nuv run scripts\u002Fserve_policy.py policy:checkpoint \\\n  --policy.config=\u003Cconfig_name> \\\n  --policy.dir=checkpoints\u002F\u003Cconfig_name>\u002F\u003Crun_name>\u002F\u003Cstep>\n```\n\nVerify with a test client:\n\n```python\nfrom openpi_client import websocket_client_policy\n\nclient = websocket_client_policy.WebsocketClientPolicy(host=\"localhost\", port=8000)\n# Build observation matching your config's expected keys\nobs = {\"image\": img_array, \"state\": state_array, \"prompt\": \"pick up the cup\"}\nresult = client.infer(obs)\nprint(f\"Action shape: {result['actions'].shape}\")  # (chunk_size, action_dim)\n```\n\n---\n\n## Workflow 2: PyTorch training and checkpoint conversion\n\nCopy this checklist and track progress:\n\n```text\nPyTorch Setup Progress:\n- [ ] Step 1: Sync dependencies and verify transformer version\n- [ ] Step 2: Apply OpenPI transformer patches\n- [ ] Step 3: Convert JAX checkpoint to PyTorch format\n- [ ] Step 4: Launch PyTorch training or serve converted checkpoint\n```\n\n**Step 1: Sync dependencies**\n\n```bash\nuv sync\nuv pip show transformers\n```\n\n**Step 2: Apply required patches**\n\nOpenPI PyTorch requires custom modifications to the installed `transformers` package:\n\n```bash\ncp -r .\u002Fsrc\u002Fopenpi\u002Fmodels_pytorch\u002Ftransformers_replace\u002F* \\\n  .venv\u002Flib\u002Fpython3.11\u002Fsite-packages\u002Ftransformers\u002F\n```\n\n**Step 3: Convert JAX checkpoint**\n\n```bash\nuv run examples\u002Fconvert_jax_model_to_pytorch.py \\\n  --checkpoint_dir \u003Cjax_checkpoint_dir> \\\n  --config_name \u003Cconfig_name> \\\n  --output_path \u003Cpytorch_checkpoint_dir>\n```\n\n**Step 4: Train or serve**\n\nSingle GPU training:\n\n```bash\nuv run scripts\u002Ftrain_pytorch.py \u003Cconfig_name> --exp_name \u003Crun_name>\n```\n\nMulti-GPU distributed training:\n\n```bash\nuv run torchrun --standalone --nnodes=1 --nproc_per_node=\u003Cnum_gpus> \\\n  scripts\u002Ftrain_pytorch.py \u003Cconfig_name> --exp_name \u003Crun_name>\n```\n\nProgrammatic inference with converted checkpoint:\n\n```python\nfrom openpi.training import config as _config\nfrom openpi.policies import policy_config\n\nconfig = _config.get_config(\"pi05_droid\")\npolicy = policy_config.create_trained_policy(config, \"\u003Cpytorch_checkpoint_dir>\")\nresult = policy.infer(example)\nactions = result[\"actions\"]  # numpy array\n```\n\nCheckpoints follow the convention: `checkpoints\u002F\u003Cconfig_name>\u002F\u003Cexp_name>\u002F\u003Cstep>\u002F`.\n\n---\n\n## Workflow 3: Policy inference serving\n\nCopy this checklist and track progress:\n\n```text\nInference Server Progress:\n- [ ] Step 1: Choose target environment and checkpoint\n- [ ] Step 2: Start policy server\n- [ ] Step 3: Confirm server is reachable\n- [ ] Step 4: Integrate client into robot or simulation code\n```\n\n**Step 1: Choose environment**\n\nDefault environment presets:\n\n| Environment | Config | Default checkpoint |\n|-------------|--------|--------------------|\n| `ALOHA` | `pi05_aloha` | `gs:\u002F\u002Fopenpi-assets\u002Fcheckpoints\u002Fpi05_base` |\n| `ALOHA_SIM` | `pi0_aloha_sim` | `gs:\u002F\u002Fopenpi-assets\u002Fcheckpoints\u002Fpi0_aloha_sim` |\n| `DROID` | `pi05_droid` | `gs:\u002F\u002Fopenpi-assets\u002Fcheckpoints\u002Fpi05_droid` |\n| `LIBERO` | `pi05_libero` | `gs:\u002F\u002Fopenpi-assets\u002Fcheckpoints\u002Fpi05_libero` |\n\n**Step 2: Start server**\n\nDefault mode (uses preset checkpoint):\n\n```bash\nuv run scripts\u002Fserve_policy.py --env ALOHA\n```\n\nExplicit checkpoint mode (custom or local model):\n\n```bash\nuv run scripts\u002Fserve_policy.py policy:checkpoint \\\n  --policy.config=pi05_libero \\\n  --policy.dir=checkpoints\u002Fpi05_libero\u002Fmy_run\u002F20000\n```\n\nAdd `--default_prompt \"task description\"` when runtime observations omit a prompt.\n\n**Step 3: Verify connectivity**\n\n```bash\nuv run examples\u002Fsimple_client\u002Fmain.py --env DROID\n```\n\n**Step 4: Embed remote client in robot code**\n\nInstall the lightweight client in your robot environment:\n\n```bash\npip install \"openpi-client @ git+https:\u002F\u002Fgithub.com\u002FPhysical-Intelligence\u002Fopenpi.git#subdirectory=packages\u002Fopenpi-client\"\n```\n\nFull integration example:\n\n```python\nfrom openpi_client import websocket_client_policy\nimport numpy as np\n\n# Connect to remote policy server\nclient = websocket_client_policy.WebsocketClientPolicy(\n    host=\"gpu-server.local\", port=8000\n)\n\n# Build observation (keys must match policy transforms)\nobservation = {\n    \"image\": np.random.rand(224, 224, 3),  # RGB image\n    \"state\": np.zeros(7),                   # Joint positions\n    \"prompt\": \"pick up the red block\",\n}\n\n# Get actions\nresult = client.infer(observation)\nactions = result[\"actions\"]  # shape: (action_chunk_size, action_dim)\n\n# Execute first action on robot\nrobot.step(actions[0])\n```\n\n---\n\n## Common issues\n\n**Issue: Missing norm stats error**\n\nFix: run `scripts\u002Fcompute_norm_stats.py --config-name \u003Cconfig_name>` before training.\n\n**Issue: Out of memory during JAX training**\n\nFix: set `XLA_PYTHON_CLIENT_MEM_FRACTION=0.9`, lower batch size, or configure `fsdp_devices`:\n\n```python\n# In config: use model-parallel sharding\nTrainConfig(\n    ...\n    fsdp_devices=4,  # Shard across 4 GPUs\n)\n```\n\n**Issue: OOM while loading PyTorch checkpoints**\n\nFix: `export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`\n\n**Issue: Config not found**\n\nFix: ensure config name exists in `src\u002Fopenpi\u002Ftraining\u002Fconfig.py` (exact match from `_CONFIGS` dict).\n\n**Issue: PyTorch training diverges after library changes**\n\nFix: reapply the transformer patch. Run `uv cache clean transformers` to reset, then reapply.\n\n**Issue: `serve_policy.py` crashes with `ModuleNotFoundError`**\n\nFix: resync the public workspace first:\n\n```bash\nGIT_LFS_SKIP_SMUDGE=1 uv sync\nGIT_LFS_SKIP_SMUDGE=1 uv pip install -e .\n```\n\nIf the missing module is simulator-related, install the extra runtime dependencies called for by that example:\n\n```bash\nuv pip install pytest robosuite==1.4.0 gym bddl easydict matplotlib\n```\n\n**Issue: `uv sync` fails with `rerun-sdk` wheel mismatch**\n\nFix:\n\n```bash\nuv sync --no-dev\n# or\nuv sync --no-dev --no-install-package rerun-sdk\n```\n\n**Issue: Checkpoint download times out**\n\nFix: install `gsutil` and prefetch manually:\n\n```bash\npip install gsutil\ngsutil -m cp -r gs:\u002F\u002Fopenpi-assets\u002Fcheckpoints\u002Fpi05_libero \u002Flocal\u002Fcache\u002F\n```\n\nRemove stale `.lock` files if a previous download was interrupted.\n\n**Issue: Policy server exits with code `137`**\n\nFix: OOM kill. Set JAX memory variables:\n\n```bash\nexport XLA_PYTHON_CLIENT_PREALLOCATE=false\nexport XLA_PYTHON_CLIENT_ALLOCATOR=platform\n```\n\n---\n\n## For HPC\u002Fcluster users\n\nOn Slurm-managed clusters, wrap commands with resource allocation:\n\n```bash\nsrun --partition=gpu --gpus-per-node=1 --mem=64G --cpus-per-task=8 --pty bash\n```\n\nRoute caches to scratch to avoid filling `\u002Fhome`:\n\n```bash\nexport HF_HOME=\u002Fscratch\u002F$USER\u002F.cache\u002Fhuggingface\nexport XDG_CACHE_HOME=\u002Fscratch\u002F$USER\u002F.cache\nexport PIP_CACHE_DIR=\u002Fscratch\u002F$USER\u002F.cache\u002Fpip\nexport UV_CACHE_DIR=\u002Fscratch\u002F$USER\u002F.cache\u002Fuv\n```\n\nAvoid stacking cluster Python modules when using uv-managed environments. Typically `module load cuda` is sufficient.\n\n---\n\n## Advanced topics\n\n**Config recipes and baselines**: See [references\u002Fconfig-recipes.md](references\u002Fconfig-recipes.md)\n**Training debugging guide**: See [references\u002Ftraining-debugging.md](references\u002Ftraining-debugging.md)\n**Checkpoint and environment mapping**: See [references\u002Fcheckpoints-and-env-map.md](references\u002Fcheckpoints-and-env-map.md)\n**Remote client integration**: See [references\u002Fremote-client-pattern.md](references\u002Fremote-client-pattern.md)\n**PyTorch precision and patching gotchas**: See [references\u002Fpytorch-gotchas.md](references\u002Fpytorch-gotchas.md)\n\n## Resources\n\n- OpenPI repository: https:\u002F\u002Fgithub.com\u002FPhysical-Intelligence\u002Fopenpi\n- OpenPI client package: https:\u002F\u002Fgithub.com\u002FPhysical-Intelligence\u002Fopenpi\u002Ftree\u002Fmain\u002Fpackages\u002Fopenpi-client\n- pi0 paper: https:\u002F\u002Fwww.physicalintelligence.company\u002Fblog\u002Fpi0\n- LeRobot dataset format: https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Flerobot\n","# OpenPI Fine-Tuning and Serving\n\nEnd-to-end workflows for fine-tuning and serving Physical Intelligence's OpenPI models (pi0, pi0-fast, pi0.5) on robot manipulation tasks from the public `openpi` repository. Covers blank-machine setup, JAX training, PyTorch training, checkpoint conversion, and policy inference serving.\n\n## Quick start\n\nClone the public repo, install the workspace, then serve a pretrained policy:\n\n```bash\ngit clone --recurse-submodules https:\u002F\u002Fgithub.com\u002FPhysical-Intelligence\u002Fopenpi.git\ncd openpi\nGIT_LFS_SKIP_SMUDGE=1 uv sync\nGIT_LFS_SKIP_SMUDGE=1 uv pip install -e .\nuv run scripts\u002Fserve_policy.py --env DROID\n```\n\n```python\nfrom openpi_client import websocket_client_policy\n\nclient = websocket_client_policy.WebsocketClientPolicy(host=\"localhost\", port=8000)\nresult = client.infer(observation)\nactions = result[\"actions\"]  # numpy array of shape (chunk_size, action_dim)\n```\n\n## Core concepts\n\n**Model family**: OpenPI implements three model variants from Physical Intelligence:\n\n| Model | Architecture | Speed | Quality | Typical use |\n|-------|-------------|-------|---------|-------------|\n| pi0 | Flow-matching VLA | Baseline | Highest | Research, complex tasks |\n| pi0-fast | Autoregressive action tokens | 2-5x faster | Good | Real-time control |\n| pi0.5 | pi0 + improved vision encoder | Baseline | Best | Latest default |\n\n**Key design choices**:\n- **Dual backend**: JAX (primary, official training) and PyTorch (community, deployment-friendly)\n- **Config-driven**: All training\u002Fserving parameters defined in `src\u002Fopenpi\u002Ftraining\u002Fconfig.py`\n- **Norm stats**: Every config requires precomputed normalization statistics before training\n- **WebSocket serving**: Policy servers expose a WebSocket API for low-latency inference\n\n**Training loop invariant**: After every config or dataset change, always re-run this cycle:\n1. Compute norm stats → 2. Train → 3. Serve checkpoint → 4. Validate inference\n\n## Compute requirements\n\n| Task | GPU | VRAM | Notes |\n|------|-----|------|-------|\n| Serve pi0.5 (inference) | 1x A100\u002FH100 | ~24 GB | Single GPU sufficient |\n| Fine-tune pi0.5 (JAX) | 1x A100 80GB | ~60 GB | Use `fsdp_devices` for multi-GPU |\n| Fine-tune pi0 (JAX) | 1x A100 80GB | ~40 GB | Smaller model footprint |\n| Fine-tune (PyTorch DDP) | 1-8x A100 | ~40 GB\u002FGPU | torchrun launcher |\n| Compute norm stats | CPU or 1x GPU | ~8 GB | Fast, can run on login node |\n\n## Workflow 0: Blank-machine setup\n\nCopy this checklist and track progress:\n\n```text\nSetup Progress:\n- [ ] Step 1: Clone the public openpi repo with submodules\n- [ ] Step 2: Install uv and sync the workspace\n- [ ] Step 3: Install the editable package\n- [ ] Step 4: Verify core imports and serving entrypoint\n```\n\n**Step 1: Clone repo**\n\n```bash\ngit clone --recurse-submodules https:\u002F\u002Fgithub.com\u002FPhysical-Intelligence\u002Fopenpi.git\ncd openpi\n```\n\nIf you already cloned without submodules:\n\n```bash\ngit submodule update --init --recursive\n```\n\n**Step 2: Sync dependencies**\n\n```bash\nGIT_LFS_SKIP_SMUDGE=1 uv sync\n```\n\n**Step 3: Install editable package**\n\n```bash\nGIT_LFS_SKIP_SMUDGE=1 uv pip install -e .\n```\n\n**Step 4: Verify installation**\n\n```bash\nuv run python -c \"from openpi.training import config as _config; print(_config.get_config('pi05_droid').name)\"\nuv run scripts\u002Fserve_policy.py --help\n```\n\n## When to use vs alternatives\n\n**Use this skill when:**\n- Fine-tuning pi0, pi0-fast, or pi0.5 on LeRobot or RLDS datasets\n- Serving OpenPI policies for ALOHA, DROID, or LIBERO evaluation\n- Converting JAX checkpoints to PyTorch format\n- Debugging OpenPI training issues (norm stats, memory, config)\n\n**Use `fine-tuning-openvla-oft` instead when:**\n- Fine-tuning OpenVLA with continuous action heads and LoRA\n- Reproducing OpenVLA-OFT paper results on LIBERO or ALOHA\n\n**Use `evaluating-cosmos-policy` instead when:**\n- Evaluating NVIDIA Cosmos Policy on simulation benchmarks\n\n---\n\n## Workflow 1: JAX fine-tuning on LeRobot data\n\nCopy this checklist and track progress:\n\n```text\nJAX Fine-Tuning Progress:\n- [ ] Step 1: Select and copy closest training config\n- [ ] Step 2: Update dataset mapping and base checkpoint\n- [ ] Step 3: Compute normalization statistics\n- [ ] Step 4: Launch JAX training\n- [ ] Step 5: Serve checkpoint and run inference sanity check\n```\n\n**Step 1: Select config**\n\nCopy the closest config from `src\u002Fopenpi\u002Ftraining\u002Fconfig.py`:\n\n| Config | Use case |\n|--------|----------|\n| `pi05_libero` | pi0.5 LIBERO fine-tuning |\n| `pi0_libero` | pi0 full fine-tuning on LIBERO |\n| `pi0_fast_libero` | pi0-fast on LIBERO |\n| `pi0_aloha_pen_uncap` | ALOHA custom data |\n| `pi05_droid_finetune` | Small custom DROID dataset (LeRobot format) |\n| `pi05_full_droid_finetune` | Full DROID RLDS large-scale training |\n\n**Step 2: Update dataset and transforms**\n\n```python\n# In src\u002Fopenpi\u002Ftraining\u002Fconfig.py, modify your config:\nTrainConfig(\n    name=\"my_custom_config\",\n    model_type=\"pi05\",\n    data=LeRobotDataConfig(\n        repo_id=\"your-org\u002Fyour-dataset\",\n        # Adjust transforms to match your data format\n    ),\n    weight_loader=Pi05WeightLoader(),  # Match model type\n)\n```\n\nSet `repo_id` for your dataset and ensure `weight_loader` matches the model type (pi0 vs pi0.5).\n\n**Step 3: Compute normalization statistics**\n\n```bash\nuv run scripts\u002Fcompute_norm_stats.py --config-name \u003Cconfig_name>\n```\n\nThis must run before every training launch when config, dataset, or transforms change.\n\n**Step 4: Launch JAX training**\n\n```bash\nXLA_PYTHON_CLIENT_MEM_FRACTION=0.9 uv run scripts\u002Ftrain.py \u003Cconfig_name> \\\n  --exp-name=\u003Crun_name> \\\n  --overwrite\n```\n\nFor full DROID RLDS training, add the `rlds` dependency group:\n\n```bash\nuv run --group rlds scripts\u002Fcompute_norm_stats.py \\\n  --config-name pi05_full_droid_finetune \\\n  --max-frames 10000000\n\nXLA_PYTHON_CLIENT_MEM_FRACTION=0.9 uv run --group rlds scripts\u002Ftrain.py \\\n  pi05_full_droid_finetune \\\n  --exp-name=\u003Crun_name> --overwrite\n```\n\n**Step 5: Serve and validate**\n\n```bash\nuv run scripts\u002Fserve_policy.py policy:checkpoint \\\n  --policy.config=\u003Cconfig_name> \\\n  --policy.dir=checkpoints\u002F\u003Cconfig_name>\u002F\u003Crun_name>\u002F\u003Cstep>\n```\n\nVerify with a test client:\n\n```python\nfrom openpi_client import websocket_client_policy\n\nclient = websocket_client_policy.WebsocketClientPolicy(host=\"localhost\", port=8000)\n# Build observation matching your config's expected keys\nobs = {\"image\": img_array, \"state\": state_array, \"prompt\": \"pick up the cup\"}\nresult = client.infer(obs)\nprint(f\"Action shape: {result['actions'].shape}\")  # (chunk_size, action_dim)\n```\n\n---\n\n## Workflow 2: PyTorch training and checkpoint conversion\n\nCopy this checklist and track progress:\n\n```text\nPyTorch Setup Progress:\n- [ ] Step 1: Sync dependencies and verify transformer version\n- [ ] Step 2: Apply OpenPI transformer patches\n- [ ] Step 3: Convert JAX checkpoint to PyTorch format\n- [ ] Step 4: Launch PyTorch training or serve converted checkpoint\n```\n\n**Step 1: Sync dependencies**\n\n```bash\nuv sync\nuv pip show transformers\n```\n\n**Step 2: Apply required patches**\n\nOpenPI PyTorch requires custom modifications to the installed `transformers` package:\n\n```bash\ncp -r .\u002Fsrc\u002Fopenpi\u002Fmodels_pytorch\u002Ftransformers_replace\u002F* \\\n  .venv\u002Flib\u002Fpython3.11\u002Fsite-packages\u002Ftransformers\u002F\n```\n\n**Step 3: Convert JAX checkpoint**\n\n```bash\nuv run examples\u002Fconvert_jax_model_to_pytorch.py \\\n  --checkpoint_dir \u003Cjax_checkpoint_dir> \\\n  --config_name \u003Cconfig_name> \\\n  --output_path \u003Cpytorch_checkpoint_dir>\n```\n\n**Step 4: Train or serve**\n\nSingle GPU training:\n\n```bash\nuv run scripts\u002Ftrain_pytorch.py \u003Cconfig_name> --exp_name \u003Crun_name>\n```\n\nMulti-GPU distributed training:\n\n```bash\nuv run torchrun --standalone --nnodes=1 --nproc_per_node=\u003Cnum_gpus> \\\n  scripts\u002Ftrain_pytorch.py \u003Cconfig_name> --exp_name \u003Crun_name>\n```\n\nProgrammatic inference with converted checkpoint:\n\n```python\nfrom openpi.training import config as _config\nfrom openpi.policies import policy_config\n\nconfig = _config.get_config(\"pi05_droid\")\npolicy = policy_config.create_trained_policy(config, \"\u003Cpytorch_checkpoint_dir>\")\nresult = policy.infer(example)\nactions = result[\"actions\"]  # numpy array\n```\n\nCheckpoints follow the convention: `checkpoints\u002F\u003Cconfig_name>\u002F\u003Cexp_name>\u002F\u003Cstep>\u002F`.\n\n---\n\n## Workflow 3: Policy inference serving\n\nCopy this checklist and track progress:\n\n```text\nInference Server Progress:\n- [ ] Step 1: Choose target environment and checkpoint\n- [ ] Step 2: Start policy server\n- [ ] Step 3: Confirm server is reachable\n- [ ] Step 4: Integrate client into robot or simulation code\n```\n\n**Step 1: Choose environment**\n\nDefault environment presets:\n\n| Environment | Config | Default checkpoint |\n|-------------|--------|--------------------|\n| `ALOHA` | `pi05_aloha` | `gs:\u002F\u002Fopenpi-assets\u002Fcheckpoints\u002Fpi05_base` |\n| `ALOHA_SIM` | `pi0_aloha_sim` | `gs:\u002F\u002Fopenpi-assets\u002Fcheckpoints\u002Fpi0_aloha_sim` |\n| `DROID` | `pi05_droid` | `gs:\u002F\u002Fopenpi-assets\u002Fcheckpoints\u002Fpi05_droid` |\n| `LIBERO` | `pi05_libero` | `gs:\u002F\u002Fopenpi-assets\u002Fcheckpoints\u002Fpi05_libero` |\n\n**Step 2: Start server**\n\nDefault mode (uses preset checkpoint):\n\n```bash\nuv run scripts\u002Fserve_policy.py --env ALOHA\n```\n\nExplicit checkpoint mode (custom or local model):\n\n```bash\nuv run scripts\u002Fserve_policy.py policy:checkpoint \\\n  --policy.config=pi05_libero \\\n  --policy.dir=checkpoints\u002Fpi05_libero\u002Fmy_run\u002F20000\n```\n\nAdd `--default_prompt \"task description\"` when runtime observations omit a prompt.\n\n**Step 3: Verify connectivity**\n\n```bash\nuv run examples\u002Fsimple_client\u002Fmain.py --env DROID\n```\n\n**Step 4: Embed remote client in robot code**\n\nInstall the lightweight client in your robot environment:\n\n```bash\npip install \"openpi-client @ git+https:\u002F\u002Fgithub.com\u002FPhysical-Intelligence\u002Fopenpi.git#subdirectory=packages\u002Fopenpi-client\"\n```\n\nFull integration example:\n\n```python\nfrom openpi_client import websocket_client_policy\nimport numpy as np\n\n# Connect to remote policy server\nclient = websocket_client_policy.WebsocketClientPolicy(\n    host=\"gpu-server.local\", port=8000\n)\n\n# Build observation (keys must match policy transforms)\nobservation = {\n    \"image\": np.random.rand(224, 224, 3),  # RGB image\n    \"state\": np.zeros(7),                   # Joint positions\n    \"prompt\": \"pick up the red block\",\n}\n\n# Get actions\nresult = client.infer(observation)\nactions = result[\"actions\"]  # shape: (action_chunk_size, action_dim)\n\n# Execute first action on robot\nrobot.step(actions[0])\n```\n\n---\n\n## Common issues\n\n**Issue: Missing norm stats error**\n\nFix: run `scripts\u002Fcompute_norm_stats.py --config-name \u003Cconfig_name>` before training.\n\n**Issue: Out of memory during JAX training**\n\nFix: set `XLA_PYTHON_CLIENT_MEM_FRACTION=0.9`, lower batch size, or configure `fsdp_devices`:\n\n```python\n# In config: use model-parallel sharding\nTrainConfig(\n    ...\n    fsdp_devices=4,  # Shard across 4 GPUs\n)\n```\n\n**Issue: OOM while loading PyTorch checkpoints**\n\nFix: `export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`\n\n**Issue: Config not found**\n\nFix: ensure config name exists in `src\u002Fopenpi\u002Ftraining\u002Fconfig.py` (exact match from `_CONFIGS` dict).\n\n**Issue: PyTorch training diverges after library changes**\n\nFix: reapply the transformer patch. Run `uv cache clean transformers` to reset, then reapply.\n\n**Issue: `serve_policy.py` crashes with `ModuleNotFoundError`**\n\nFix: resync the public workspace first:\n\n```bash\nGIT_LFS_SKIP_SMUDGE=1 uv sync\nGIT_LFS_SKIP_SMUDGE=1 uv pip install -e .\n```\n\nIf the missing module is simulator-related, install the extra runtime dependencies called for by that example:\n\n```bash\nuv pip install pytest robosuite==1.4.0 gym bddl easydict matplotlib\n```\n\n**Issue: `uv sync` fails with `rerun-sdk` wheel mismatch**\n\nFix:\n\n```bash\nuv sync --no-dev\n# or\nuv sync --no-dev --no-install-package rerun-sdk\n```\n\n**Issue: Checkpoint download times out**\n\nFix: install `gsutil` and prefetch manually:\n\n```bash\npip install gsutil\ngsutil -m cp -r gs:\u002F\u002Fopenpi-assets\u002Fcheckpoints\u002Fpi05_libero \u002Flocal\u002Fcache\u002F\n```\n\nRemove stale `.lock` files if a previous download was interrupted.\n\n**Issue: Policy server exits with code `137`**\n\nFix: OOM kill. Set JAX memory variables:\n\n```bash\nexport XLA_PYTHON_CLIENT_PREALLOCATE=false\nexport XLA_PYTHON_CLIENT_ALLOCATOR=platform\n```\n\n---\n\n## For HPC\u002Fcluster users\n\nOn Slurm-managed clusters, wrap commands with resource allocation:\n\n```bash\nsrun --partition=gpu --gpus-per-node=1 --mem=64G --cpus-per-task=8 --pty bash\n```\n\nRoute caches to scratch to avoid filling `\u002Fhome`:\n\n```bash\nexport HF_HOME=\u002Fscratch\u002F$USER\u002F.cache\u002Fhuggingface\nexport XDG_CACHE_HOME=\u002Fscratch\u002F$USER\u002F.cache\nexport PIP_CACHE_DIR=\u002Fscratch\u002F$USER\u002F.cache\u002Fpip\nexport UV_CACHE_DIR=\u002Fscratch\u002F$USER\u002F.cache\u002Fuv\n```\n\nAvoid stacking cluster Python modules when using uv-managed environments. Typically `module load cuda` is sufficient.\n\n---\n\n## Advanced topics\n\n**Config recipes and baselines**: See [references\u002Fconfig-recipes.md](references\u002Fconfig-recipes.md)\n**Training debugging guide**: See [references\u002Ftraining-debugging.md](references\u002Ftraining-debugging.md)\n**Checkpoint and environment mapping**: See [references\u002Fcheckpoints-and-env-map.md](references\u002Fcheckpoints-and-env-map.md)\n**Remote client integration**: See [references\u002Fremote-client-pattern.md](references\u002Fremote-client-pattern.md)\n**PyTorch precision and patching gotchas**: See [references\u002Fpytorch-gotchas.md](references\u002Fpytorch-gotchas.md)\n\n## Resources\n\n- OpenPI repository: https:\u002F\u002Fgithub.com\u002FPhysical-Intelligence\u002Fopenpi\n- OpenPI client package: https:\u002F\u002Fgithub.com\u002FPhysical-Intelligence\u002Fopenpi\u002Ftree\u002Fmain\u002Fpackages\u002Fopenpi-client\n- pi0 paper: https:\u002F\u002Fwww.physicalintelligence.company\u002Fblog\u002Fpi0\n- LeRobot dataset format: https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Flerobot\n",{"name":5,"tags":24,"author":37,"license":15,"version":38,"description":7,"dependencies":39},[25,26,27,28,29,30,31,32,33,34,35,36],"OpenPI","Physical Intelligence","VLA","Robotics","JAX","PyTorch","Fine-Tuning","Policy Serving","ALOHA","DROID","LIBERO","pi0","Orchestra Research","1.0.0",[40,41,42,43],"uv>=0.4.0","jax>=0.4.30","torch>=2.1.0","transformers>=4.53.2","master","c247302c836de6438d746b45670990eb71691166","skills_collected\u002FOrchestra-Research--AI-research-SKILLs--18-multimodal__openpi",1,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool","https:\u002F\u002Fgithub.com\u002Fqhjqhj00","chien.io","2026-05-11 15:07:59",{"source":8,"skillName":5,"commands":53,"prompt":63,"zipAvailable":64},[54,57,60],{"tool":55,"command":56},"npx","npx skills add qhjqhj00\u002Fresearch-skills-pool --skill fine-tuning-serving-openpi",{"tool":58,"command":59},"bunx","bunx skills add qhjqhj00\u002Fresearch-skills-pool --skill fine-tuning-serving-openpi",{"tool":61,"command":62},"pnpm","pnpm dlx skills add qhjqhj00\u002Fresearch-skills-pool --skill fine-tuning-serving-openpi","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--18-multimodal__openpi\u002FSKILL.md",true,[66,72,76,80,84],{"path":67,"resourceType":68,"resourceTypeDesc":69,"sizeBytes":70,"sourceHtmlUrl":71},"references\u002Fcheckpoints-and-env-map.md","REFERENCE","参考文档",2571,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--18-multimodal__openpi\u002Freferences\u002Fcheckpoints-and-env-map.md",{"path":73,"resourceType":68,"resourceTypeDesc":69,"sizeBytes":74,"sourceHtmlUrl":75},"references\u002Fconfig-recipes.md",1782,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--18-multimodal__openpi\u002Freferences\u002Fconfig-recipes.md",{"path":77,"resourceType":68,"resourceTypeDesc":69,"sizeBytes":78,"sourceHtmlUrl":79},"references\u002Fpytorch-gotchas.md",1446,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--18-multimodal__openpi\u002Freferences\u002Fpytorch-gotchas.md",{"path":81,"resourceType":68,"resourceTypeDesc":69,"sizeBytes":82,"sourceHtmlUrl":83},"references\u002Fremote-client-pattern.md",1205,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--18-multimodal__openpi\u002Freferences\u002Fremote-client-pattern.md",{"path":85,"resourceType":68,"resourceTypeDesc":69,"sizeBytes":86,"sourceHtmlUrl":87},"references\u002Ftraining-debugging.md",1207,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--18-multimodal__openpi\u002Freferences\u002Ftraining-debugging.md",[89,96,103,110,117,124],{"skillNo":90,"name":91,"nameZh":6,"description":92,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":93,"categories":94,"keywords":95,"publishedAt":19},"01M73JJB6EQPFPPVQK2Y9N67Q0","transformer-lens-interpretability","Provides guidance for mechanistic interpretability research using TransformerLens to inspect and manipulate transformer internals via HookPoints and activation caching. Use when reverse-engineering model algorithms, studying attention patterns, or performing activation patching experiments.","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Ftop100-runs\u002F130-transformer-lens\u002FSKILL.md",[],[],{"skillNo":97,"name":98,"nameZh":6,"description":99,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":100,"categories":101,"keywords":102,"publishedAt":19},"01M73JHTANRRWEE9HP98TPEHAY","wiki-vitepress","Packages generated wiki Markdown into a VitePress static site with dark theme, dark-mode Mermaid diagrams with click-to-zoom, and production build output. Use when the user wants to create a browsable website from generated wiki pages.","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Ftop100-runs\u002F126-wiki-vitepress\u002FSKILL.md",[],[],{"skillNo":104,"name":105,"nameZh":6,"description":106,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":107,"categories":108,"keywords":109,"publishedAt":19},"01M73JHJBMVFW46F1Y8K3SBF43","phoenix-observability","Open-source AI observability platform for LLM tracing, evaluation, and monitoring. Use when debugging LLM applications with detailed traces, running evaluations on datasets, or monitoring production AI systems with real-time insights.","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Ftop100-runs\u002F124-phoenix\u002FSKILL.md",[],[],{"skillNo":111,"name":112,"nameZh":6,"description":113,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":114,"categories":115,"keywords":116,"publishedAt":19},"01M73JHFPA3A2806PMP42ZN86Q","frontend-ui-dark-ts","Build dark-themed React applications using Tailwind CSS with custom theming, glassmorphism effects, and Framer Motion animations. Use when creating dashboards, admin panels, or data-rich interfaces with a refined dark aesthetic.","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Ftop100-runs\u002F123-frontend-ui-dark-ts\u002FSKILL.md",[],[],{"skillNo":118,"name":119,"nameZh":6,"description":120,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":121,"categories":122,"keywords":123,"publishedAt":19},"01M73JFVBVK57NRJS49TT2VDCT","tensorboard","Visualize training metrics, debug models with histograms, compare experiments, visualize model graphs, and profile performance with TensorBoard - Google's ML visualization toolkit","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Ftop100-runs\u002F113-tensorboard\u002FSKILL.md",[],[],{"skillNo":125,"name":126,"nameZh":6,"description":127,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":128,"licenseFileUrl":6,"sourceHtmlUrl":129,"categories":130,"keywords":131,"publishedAt":19},"01M73JFF6PK6ZCN41C89J7HS2B","markdown-mermaid-writing","Comprehensive markdown and Mermaid diagram writing skill. Use when creating any scientific document, report, analysis, or visualization. Establishes text-based diagrams as the default documentation standard with full style guides (markdown + mermaid), 24 diagram type references, and 9 document templates.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Ftop100-runs\u002F110-markdown-mermaid-writing\u002FSKILL.md",[],[]]