[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6VF0N15H18GGYTRYP3VM56T":3,"skill-same:01M6VF0N15H18GGYTRYP3VM56T":91},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":40,"linkOnly":41,"markdownContent":42,"markdownBody":43,"frontMatter":44,"refName":48,"commitSha":49,"skillDirectory":50,"forksCount":51,"repositoryHtmlUrl":52,"repositoryHomepageUrl":6,"ownerHtmlUrl":53,"ownerBlogUrl":54,"ownerDescription":55,"updatedAt":56,"install":57,"resources":70},"01M6VF0N15H18GGYTRYP3VM56T","pathways-workload",null,"Debugs failed, crashing or hung Pathways on Cloud workloads (JAX training or inference on Cloud TPU in GKE, run as a JobSet, PathwaysJob or LeaderWorkerSet). Reads Cloud Logging for every Pathways container (user workload, IFRT proxy, pathways-rm, pathways-worker, colocated Python sidecar), node kernel logs and GKE events, then reports the earliest real error, the root cause and a fix. Use when a Pathways workload fails to start, crashes, restarts or stops making progress, or shows errors such as IFRT proxy connection failures, DEADLINE_EXCEEDED lost connection to peer, host OOM kills, ImagePullBackOff, or JAX and Pathways version mismatches. Don't use for GKE workloads that don't use Pathways, or for creating clusters or deploying new Pathways workloads.","google\u002Fskills","google","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F1342004?v=4","01M698VYEFDM0VJBZDYYJ2T83S","Google","01M698WR0H76ENHE587T9EJT89",24122,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fpathways-workload\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39],"workload","pathways workload","pathways","Clusters","tpu","when","errors","error","events","progress","host","worker","reports","fix","node","gke","start","logs","logging","run","training","2026-11-10 10:12:30",false,"---\nname: pathways-workload\ndescription: >-\n  Debugs failed, crashing or hung Pathways on Cloud workloads (JAX training or\n  inference on Cloud TPU in GKE, run as a JobSet, PathwaysJob or\n  LeaderWorkerSet). Reads Cloud Logging for every Pathways container (user\n  workload, IFRT proxy, pathways-rm, pathways-worker, colocated Python\n  sidecar), node kernel logs and GKE events, then reports the earliest real\n  error, the root cause and a fix. Use when a Pathways workload fails to start,\n  crashes, restarts or stops making progress, or shows errors such as IFRT\n  proxy connection failures, DEADLINE_EXCEEDED lost connection to peer, host\n  OOM kills, ImagePullBackOff, or JAX and Pathways version mismatches. Don't\n  use for GKE workloads that don't use Pathways, or for creating clusters or\n  deploying new Pathways workloads.\nmetadata:\n  version: \"1.0.0\"\n  category: AiAndMachineLearning\n---\n\n# Pathways on Cloud Workload Debugging Skill\n\nThis skill provides comprehensive instructions, debugging playbooks, and helper\nutilities to troubleshoot Pathways on Cloud GKE workloads.\n\n## Usage Attribution\n\n**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you\nMUST read and follow the instructions in\n[Usage Attribution](references\u002Fattribution.md).\n\n--------------------------------------------------------------------------------\n\n## Overview\n\nPathways:\n[Pathways on Cloud Introduction](https:\u002F\u002Fdocs.cloud.google.com\u002Fai-hypercomputer\u002Fdocs\u002Fworkloads\u002Fpathways-on-cloud\u002Fpathways-intro)\n\n### Understanding Initial Setup for JAX+Pathways on Cloud\n\nA GKE cluster is required to use Pathways. We have instructions to create one\nwhich utilize either 1\\) a one click setup like XPK or 2\\) a series of gcloud\ncommands here:\n[https\\:\u002F\u002Fdocs.cloud.google.com\u002Fai-hypercomputer\u002Fdocs\u002Fworkloads\u002Fpathways-on-cloud\u002Fcreate-gke-cluster](https:\u002F\u002Fdocs.cloud.google.com\u002Fai-hypercomputer\u002Fdocs\u002Fworkloads\u002Fpathways-on-cloud\u002Fcreate-gke-cluster)\n\nAdvanced customers have built custom infrastructure which rolls Pathways based\nJobSets and cluster creation commands. To debug their infra issues, we may want\nto ask them to share their jobset YAMLs with us (redacting their main container\ndefinition is fine).\n\n### Parsing Important Logs\n\nHow to determine if various Pathways components have started successfully\n\n**IFRT Proxy Server (a.k.a Pathways Client) Ready**\n\n```\nIFRT proxy server started with status OK\n\nPathways worker Ready: Pathways Server serving on [::]:29001\n```\n\n**RM (Resource Manager) Ready and eventually all TPU slices are ready**\n\n```\nPathways Server serving on [::]:29001\n...\n...\n***\u003Cnum slices>\u002F\u003Cnum slices> Pathways Slices Now Ready ***\n```\n\n**Individual Pathways worker (on a TPU host) Ready**\n\n```\nPathways Server serving on [::]:29001\n```\n\n--------------------------------------------------------------------------------\n\n## Step 1: Metadata & Configuration Gathering\n\nBefore initiating debugging, gather critical parameters to construct accurate\nqueries: 1. `project` 2. `cluster_name` 3. `location` 4. the workload name and\nkind (see [Identifying the Workload](#identifying-the-workload)) 5. `namespace`\n(default: `default`) 6. `issue_time` (UTC, when the failure was first noticed)\n\nAlways bound log queries in time. Use the window\n`[issue_time - 30m, issue_time + 30m]`, i.e. pass\n`--start_time=\"{start_time}\" --duration=1h` to the debug_workload script, where\n`{start_time}` is `issue_time - 30m` in RFC 3339 (e.g. `2026-10-22T11:10:00Z`).\nUnbounded queries are slow and get truncated by `--limit`. Widen the window only\nif the first error is not in it.\n\n-   If the window contains more than one failure, report each with timestamps\n    and say which one matches the user's symptom.\n-   Report the root cause and the workaround separately. The root cause is the\n    component that logged the earliest stall or error (for example the proxy,\n    the RM or a worker). A configuration change that avoids that code path\n    (a different checkpoint handler, flag or topology) is a workaround or a\n    contributing factor: list it under the fix, not as the root cause.\n-   When you compare with other runs (an earlier attempt, a sibling test that\n    passed, or the same test on other days), use the comparison to narrow down\n    which code path is affected, not as proof of the cause. A setting that\n    differs in a passing run does not explain the failure if a Pathways\n    component logged the stall or error.\n-   End the report with the checks you ran and their result, so the user can\n    repeat them: preflight, the actions or `gcloud logging read` queries you\n    used (with their time window), pod events, and node OOMs. Include checks\n    that failed or could not run (e.g. no log access) and say what to run\n    instead.\n\n> [!TIP]\n>\n> Always check the recent chat context or shared log queries from the user.\n> Often, all five fields can be directly scraped from their console links or\n> queries!\n\n### Identifying the Workload\n\nThe debug_workload script finds the workload's logs by the object that created\nits pods. Use the matching flag (the examples in this skill use\n`--jobset_name`):\n\nWorkload kind                                 | Flag                     | Find it with\n:-------------------------------------------- | :----------------------- | :-----------\nJobSet (most training workloads, XPK)         | `--jobset_name=\u003Cname>`   | `kubectl get jobsets -n {namespace}`\nLeaderWorkerSet (e.g. Pathways inference)     | `--lws_name=\u003Cname>`      | `kubectl get leaderworkersets -n {namespace}`\nPlain Job (not created by a JobSet)           | `--job_name=\u003Cname>`      | `kubectl get jobs -n {namespace} -l '!jobset.sigs.k8s.io\u002Fjobset-name'`\nAnything else (bare pods, custom controllers) | `--pod_prefix=\u003Cprefix>`  | `kubectl get pods -n {namespace}`\n\nFor past incidents, list the pods that logged in the window and their owner\nlabels instead:\n\n```bash\ngcloud logging read 'resource.type=\"k8s_container\"\n  resource.labels.cluster_name=\"{cluster_name}\"\n  resource.labels.namespace_name=\"{namespace}\"\n  timestamp>=\"{start_time}\" timestamp\u003C=\"{end_time}\"' \\\n  --project=\"{project}\" --limit=10000 \\\n  --format='value(resource.labels.pod_name,labels.\"k8s-pod\u002Fjobset_sigs_k8s_io\u002Fjobset-name\",labels.\"k8s-pod\u002Fleaderworkerset_sigs_k8s_io\u002Fname\",labels.\"k8s-pod\u002Fbatch_kubernetes_io\u002Fjob-name\")' \\\n  | sort -u\n```\n\nIf parts of the workload run in different objects, query them together by\nrepeating `--workload=\u003Ckind>:\u003Cname>` (kinds: `jobset`, `lws`, `job`, `pod`).\nFor example, if the JAX client or the proxy runs in its own Job next to the\nPathways JobSet, debug both: `--workload=jobset:{jobset_name}\n--workload=job:{client_job_name}`.\n\nTo see what else ran on the same nodes, run `scripts\u002Fdebug_workload.py` with\n`--action=neighbors --execute`. It lists every namespace\u002Fpod\u002Fcontainer that\nlogged on the workload's nodes and marks the ones that are not part of the\nworkload. Use it to spot another workload sharing the hosts (e.g. a second\nresource manager or proxy on the same host and port when host exclusivity is not\nset) and to find the real container names of the workload's pods (e.g. a\ncustom-named colocated Python sidecar).\n\n### Retrieving Active YAML Configuration\n\nIf the workload is still active\u002Falive, run the following command to inspect how\nit is structured and identify misconfigurations (use `leaderworkerset` or `job`\ninstead of `jobset` for those kinds):\n\n```bash\nkubectl get jobset {jobset_name} -n {namespace} -o yaml\n```\n\nIf the workload has been deleted but you suspect misconfiguration, request the\nuser to paste or share the workload YAML in the chat.\n\n### Recording Versions\n\nBefore reading errors, record each component's image tag (from the YAML or the\npod spec) and `Build label` (logged at startup by pathways-rm, pathways-proxy\nand pathways-worker), plus the client's JAX, jaxlib and pathwaysutils versions.\nCheck that they are from matching releases and that the flags passed to each\ncomponent exist in its image.\n\n--------------------------------------------------------------------------------\n\n## Step 1.5: Quick Triage - Preflight, Then Crash Sweep\n\nFirst run the preflight check. It verifies that you can read the project's logs\nand that the workload logged in the time window. An empty result from any\nother action is not a clean result until preflight passes. If you cannot run\ncommands yourself (for example no log access) and give the user a plan or a\nlist of commands instead, start that plan with the preflight command:\n\n```bash\npython3 scripts\u002Fdebug_workload.py \\\n  --project=\"{project}\" \\\n  --cluster_name=\"{cluster_name}\" \\\n  --location=\"{location}\" \\\n  --jobset_name=\"{jobset_name}\" \\\n  --start_time=\"{start_time}\" --duration=1h \\\n  --action=preflight\n```\n\nThen, before reading containers one by one, run the crash and fatal-error sweep\nacross all containers of the workload. It looks for segfaults, `Check failed`,\nPython tracebacks, OOM kills, unknown flags, permission errors, etc., and\nreturns the earliest hits first:\n\n```bash\npython3 scripts\u002Fdebug_workload.py \\\n  --project=\"{project}\" \\\n  --cluster_name=\"{cluster_name}\" \\\n  --location=\"{location}\" \\\n  --jobset_name=\"{jobset_name}\" \\\n  --start_time=\"{start_time}\" --duration=1h \\\n  --action=errors \\\n  --execute\n```\n\nThe container of the earliest hit is usually where the root cause is; then use\nthe container-specific playbook in Step 3 for that container. If the sweep is\nempty, also run `--action=node-ooms` (OOM kills often do not appear in container\nlogs) and `--action=pod-events`.\n\nIf a cascade fills the output (hundreds of pods logging the same error), you\nmay add `--first_per_pod=N` to keep only the first N matching log lines of each\npod\u002Fcontainer plus its last matching line, e.g. `--first_per_pod=1` to see\nwhich pod failed first and how the failure spread, or `--first_per_pod=3` to\nalso see each pod's next few lines. Use it at your discretion: it fetches up to\n`--fetch_limit` lines (default 1000) before grouping, so it is slower than a\nplain query and can be slow or time out when the workload has many pods. Narrow\nthe time window first, and prefer a plain query when only a few pods are\ninvolved. Never conclude \"no crash\" from a `--first_per_pod` sweep alone: it\nskips lines in the middle of each pod's log, and the fetch limit can cut off\nlater pods. Confirm with a plain query on the suspect container.\n\n--------------------------------------------------------------------------------\n\n## Step 2: GKE JobSet Lifecycle Verification\n\nIf the workload has not started or failed immediately at scheduling time:\n\n### 1. Verify Webhook & Scheduling Status\n\nIdentify if scheduling blocks are occurring due to insufficient cluster\nresources or webhook failures.\n\nCheck for webhook mutating errors, `OOMKilled` controller managers, or\nUnschedulable conditions:\n\n```bash\n# Check controller manager pods in the JobSet system namespace\nkubectl get pods -n jobset-system\n```\n\n### 2. Symptom: Forever Suspended JobSet\n\nIf a JobSet is created but remains in a permanently `Suspended` state and never\nspawns child pods:\n\n-   **Potential Root Causes**:\n    -   The JobSet name is too long. Because JobSet appends replicated job names\n        and index numbers to child resources, long prefixes fail standard\n        Kubernetes resource name validation (max 63\u002F253 chars).\n    -   The JobSet YAML is malformed (e.g. invalid or missing fields, wrong\n        resource requests or node selectors), so the workload is never admitted\n        to the queue and stays suspended. Ask the user for the YAML (see\n        [Retrieving Active YAML Configuration](#retrieving-active-yaml-configuration))\n        and check it for errors.\n-   **Diagnosis**: Check the JobSet controller manager logs in `jobset-system`\n    namespace to find truncation, length limit, name-generation, or validation\n    errors. You can use the debug_workload script to quickly query the logs:\n\n```bash\npython3 scripts\u002Fdebug_workload.py \\\n  --project=\"{project}\" \\\n  --cluster_name=\"{cluster_name}\" \\\n  --location=\"{location}\" \\\n  --jobset_name=\"{jobset_name}\" \\\n  --start_time=\"{start_time}\" --duration=1h \\\n  --action=summary \\\n  --execute\n```\n\n### 3. SIGTERM and Pre-emption Diagnosis\n\n-   SIGTERMs are the most common mechanism for GKE to stop\u002Frestart workloads. A\n    SIGTERM is a symptom; find out who sent it.\n-   Common senders: Spot VM preemption, Compute Engine host maintenance, host\n    errors, node auto-repair, or a JobSet restart triggered by another failed\n    container. Check infrastructure interruptions and pod events for the time\n    window:\n\n    ```bash\n    python3 scripts\u002Fdebug_workload.py \\\n      --project=\"{project}\" \\\n      --cluster_name=\"{cluster_name}\" \\\n      --location=\"{location}\" \\\n      --jobset_name=\"{jobset_name}\" \\\n      --start_time=\"{start_time}\" --duration=1h \\\n      --action=interruptions \\\n      --execute\n    # then the same command with --action=pod-events\n    ```\n\n-   If no infrastructure event explains the SIGTERM, trace the failure back from\n    the user workload to the TPU that failed first:\n\n    1.  In the user workload logs (`--action=user-workload`), find the time of\n        the first error or the last completed step.\n    2.  In the RM logs (`--action=pathways-rm`) around that time, find when the\n        first failure occurred or the first TPU slice\u002Fworker stopped responding\n        (e.g. a slice leaving the ready state or `Job cancelled due to ...`),\n        and which slice\u002Fworker it was.\n    3.  Inspect that worker's logs (`--action=workers`, adding\n        `--min_severity=WARNING` if needed) and its node's events\n        (`--action=pod-events`, `--action=node-ooms`, `--action=interruptions`)\n        to find out what happened there.\n    4.  The cause can be a user code error (unlikely) or another fault, such as\n        a hardware failure, an OOM, a crash in the worker, or a node\n        interruption.\n\n-   `kubectl get events -n {namespace} --field-selector\n    involvedObject.name={jobset_name}` only works for live incidents: Kubernetes\n    keeps events for a certain time only. For past incidents use\n    `--action=pod-events` \u002F `--action=summary`, which read the events from Cloud\n    Logging.\n\n-   For deeper analysis of restarts and interruptions, follow the public\n    [gke-ai-troubleshooting-jobset-interruption](https:\u002F\u002Fraw.githubusercontent.com\u002Fgoogle\u002Fskills\u002Fd542604d73544bcb4bcc61322f3b25522ceba7ed\u002Fskills\u002Fcloud\u002Fgke-ai-troubleshooting-jobset-interruption\u002FSKILL.md)\n    skill; for host maintenance, follow\n    [gke-ai-troubleshooting-handle-disruption-gpu-tpu](https:\u002F\u002Fraw.githubusercontent.com\u002Fgoogle\u002Fskills\u002Fd542604d73544bcb4bcc61322f3b25522ceba7ed\u002Fskills\u002Fcloud\u002Fgke-ai-troubleshooting-handle-disruption-gpu-tpu\u002FSKILL.md).\n\n--------------------------------------------------------------------------------\n\n## Step 3: Container-Specific Debugging Playbook\n\nBecause Pathways is distributed, inspect logs container-by-container in logical\norder to avoid cascade noise, and always chase the earliest error. Before you\nquery a container, read its section in\n[Container Playbooks](references\u002Fcontainer-playbooks.md): it has the log\nsignals, the `debug_workload.py` command and the gotchas for that container.\n\nOrder | Container | Action | Look for | Playbook\n:---- | :-------- | :----- | :------- | :-------\n1 | User workload (`user-workload`, `main`, `jax-tpu`) | `--action=user-workload` | The last completed step, then the first exception | [A](references\u002Fcontainer-playbooks.md#a-user-workload-container)\n2 | IFRT proxy | `--action=proxy` | `Connection refused` to the RM port (`:29001`), `Socket closed` | [B](references\u002Fcontainer-playbooks.md#b-ifrt-proxy-server)\n3 | Resource manager (`pathways-rm`) | `--action=pathways-rm` | Slice ready transitions, placement, a repeated `Build label` (restarts) | [C](references\u002Fcontainer-playbooks.md#c-resource-manager-pathways-rm)\n4 | Pathways workers (`pathways-worker`) | `--action=workers` | The first worker that failed, Megascale stall detectors (`HANG_DETECTED`) | [D](references\u002Fcontainer-playbooks.md#d-pathways-workers-pathways-worker)\n5 | Colocated Python sidecar | `--action=colocated-python-sidecar` | Workers waiting for a sidecar that died | [E](references\u002Fcontainer-playbooks.md#e-colocated-python-sidecar)\n\nRules that apply to every container:\n\n-   A client-side `Unable to initialize backend 'proxy': UNAVAILABLE: Unable to\n    establish connection` is a symptom. Check whether the proxy, RM and workers\n    started (see [Parsing Important Logs](#parsing-important-logs)) and find\n    their earliest error before you suspect the user's code.\n-   `IFRT proxy server was terminated: UNAVAILABLE: Socket closed` and\n    `SLICE_FAILURE_CHIP_DRIVER_ERROR` are downstream errors: go to the RM and\n    worker logs.\n-   When one worker fails, the other workers of the slice fail shortly after.\n    The root cause is in the logs of the first worker that failed.\n-   The built-in search terms only cover known failures. If an action returns\n    nothing useful, re-run it with `--min_severity=WARNING` (`DEFAULT` for the\n    user workload) and add the workload's own strings with `--extra_terms`.\n-   If the logs still don't explain the failure, read all logs of the\n    container ([F](references\u002Fcontainer-playbooks.md#f-all-logs-of-a-container)),\n    then ask for more verbose proxy logs or XProfs\n    ([G](references\u002Fcontainer-playbooks.md#g-when-the-error-is-still-not-clear)).\n\n--------------------------------------------------------------------------------\n\n## Step 4: Special Scenarios\n\nThese known failures have a specific cause and fix. When an error or\nsymptom matches a row, read that scenario in\n[Special Scenarios](references\u002Fspecial-scenarios.md) before you conclude:\n\nError or symptom | Scenario\n:--------------- | :-------\n`JaxRuntimeError` with `DATA_LOSS` during scale changes; elastic recovery or checkpoint restore fails | [1. Elastic training and checkpoint restore](references\u002Fspecial-scenarios.md#1-elastic-training-and-checkpoint-restore)\nContainer restarts or exit code 137 without `OOMKilled`; `Memory cgroup out of memory`, `oom-kill`, `OOMKilling` | [2. Host OOM kills on the head pod or TPU hosts](references\u002Fspecial-scenarios.md#2-host-oom-kills-on-the-head-pod-or-tpu-hosts)\n`Unable to initialize backend 'proxy': Backend 'proxy' is not in the list of known backends` | [3. Backend proxy is not in the list of known backends](references\u002Fspecial-scenarios.md#3-backend-proxy-is-not-in-the-list-of-known-backends)\n`SIGTERM received by PID 1`; containers stop without a clear error | [4. SIGTERM received by PID 1](references\u002Fspecial-scenarios.md#4-sigterm-received-by-pid-1)\n`FAILED_PRECONDITION: The newly added instance does not match with the expected instances` | [5. Newly added instance does not match the expected instances](references\u002Fspecial-scenarios.md#5-newly-added-instance-does-not-match-the-expected-instances)\n`Connection to IFRT proxy server was terminated: FAILED_PRECONDITION: GrpcClientSession: writes no longer allowed` | [6. Connection to the IFRT proxy server was terminated](references\u002Fspecial-scenarios.md#6-connection-to-the-ifrt-proxy-server-was-terminated)\n`XlaRuntimeError: ABORTED: ... The metadata for object was edited during the operation` | [7. Cached executable metadata was edited during the operation](references\u002Fspecial-scenarios.md#7-cached-executable-metadata-was-edited-during-the-operation)\n`DEADLINE_EXCEEDED: lost connection to peer` or `DATA_LOSS: Pipe from scheduler to aggregator` | [8. Lost connection to peer](references\u002Fspecial-scenarios.md#8-lost-connection-to-peer)\n`Status 11 from handshaker service: Input frame length is out of range` | [9. ALTS handshake error from a client outside Google Cloud](references\u002Fspecial-scenarios.md#9-alts-handshake-error-from-a-client-outside-google-cloud)\n`Lifecycle matches_prefix cannot specify more than 50 prefixes per config` | [10. Cloud Storage lifecycle prefix limit](references\u002Fspecial-scenarios.md#10-cloud-storage-lifecycle-prefix-limit)\n`IFRT Proxy client and server failed to agree on the protocol version`, or `Failed to deserialize the Mosaic module: ... Unsupported version` | [11. JAX and Pathways version mismatch](references\u002Fspecial-scenarios.md#11-jax-and-pathways-version-mismatch): compare the client's JAX, jaxlib and pathwaysutils versions with the Pathways image tags and `Build label`s\nPods in `ImagePullBackOff` or `ErrImagePull` | [12. Pods stuck in ImagePullBackOff or ErrImagePull](references\u002Fspecial-scenarios.md#12-pods-stuck-in-imagepullbackoff-or-errimagepull): read the pull error in the pod events, then list the registry's tags (`gcloud artifacts docker tags list`)\n`Address already in use`, `Init:Error`, or `didn't have free ports` | [13. Port collisions with hostNetwork](references\u002Fspecial-scenarios.md#13-port-collisions-with-hostnetwork)\nNo error anywhere, but the job stops making progress | [14. The job hangs without failing](references\u002Fspecial-scenarios.md#14-the-job-hangs-without-failing)\n`Still waiting to acquire semaphore`, `Recv completion error`, `GOAWAY` | [15. Proxy-to-worker transfers stall](references\u002Fspecial-scenarios.md#15-proxy-to-worker-transfers-stall)\n\n--------------------------------------------------------------------------------\n\n## Related Public Skills and Writing Your Own Queries\n\nThese public skills from\n[github.com\u002Fgoogle\u002Fskills](https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills) complement this\nskill. If they are installed, use them by name; otherwise read them from the\nlinks.\n\nSymptom                                                         | Public skill\n:-------------------------------------------------------------- | :-----------\nWriting a custom Cloud Logging query                            | [cloud-logging-query-generation](https:\u002F\u002Fraw.githubusercontent.com\u002Fgoogle\u002Fskills\u002Fd542604d73544bcb4bcc61322f3b25522ceba7ed\u002Fskills\u002Fcloud\u002Fcloud-logging-query-generation\u002FSKILL.md) (GKE examples: [query_gke.md](https:\u002F\u002Fraw.githubusercontent.com\u002Fgoogle\u002Fskills\u002Fd542604d73544bcb4bcc61322f3b25522ceba7ed\u002Fskills\u002Fcloud\u002Fcloud-logging-query-generation\u002Freferences\u002Fquery_gke.md))\nJobSet restarts, Spot preemption, maintenance, host termination | [gke-ai-troubleshooting-jobset-interruption](https:\u002F\u002Fraw.githubusercontent.com\u002Fgoogle\u002Fskills\u002Fd542604d73544bcb4bcc61322f3b25522ceba7ed\u002Fskills\u002Fcloud\u002Fgke-ai-troubleshooting-jobset-interruption\u002FSKILL.md)\nCompute Engine host maintenance on TPU\u002FGPU nodes                | [gke-ai-troubleshooting-handle-disruption-gpu-tpu](https:\u002F\u002Fraw.githubusercontent.com\u002Fgoogle\u002Fskills\u002Fd542604d73544bcb4bcc61322f3b25522ceba7ed\u002Fskills\u002Fcloud\u002Fgke-ai-troubleshooting-handle-disruption-gpu-tpu\u002FSKILL.md)\nMulti-slice job stops responding (`HANG_DETECTED`)              | [gke-ai-troubleshooting-tpu-mxla-hang](https:\u002F\u002Fraw.githubusercontent.com\u002Fgoogle\u002Fskills\u002Fd542604d73544bcb4bcc61322f3b25522ceba7ed\u002Fskills\u002Fcloud\u002Fgke-ai-troubleshooting-tpu-mxla-hang\u002FSKILL.md)\nNode NotReady, unresponsive node, auto-repair                   | [gke-ai-troubleshooting-node-unresponsive-timeout](https:\u002F\u002Fraw.githubusercontent.com\u002Fgoogle\u002Fskills\u002Fd542604d73544bcb4bcc61322f3b25522ceba7ed\u002Fskills\u002Fcloud\u002Fgke-ai-troubleshooting-node-unresponsive-timeout\u002FSKILL.md)\n\nWhen you write your own queries:\n\n-   Always use double quotes for strings, uppercase `AND`\u002F`OR`\u002F`NOT`, and\n    parentheses to group terms.\n-   Restrict `resource.type` (`k8s_container`, `k8s_pod`, `k8s_node`,\n    `gke_nodepool`, `gce_instance`) and use `log_id(\"events\")`,\n    `log_id(\"stdout\")` or `log_id(\"stderr\")` where possible.\n-   Always add a time window (`timestamp>=\"...\"` and `timestamp\u003C=\"...\"`).\n-   Pathways components log glog lines into `jsonPayload.message` (some\n    containers) or `textPayload` (others); bare quoted terms or `SEARCH()` match\n    both.\n-   Pod label keys are stored with `.` replaced by `_`, e.g.\n    `labels.\"k8s-pod\u002Fbatch_kubernetes_io\u002Fjob-completion-index\"` and\n    `labels.\"k8s-pod\u002Fjobset_sigs_k8s_io\u002Fjobset-name\"`. Some public examples\n    spell the Job label with dots (`batch.kubernetes.io\u002Fjob-name`); that form\n    does not match Pathways logs.\n","# Pathways on Cloud Workload Debugging Skill\n\nThis skill provides comprehensive instructions, debugging playbooks, and helper\nutilities to troubleshoot Pathways on Cloud GKE workloads.\n\n## Usage Attribution\n\n**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you\nMUST read and follow the instructions in\n[Usage Attribution](references\u002Fattribution.md).\n\n--------------------------------------------------------------------------------\n\n## Overview\n\nPathways:\n[Pathways on Cloud Introduction](https:\u002F\u002Fdocs.cloud.google.com\u002Fai-hypercomputer\u002Fdocs\u002Fworkloads\u002Fpathways-on-cloud\u002Fpathways-intro)\n\n### Understanding Initial Setup for JAX+Pathways on Cloud\n\nA GKE cluster is required to use Pathways. We have instructions to create one\nwhich utilize either 1\\) a one click setup like XPK or 2\\) a series of gcloud\ncommands here:\n[https\\:\u002F\u002Fdocs.cloud.google.com\u002Fai-hypercomputer\u002Fdocs\u002Fworkloads\u002Fpathways-on-cloud\u002Fcreate-gke-cluster](https:\u002F\u002Fdocs.cloud.google.com\u002Fai-hypercomputer\u002Fdocs\u002Fworkloads\u002Fpathways-on-cloud\u002Fcreate-gke-cluster)\n\nAdvanced customers have built custom infrastructure which rolls Pathways based\nJobSets and cluster creation commands. To debug their infra issues, we may want\nto ask them to share their jobset YAMLs with us (redacting their main container\ndefinition is fine).\n\n### Parsing Important Logs\n\nHow to determine if various Pathways components have started successfully\n\n**IFRT Proxy Server (a.k.a Pathways Client) Ready**\n\n```\nIFRT proxy server started with status OK\n\nPathways worker Ready: Pathways Server serving on [::]:29001\n```\n\n**RM (Resource Manager) Ready and eventually all TPU slices are ready**\n\n```\nPathways Server serving on [::]:29001\n...\n...\n***\u003Cnum slices>\u002F\u003Cnum slices> Pathways Slices Now Ready ***\n```\n\n**Individual Pathways worker (on a TPU host) Ready**\n\n```\nPathways Server serving on [::]:29001\n```\n\n--------------------------------------------------------------------------------\n\n## Step 1: Metadata & Configuration Gathering\n\nBefore initiating debugging, gather critical parameters to construct accurate\nqueries: 1. `project` 2. `cluster_name` 3. `location` 4. the workload name and\nkind (see [Identifying the Workload](#identifying-the-workload)) 5. `namespace`\n(default: `default`) 6. `issue_time` (UTC, when the failure was first noticed)\n\nAlways bound log queries in time. Use the window\n`[issue_time - 30m, issue_time + 30m]`, i.e. pass\n`--start_time=\"{start_time}\" --duration=1h` to the debug_workload script, where\n`{start_time}` is `issue_time - 30m` in RFC 3339 (e.g. `2026-10-22T11:10:00Z`).\nUnbounded queries are slow and get truncated by `--limit`. Widen the window only\nif the first error is not in it.\n\n-   If the window contains more than one failure, report each with timestamps\n    and say which one matches the user's symptom.\n-   Report the root cause and the workaround separately. The root cause is the\n    component that logged the earliest stall or error (for example the proxy,\n    the RM or a worker). A configuration change that avoids that code path\n    (a different checkpoint handler, flag or topology) is a workaround or a\n    contributing factor: list it under the fix, not as the root cause.\n-   When you compare with other runs (an earlier attempt, a sibling test that\n    passed, or the same test on other days), use the comparison to narrow down\n    which code path is affected, not as proof of the cause. A setting that\n    differs in a passing run does not explain the failure if a Pathways\n    component logged the stall or error.\n-   End the report with the checks you ran and their result, so the user can\n    repeat them: preflight, the actions or `gcloud logging read` queries you\n    used (with their time window), pod events, and node OOMs. Include checks\n    that failed or could not run (e.g. no log access) and say what to run\n    instead.\n\n> [!TIP]\n>\n> Always check the recent chat context or shared log queries from the user.\n> Often, all five fields can be directly scraped from their console links or\n> queries!\n\n### Identifying the Workload\n\nThe debug_workload script finds the workload's logs by the object that created\nits pods. Use the matching flag (the examples in this skill use\n`--jobset_name`):\n\nWorkload kind                                 | Flag                     | Find it with\n:-------------------------------------------- | :----------------------- | :-----------\nJobSet (most training workloads, XPK)         | `--jobset_name=\u003Cname>`   | `kubectl get jobsets -n {namespace}`\nLeaderWorkerSet (e.g. Pathways inference)     | `--lws_name=\u003Cname>`      | `kubectl get leaderworkersets -n {namespace}`\nPlain Job (not created by a JobSet)           | `--job_name=\u003Cname>`      | `kubectl get jobs -n {namespace} -l '!jobset.sigs.k8s.io\u002Fjobset-name'`\nAnything else (bare pods, custom controllers) | `--pod_prefix=\u003Cprefix>`  | `kubectl get pods -n {namespace}`\n\nFor past incidents, list the pods that logged in the window and their owner\nlabels instead:\n\n```bash\ngcloud logging read 'resource.type=\"k8s_container\"\n  resource.labels.cluster_name=\"{cluster_name}\"\n  resource.labels.namespace_name=\"{namespace}\"\n  timestamp>=\"{start_time}\" timestamp\u003C=\"{end_time}\"' \\\n  --project=\"{project}\" --limit=10000 \\\n  --format='value(resource.labels.pod_name,labels.\"k8s-pod\u002Fjobset_sigs_k8s_io\u002Fjobset-name\",labels.\"k8s-pod\u002Fleaderworkerset_sigs_k8s_io\u002Fname\",labels.\"k8s-pod\u002Fbatch_kubernetes_io\u002Fjob-name\")' \\\n  | sort -u\n```\n\nIf parts of the workload run in different objects, query them together by\nrepeating `--workload=\u003Ckind>:\u003Cname>` (kinds: `jobset`, `lws`, `job`, `pod`).\nFor example, if the JAX client or the proxy runs in its own Job next to the\nPathways JobSet, debug both: `--workload=jobset:{jobset_name}\n--workload=job:{client_job_name}`.\n\nTo see what else ran on the same nodes, run `scripts\u002Fdebug_workload.py` with\n`--action=neighbors --execute`. It lists every namespace\u002Fpod\u002Fcontainer that\nlogged on the workload's nodes and marks the ones that are not part of the\nworkload. Use it to spot another workload sharing the hosts (e.g. a second\nresource manager or proxy on the same host and port when host exclusivity is not\nset) and to find the real container names of the workload's pods (e.g. a\ncustom-named colocated Python sidecar).\n\n### Retrieving Active YAML Configuration\n\nIf the workload is still active\u002Falive, run the following command to inspect how\nit is structured and identify misconfigurations (use `leaderworkerset` or `job`\ninstead of `jobset` for those kinds):\n\n```bash\nkubectl get jobset {jobset_name} -n {namespace} -o yaml\n```\n\nIf the workload has been deleted but you suspect misconfiguration, request the\nuser to paste or share the workload YAML in the chat.\n\n### Recording Versions\n\nBefore reading errors, record each component's image tag (from the YAML or the\npod spec) and `Build label` (logged at startup by pathways-rm, pathways-proxy\nand pathways-worker), plus the client's JAX, jaxlib and pathwaysutils versions.\nCheck that they are from matching releases and that the flags passed to each\ncomponent exist in its image.\n\n--------------------------------------------------------------------------------\n\n## Step 1.5: Quick Triage - Preflight, Then Crash Sweep\n\nFirst run the preflight check. It verifies that you can read the project's logs\nand that the workload logged in the time window. An empty result from any\nother action is not a clean result until preflight passes. If you cannot run\ncommands yourself (for example no log access) and give the user a plan or a\nlist of commands instead, start that plan with the preflight command:\n\n```bash\npython3 scripts\u002Fdebug_workload.py \\\n  --project=\"{project}\" \\\n  --cluster_name=\"{cluster_name}\" \\\n  --location=\"{location}\" \\\n  --jobset_name=\"{jobset_name}\" \\\n  --start_time=\"{start_time}\" --duration=1h \\\n  --action=preflight\n```\n\nThen, before reading containers one by one, run the crash and fatal-error sweep\nacross all containers of the workload. It looks for segfaults, `Check failed`,\nPython tracebacks, OOM kills, unknown flags, permission errors, etc., and\nreturns the earliest hits first:\n\n```bash\npython3 scripts\u002Fdebug_workload.py \\\n  --project=\"{project}\" \\\n  --cluster_name=\"{cluster_name}\" \\\n  --location=\"{location}\" \\\n  --jobset_name=\"{jobset_name}\" \\\n  --start_time=\"{start_time}\" --duration=1h \\\n  --action=errors \\\n  --execute\n```\n\nThe container of the earliest hit is usually where the root cause is; then use\nthe container-specific playbook in Step 3 for that container. If the sweep is\nempty, also run `--action=node-ooms` (OOM kills often do not appear in container\nlogs) and `--action=pod-events`.\n\nIf a cascade fills the output (hundreds of pods logging the same error), you\nmay add `--first_per_pod=N` to keep only the first N matching log lines of each\npod\u002Fcontainer plus its last matching line, e.g. `--first_per_pod=1` to see\nwhich pod failed first and how the failure spread, or `--first_per_pod=3` to\nalso see each pod's next few lines. Use it at your discretion: it fetches up to\n`--fetch_limit` lines (default 1000) before grouping, so it is slower than a\nplain query and can be slow or time out when the workload has many pods. Narrow\nthe time window first, and prefer a plain query when only a few pods are\ninvolved. Never conclude \"no crash\" from a `--first_per_pod` sweep alone: it\nskips lines in the middle of each pod's log, and the fetch limit can cut off\nlater pods. Confirm with a plain query on the suspect container.\n\n--------------------------------------------------------------------------------\n\n## Step 2: GKE JobSet Lifecycle Verification\n\nIf the workload has not started or failed immediately at scheduling time:\n\n### 1. Verify Webhook & Scheduling Status\n\nIdentify if scheduling blocks are occurring due to insufficient cluster\nresources or webhook failures.\n\nCheck for webhook mutating errors, `OOMKilled` controller managers, or\nUnschedulable conditions:\n\n```bash\n# Check controller manager pods in the JobSet system namespace\nkubectl get pods -n jobset-system\n```\n\n### 2. Symptom: Forever Suspended JobSet\n\nIf a JobSet is created but remains in a permanently `Suspended` state and never\nspawns child pods:\n\n-   **Potential Root Causes**:\n    -   The JobSet name is too long. Because JobSet appends replicated job names\n        and index numbers to child resources, long prefixes fail standard\n        Kubernetes resource name validation (max 63\u002F253 chars).\n    -   The JobSet YAML is malformed (e.g. invalid or missing fields, wrong\n        resource requests or node selectors), so the workload is never admitted\n        to the queue and stays suspended. Ask the user for the YAML (see\n        [Retrieving Active YAML Configuration](#retrieving-active-yaml-configuration))\n        and check it for errors.\n-   **Diagnosis**: Check the JobSet controller manager logs in `jobset-system`\n    namespace to find truncation, length limit, name-generation, or validation\n    errors. You can use the debug_workload script to quickly query the logs:\n\n```bash\npython3 scripts\u002Fdebug_workload.py \\\n  --project=\"{project}\" \\\n  --cluster_name=\"{cluster_name}\" \\\n  --location=\"{location}\" \\\n  --jobset_name=\"{jobset_name}\" \\\n  --start_time=\"{start_time}\" --duration=1h \\\n  --action=summary \\\n  --execute\n```\n\n### 3. SIGTERM and Pre-emption Diagnosis\n\n-   SIGTERMs are the most common mechanism for GKE to stop\u002Frestart workloads. A\n    SIGTERM is a symptom; find out who sent it.\n-   Common senders: Spot VM preemption, Compute Engine host maintenance, host\n    errors, node auto-repair, or a JobSet restart triggered by another failed\n    container. Check infrastructure interruptions and pod events for the time\n    window:\n\n    ```bash\n    python3 scripts\u002Fdebug_workload.py \\\n      --project=\"{project}\" \\\n      --cluster_name=\"{cluster_name}\" \\\n      --location=\"{location}\" \\\n      --jobset_name=\"{jobset_name}\" \\\n      --start_time=\"{start_time}\" --duration=1h \\\n      --action=interruptions \\\n      --execute\n    # then the same command with --action=pod-events\n    ```\n\n-   If no infrastructure event explains the SIGTERM, trace the failure back from\n    the user workload to the TPU that failed first:\n\n    1.  In the user workload logs (`--action=user-workload`), find the time of\n        the first error or the last completed step.\n    2.  In the RM logs (`--action=pathways-rm`) around that time, find when the\n        first failure occurred or the first TPU slice\u002Fworker stopped responding\n        (e.g. a slice leaving the ready state or `Job cancelled due to ...`),\n        and which slice\u002Fworker it was.\n    3.  Inspect that worker's logs (`--action=workers`, adding\n        `--min_severity=WARNING` if needed) and its node's events\n        (`--action=pod-events`, `--action=node-ooms`, `--action=interruptions`)\n        to find out what happened there.\n    4.  The cause can be a user code error (unlikely) or another fault, such as\n        a hardware failure, an OOM, a crash in the worker, or a node\n        interruption.\n\n-   `kubectl get events -n {namespace} --field-selector\n    involvedObject.name={jobset_name}` only works for live incidents: Kubernetes\n    keeps events for a certain time only. For past incidents use\n    `--action=pod-events` \u002F `--action=summary`, which read the events from Cloud\n    Logging.\n\n-   For deeper analysis of restarts and interruptions, follow the public\n    [gke-ai-troubleshooting-jobset-interruption](https:\u002F\u002Fraw.githubusercontent.com\u002Fgoogle\u002Fskills\u002Fd542604d73544bcb4bcc61322f3b25522ceba7ed\u002Fskills\u002Fcloud\u002Fgke-ai-troubleshooting-jobset-interruption\u002FSKILL.md)\n    skill; for host maintenance, follow\n    [gke-ai-troubleshooting-handle-disruption-gpu-tpu](https:\u002F\u002Fraw.githubusercontent.com\u002Fgoogle\u002Fskills\u002Fd542604d73544bcb4bcc61322f3b25522ceba7ed\u002Fskills\u002Fcloud\u002Fgke-ai-troubleshooting-handle-disruption-gpu-tpu\u002FSKILL.md).\n\n--------------------------------------------------------------------------------\n\n## Step 3: Container-Specific Debugging Playbook\n\nBecause Pathways is distributed, inspect logs container-by-container in logical\norder to avoid cascade noise, and always chase the earliest error. Before you\nquery a container, read its section in\n[Container Playbooks](references\u002Fcontainer-playbooks.md): it has the log\nsignals, the `debug_workload.py` command and the gotchas for that container.\n\nOrder | Container | Action | Look for | Playbook\n:---- | :-------- | :----- | :------- | :-------\n1 | User workload (`user-workload`, `main`, `jax-tpu`) | `--action=user-workload` | The last completed step, then the first exception | [A](references\u002Fcontainer-playbooks.md#a-user-workload-container)\n2 | IFRT proxy | `--action=proxy` | `Connection refused` to the RM port (`:29001`), `Socket closed` | [B](references\u002Fcontainer-playbooks.md#b-ifrt-proxy-server)\n3 | Resource manager (`pathways-rm`) | `--action=pathways-rm` | Slice ready transitions, placement, a repeated `Build label` (restarts) | [C](references\u002Fcontainer-playbooks.md#c-resource-manager-pathways-rm)\n4 | Pathways workers (`pathways-worker`) | `--action=workers` | The first worker that failed, Megascale stall detectors (`HANG_DETECTED`) | [D](references\u002Fcontainer-playbooks.md#d-pathways-workers-pathways-worker)\n5 | Colocated Python sidecar | `--action=colocated-python-sidecar` | Workers waiting for a sidecar that died | [E](references\u002Fcontainer-playbooks.md#e-colocated-python-sidecar)\n\nRules that apply to every container:\n\n-   A client-side `Unable to initialize backend 'proxy': UNAVAILABLE: Unable to\n    establish connection` is a symptom. Check whether the proxy, RM and workers\n    started (see [Parsing Important Logs](#parsing-important-logs)) and find\n    their earliest error before you suspect the user's code.\n-   `IFRT proxy server was terminated: UNAVAILABLE: Socket closed` and\n    `SLICE_FAILURE_CHIP_DRIVER_ERROR` are downstream errors: go to the RM and\n    worker logs.\n-   When one worker fails, the other workers of the slice fail shortly after.\n    The root cause is in the logs of the first worker that failed.\n-   The built-in search terms only cover known failures. If an action returns\n    nothing useful, re-run it with `--min_severity=WARNING` (`DEFAULT` for the\n    user workload) and add the workload's own strings with `--extra_terms`.\n-   If the logs still don't explain the failure, read all logs of the\n    container ([F](references\u002Fcontainer-playbooks.md#f-all-logs-of-a-container)),\n    then ask for more verbose proxy logs or XProfs\n    ([G](references\u002Fcontainer-playbooks.md#g-when-the-error-is-still-not-clear)).\n\n--------------------------------------------------------------------------------\n\n## Step 4: Special Scenarios\n\nThese known failures have a specific cause and fix. When an error or\nsymptom matches a row, read that scenario in\n[Special Scenarios](references\u002Fspecial-scenarios.md) before you conclude:\n\nError or symptom | Scenario\n:--------------- | :-------\n`JaxRuntimeError` with `DATA_LOSS` during scale changes; elastic recovery or checkpoint restore fails | [1. Elastic training and checkpoint restore](references\u002Fspecial-scenarios.md#1-elastic-training-and-checkpoint-restore)\nContainer restarts or exit code 137 without `OOMKilled`; `Memory cgroup out of memory`, `oom-kill`, `OOMKilling` | [2. Host OOM kills on the head pod or TPU hosts](references\u002Fspecial-scenarios.md#2-host-oom-kills-on-the-head-pod-or-tpu-hosts)\n`Unable to initialize backend 'proxy': Backend 'proxy' is not in the list of known backends` | [3. Backend proxy is not in the list of known backends](references\u002Fspecial-scenarios.md#3-backend-proxy-is-not-in-the-list-of-known-backends)\n`SIGTERM received by PID 1`; containers stop without a clear error | [4. SIGTERM received by PID 1](references\u002Fspecial-scenarios.md#4-sigterm-received-by-pid-1)\n`FAILED_PRECONDITION: The newly added instance does not match with the expected instances` | [5. Newly added instance does not match the expected instances](references\u002Fspecial-scenarios.md#5-newly-added-instance-does-not-match-the-expected-instances)\n`Connection to IFRT proxy server was terminated: FAILED_PRECONDITION: GrpcClientSession: writes no longer allowed` | [6. Connection to the IFRT proxy server was terminated](references\u002Fspecial-scenarios.md#6-connection-to-the-ifrt-proxy-server-was-terminated)\n`XlaRuntimeError: ABORTED: ... The metadata for object was edited during the operation` | [7. Cached executable metadata was edited during the operation](references\u002Fspecial-scenarios.md#7-cached-executable-metadata-was-edited-during-the-operation)\n`DEADLINE_EXCEEDED: lost connection to peer` or `DATA_LOSS: Pipe from scheduler to aggregator` | [8. Lost connection to peer](references\u002Fspecial-scenarios.md#8-lost-connection-to-peer)\n`Status 11 from handshaker service: Input frame length is out of range` | [9. ALTS handshake error from a client outside Google Cloud](references\u002Fspecial-scenarios.md#9-alts-handshake-error-from-a-client-outside-google-cloud)\n`Lifecycle matches_prefix cannot specify more than 50 prefixes per config` | [10. Cloud Storage lifecycle prefix limit](references\u002Fspecial-scenarios.md#10-cloud-storage-lifecycle-prefix-limit)\n`IFRT Proxy client and server failed to agree on the protocol version`, or `Failed to deserialize the Mosaic module: ... Unsupported version` | [11. JAX and Pathways version mismatch](references\u002Fspecial-scenarios.md#11-jax-and-pathways-version-mismatch): compare the client's JAX, jaxlib and pathwaysutils versions with the Pathways image tags and `Build label`s\nPods in `ImagePullBackOff` or `ErrImagePull` | [12. Pods stuck in ImagePullBackOff or ErrImagePull](references\u002Fspecial-scenarios.md#12-pods-stuck-in-imagepullbackoff-or-errimagepull): read the pull error in the pod events, then list the registry's tags (`gcloud artifacts docker tags list`)\n`Address already in use`, `Init:Error`, or `didn't have free ports` | [13. Port collisions with hostNetwork](references\u002Fspecial-scenarios.md#13-port-collisions-with-hostnetwork)\nNo error anywhere, but the job stops making progress | [14. The job hangs without failing](references\u002Fspecial-scenarios.md#14-the-job-hangs-without-failing)\n`Still waiting to acquire semaphore`, `Recv completion error`, `GOAWAY` | [15. Proxy-to-worker transfers stall](references\u002Fspecial-scenarios.md#15-proxy-to-worker-transfers-stall)\n\n--------------------------------------------------------------------------------\n\n## Related Public Skills and Writing Your Own Queries\n\nThese public skills from\n[github.com\u002Fgoogle\u002Fskills](https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills) complement this\nskill. If they are installed, use them by name; otherwise read them from the\nlinks.\n\nSymptom                                                         | Public skill\n:-------------------------------------------------------------- | :-----------\nWriting a custom Cloud Logging query                            | [cloud-logging-query-generation](https:\u002F\u002Fraw.githubusercontent.com\u002Fgoogle\u002Fskills\u002Fd542604d73544bcb4bcc61322f3b25522ceba7ed\u002Fskills\u002Fcloud\u002Fcloud-logging-query-generation\u002FSKILL.md) (GKE examples: [query_gke.md](https:\u002F\u002Fraw.githubusercontent.com\u002Fgoogle\u002Fskills\u002Fd542604d73544bcb4bcc61322f3b25522ceba7ed\u002Fskills\u002Fcloud\u002Fcloud-logging-query-generation\u002Freferences\u002Fquery_gke.md))\nJobSet restarts, Spot preemption, maintenance, host termination | [gke-ai-troubleshooting-jobset-interruption](https:\u002F\u002Fraw.githubusercontent.com\u002Fgoogle\u002Fskills\u002Fd542604d73544bcb4bcc61322f3b25522ceba7ed\u002Fskills\u002Fcloud\u002Fgke-ai-troubleshooting-jobset-interruption\u002FSKILL.md)\nCompute Engine host maintenance on TPU\u002FGPU nodes                | [gke-ai-troubleshooting-handle-disruption-gpu-tpu](https:\u002F\u002Fraw.githubusercontent.com\u002Fgoogle\u002Fskills\u002Fd542604d73544bcb4bcc61322f3b25522ceba7ed\u002Fskills\u002Fcloud\u002Fgke-ai-troubleshooting-handle-disruption-gpu-tpu\u002FSKILL.md)\nMulti-slice job stops responding (`HANG_DETECTED`)              | [gke-ai-troubleshooting-tpu-mxla-hang](https:\u002F\u002Fraw.githubusercontent.com\u002Fgoogle\u002Fskills\u002Fd542604d73544bcb4bcc61322f3b25522ceba7ed\u002Fskills\u002Fcloud\u002Fgke-ai-troubleshooting-tpu-mxla-hang\u002FSKILL.md)\nNode NotReady, unresponsive node, auto-repair                   | [gke-ai-troubleshooting-node-unresponsive-timeout](https:\u002F\u002Fraw.githubusercontent.com\u002Fgoogle\u002Fskills\u002Fd542604d73544bcb4bcc61322f3b25522ceba7ed\u002Fskills\u002Fcloud\u002Fgke-ai-troubleshooting-node-unresponsive-timeout\u002FSKILL.md)\n\nWhen you write your own queries:\n\n-   Always use double quotes for strings, uppercase `AND`\u002F`OR`\u002F`NOT`, and\n    parentheses to group terms.\n-   Restrict `resource.type` (`k8s_container`, `k8s_pod`, `k8s_node`,\n    `gke_nodepool`, `gce_instance`) and use `log_id(\"events\")`,\n    `log_id(\"stdout\")` or `log_id(\"stderr\")` where possible.\n-   Always add a time window (`timestamp>=\"...\"` and `timestamp\u003C=\"...\"`).\n-   Pathways components log glog lines into `jsonPayload.message` (some\n    containers) or `textPayload` (others); bare quoted terms or `SEARCH()` match\n    both.\n-   Pod label keys are stored with `.` replaced by `_`, e.g.\n    `labels.\"k8s-pod\u002Fbatch_kubernetes_io\u002Fjob-completion-index\"` and\n    `labels.\"k8s-pod\u002Fjobset_sigs_k8s_io\u002Fjobset-name\"`. Some public examples\n    spell the Job label with dots (`batch.kubernetes.io\u002Fjob-name`); that form\n    does not match Pathways logs.\n",{"name":5,"metadata":45,"description":7},{"version":46,"category":47},"1.0.0","AiAndMachineLearning","main","2194d3b8b1f31975fef2f1a0ae6b9ab55bd7ceee","skills\u002Fcloud\u002Fpathways-workload",2005,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills","https:\u002F\u002Fgithub.com\u002Fgoogle","https:\u002F\u002Fopensource.google\u002F","Google ❤️ Open Source","2026-11-14 01:51:35",{"source":8,"skillName":5,"commands":58,"prompt":68,"zipAvailable":69},[59,62,65],{"tool":60,"command":61},"npx","npx skills add google\u002Fskills --skill pathways-workload",{"tool":63,"command":64},"bunx","bunx skills add google\u002Fskills --skill pathways-workload",{"tool":66,"command":67},"pnpm","pnpm dlx skills add google\u002Fskills --skill pathways-workload","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fpathways-workload\u002FSKILL.md",true,[71,77,81,85],{"path":72,"resourceType":73,"resourceTypeDesc":74,"sizeBytes":75,"sourceHtmlUrl":76},"references\u002Fattribution.md","REFERENCE","参考文档",1891,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fpathways-workload\u002Freferences\u002Fattribution.md",{"path":78,"resourceType":73,"resourceTypeDesc":74,"sizeBytes":79,"sourceHtmlUrl":80},"references\u002Fcontainer-playbooks.md",11661,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fpathways-workload\u002Freferences\u002Fcontainer-playbooks.md",{"path":82,"resourceType":73,"resourceTypeDesc":74,"sizeBytes":83,"sourceHtmlUrl":84},"references\u002Fspecial-scenarios.md",14489,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fpathways-workload\u002Freferences\u002Fspecial-scenarios.md",{"path":86,"resourceType":87,"resourceTypeDesc":88,"sizeBytes":89,"sourceHtmlUrl":90},"scripts\u002Fdebug_workload.py","SCRIPT","脚本",47005,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fpathways-workload\u002Fscripts\u002Fdebug_workload.py",[92,130,160,189,215,240],{"skillNo":93,"name":94,"nameZh":6,"description":95,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":96,"categories":97,"keywords":106,"publishedAt":40},"01M6998S2EZKD3GEJ7F943RZTE","sign-in-with-google-web","Implement, configure, and secure Sign In With Google (SiwG) using Google Identity Services (GIS \u002F `https:\u002F\u002Faccounts.google.com\u002Fgsi\u002Fclient`) across web architectures. Use when creating Google sign-in buttons, implementing Google One Tap with FedCM, integrating GIS in React\u002FNext.js\u002FAngular\u002FHTML, verifying ID tokens on backend runtimes (Python\u002FNode.js\u002FGo\u002FJava), enforcing Google Workspace domain restrictions (`hd`), securing client-side ID tokens via WebCrypto nonces or encrypted IndexedDB, embedding in cross-origin iframes via the Intermediate Iframe API (`https:\u002F\u002Faccounts.google.com\u002Fgsi\u002Fintermediate`), configuring Content Security Policy (CSP), COOP, and Permissions Policy (`identity-credentials-get`) headers, or handling sign-out and revocation. Don't use for legacy Google Sign-In (`gapi.auth2`), Google Cloud IAM service accounts, or Android\u002FiOS native Credential Manager.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fidentity\u002Fsign-in-with-google-web\u002FSKILL.md",[98,103],{"categoryNo":99,"categoryName":100,"taxonomyType":101,"taxonomyTypeDesc":102},"01JBSEEDCAT000000000000009","前端开发","SCENARIO","使用场景",{"categoryNo":104,"categoryName":105,"taxonomyType":101,"taxonomyTypeDesc":102},"01JBSEEDCAT000000000000010","后端与接口",[9,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129],"sign in with google web","sign","with","web","android","go","security","google workspace","google cloud","implement","policy","python","implementing","react","iam","permissions","buttons","secure","manager","identity","architectures","domain","runtimes",{"skillNo":131,"name":132,"nameZh":6,"description":133,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":134,"categories":135,"keywords":145,"publishedAt":40},"01M6998PHPDCAD684VW5WRG83D","dpop-adoption","Implement and debug OAuth 2.0 DPoP (RFC 9449) refresh token sender-constraining for WebCrypto, Node.js ES6, and browser runtimes integrating with Google's OAuth platform. Use when configuring non-extractable asymmetric key pairs (P-256), generating DPoP Proof JWTs for authorization code exchange and token refresh, or handling 400 use_dpop_nonce challenge retry loops at oauth2.googleapis.com\u002Ftoken. Don't use for unconstrained OAuth 2.0 flows (where refresh tokens are not bound to a client key pair), or for Google Cloud IAM \u002F service account authentication.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fidentity\u002Fdpop-adoption\u002FSKILL.md",[136,137,140],{"categoryNo":104,"categoryName":105,"taxonomyType":101,"taxonomyTypeDesc":102},{"categoryNo":138,"categoryName":139,"taxonomyType":101,"taxonomyTypeDesc":102},"01JBSEEDCAT000000000000023","Skill与智能体开发",{"categoryNo":141,"categoryName":142,"taxonomyType":143,"taxonomyTypeDesc":144},"01JBSEEDCAT000000000000030","开发工程师","PROFESSION","适用职业",[146,147,9,115,148,149,150,151,152,153,121,154,155,156,33,157,158,126,129,159,24],"dpop adoption","adoption","es6","retry","authentication","flows","challenge","code","browser","loops","at","token","pair","platform",{"skillNo":161,"name":162,"nameZh":6,"description":163,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":164,"categories":165,"keywords":172,"publishedAt":40},"01M6998M72TC73EEAQH4XC7EXW","retrieving-developer-knowledge","Searches, retrieves, and synthesizes official Google developer documentation across Google Cloud, AI\u002FGemini, Android, Chrome, Web, Flutter, Go, Firebase, and other Google developer platforms. Integrates with the Developer Knowledge MCP server (search_documents, get_documents, answer_query) or the Developer Knowledge REST API fallback. Use when searching for gcloud CLI commands, API syntax, IAM permissions, official documentation, architectural comparisons, or product choice overviews. Don't use for local filesystem lookups or non-Google documentation.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fdevelopers\u002Fretrieving-developer-knowledge\u002FSKILL.md",[166,167,170,171],{"categoryNo":104,"categoryName":105,"taxonomyType":101,"taxonomyTypeDesc":102},{"categoryNo":168,"categoryName":169,"taxonomyType":101,"taxonomyTypeDesc":102},"01JBSEEDCAT000000000000021","资料检索与研究",{"categoryNo":138,"categoryName":139,"taxonomyType":101,"taxonomyTypeDesc":102},{"categoryNo":141,"categoryName":142,"taxonomyType":143,"taxonomyTypeDesc":144},[173,174,175,176,115,177,178,179,180,181,182,183,184,24,185,186,187,121,188,122],"retrieving developer knowledge","knowledge","developer","MCP","mcp server","official","developer documentation","get","search","local","choice","documents","query","api","across","documentation",{"skillNo":190,"name":191,"nameZh":6,"description":192,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":193,"categories":194,"keywords":195,"publishedAt":40},"01M6998J2CZEXS9BES9S3WWFKC","migrate-to-rust","Clones, analyzes, plans, and migrates legacy codebases (C++, Fortran, Java, Python, C, Shell) from a GitHub URL or local directory to a modern, high-performance, deterministic, 100% Safe Rust architecture, tracking live telemetry across every task bucket, test failure\u002Ffix cycle, and Profile-Guided Optimization (PGO) stage, and producing a first-person AI storytelling report and interactive HTML infographic. Use when migrating a GitHub repository or local codebase to Rust, porting legacy C++, Fortran, Python, or Java workloads to memory-safe Rust, designing Struct-of-Arrays (SoA) or SIMD Rust architectures, or optimizing Rust ports with LLVM PGO. Don't use for unrelated non-Rust refactoring or single-line syntax questions.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fdevelopers\u002Fmigrate-to-rust\u002FSKILL.md",[],[196,197,198,199,200,201,202,203,118,204,205,206,207,208,32,182,209,210,211,24,212,213,187,214],"migrate to rust","rust","migrate","github","memory","performance","directory","safe","ports","llvm","soa","cycle","porting","arrays","infographic","high-performance","URL","architecture","unrelated",{"skillNo":216,"name":217,"nameZh":6,"description":218,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":219,"categories":220,"keywords":224,"publishedAt":40},"01M6998GCC3K29WZ82KXMB9E02","finding-google-skills","Google platform decision and setup guidance, loaded on demand from Google's skill catalog. Use when a developer is choosing or setting up part of their stack, such as where to run a service, a database, storage, messaging, authentication, analytics, ads, or AI model serving, and a Google product is a reasonable candidate - whether or not a vendor is named - or when a request names a Google product or API. Brings in the matching Google skill so the answer can weigh Google options, their trade-offs, and when they are not the right fit. Skip when the stack is already settled on another provider and no Google product is named, or the task involves no platform choice.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fdevelopers\u002Ffinding-google-skills\u002FSKILL.md",[221,222,223],{"categoryNo":104,"categoryName":105,"taxonomyType":101,"taxonomyTypeDesc":102},{"categoryNo":138,"categoryName":139,"taxonomyType":101,"taxonomyTypeDesc":102},{"categoryNo":141,"categoryName":142,"taxonomyType":143,"taxonomyTypeDesc":144},[9,225,226,227,228,159,229,230,231,232,233,150,234,235,236,183,237,238,24,239,186],"finding google skills","storage","as","messaging","model","trade-offs","provider","answer","fit","catalog","up","decision","use","database","candidate",{"skillNo":241,"name":242,"nameZh":6,"description":243,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":244,"categories":245,"keywords":248,"publishedAt":40},"01M6998DWNHTH0BS6W1CY8ZXMP","workload-manager-basics","Use this skill to manage Google Cloud Workload Manager evaluations, rules, scanned resources, and validation results by using public client libraries and the REST API. Use when you need to inspect workload best-practice rules, create and run evaluations for Google Cloud general best practices, SAP, SQL Server, or custom organizational rules, review violations, export results to BigQuery, or automate Workload Manager through client libraries because no service-specific public CLI or MCP server is available. Don't use for general Google Compute Engine instance management, VPC configuration, or standard IAM auditing.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fworkload-manager-basics\u002FSKILL.md",[246,247],{"categoryNo":104,"categoryName":105,"taxonomyType":101,"taxonomyTypeDesc":102},{"categoryNo":138,"categoryName":139,"taxonomyType":101,"taxonomyTypeDesc":102},[249,19,125,250,9,176,251,115,177,252,253,254,255,24,256,186,121,257,258,259,38],"workload manager basics","basics","cli","review","vpc","resources","inspect","you","create","libraries","server"]