[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M69931J2PCGJ8EDJ79BQPA76":3,"skill-same:01M69931J2PCGJ8EDJ79BQPA76":98},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":41,"linkOnly":42,"markdownContent":43,"markdownBody":44,"frontMatter":45,"refName":49,"commitSha":50,"skillDirectory":51,"forksCount":52,"repositoryHtmlUrl":53,"repositoryHomepageUrl":6,"ownerHtmlUrl":54,"ownerBlogUrl":55,"ownerDescription":56,"updatedAt":57,"install":58,"resources":71},"01M69931J2PCGJ8EDJ79BQPA76","gke-ai-troubleshooting-tpu-mxla-hang",null,"Diagnose GKE Cloud TPU multi-slice training hangs (Megascale `HANG_DETECTED` logs and hang monitored events) using the ML Diagnostics `Megascale XLA (MXLA) Hang Analyzer` (`gcloud alpha mldiagnostics monitored-events`) and 1-minute Cloud Monitoring multi-slice latency metrics (`kubernetes.io\u002Fcontainer\u002Fmultislice\u002F*`). Distinguishes XLA compiler\u002FHLO launch divergence and host data-input stalls from TPU chip, SparseCore, ICI, or network fabric faults. Use when multi-slice TPU training jobs freeze without progressing steps, emit `HANG_DETECTED`, or stall in collective operations. Don't use for gradual step-time throughput drops without hangs (use gke-ai-troubleshooting-tpu-performance-degradation) or pod preemption\u002Feviction restarts (use gke-ai-troubleshooting-jobset-interruption).","google\u002Fskills","google","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F1342004?v=4","01M698VYEFDM0VJBZDYYJ2T83S","Google","01M698WR0H76ENHE587T9EJT89",24122,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fgke-ai-troubleshooting-tpu-mxla-hang\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40],"gke ai troubleshooting tpu mxla hang","tpu","troubleshooting","gke","data","Training Jobs","Preemption","when","collective","input","io","host","time","gcloud","network","monitoring","logs","alpha","training","diagnostics","performance","kubernetes","2026-11-10 10:12:30",false,"---\nname: gke-ai-troubleshooting-tpu-mxla-hang\ndescription: >-\n  Diagnose GKE Cloud TPU multi-slice training hangs (Megascale `HANG_DETECTED`\n  logs and hang monitored events) using the ML Diagnostics `Megascale XLA (MXLA)\n  Hang Analyzer` (`gcloud alpha mldiagnostics monitored-events`) and 1-minute\n  Cloud Monitoring multi-slice latency metrics\n  (`kubernetes.io\u002Fcontainer\u002Fmultislice\u002F*`). Distinguishes XLA compiler\u002FHLO\n  launch divergence and host data-input stalls from TPU chip, SparseCore, ICI,\n  or network fabric faults. Use when multi-slice TPU training jobs freeze\n  without progressing steps, emit `HANG_DETECTED`, or stall in collective\n  operations. Don't use for gradual step-time throughput drops without hangs\n  (use gke-ai-troubleshooting-tpu-performance-degradation) or pod\n  preemption\u002Feviction restarts (use gke-ai-troubleshooting-jobset-interruption).\nmetadata:\n  version: \"1.0.1\"\n  category: AiAndMachineLearning\n---\n\n# Troubleshoot GKE TPU multi-slice hangs with the MXLA Hang Analyzer\n\nDiagnose Cloud TPU multi-slice training hangs on Google Kubernetes Engine (GKE)\nby correlating Megascale `HANG_DETECTED` logs and **ML Diagnostics Workload\nMonitoring** `Megascale XLA (MXLA) Hang Analyzer` reports with **1-minute\nmulti-slice latency metrics** (`kubernetes.io\u002Fcontainer\u002Fmultislice\u002F*`) and GKE\nnode topology labels.\n\n---\n\n## Attribution\n\n**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you\nMUST read [Usage Attribution](references\u002Fattribution.md). For commands executed\ndirectly in the terminal (never in generated scripts, Terraform, or user code),\nprefix `gcloud` with\n`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION=\"agent-skills\u002F1.0 (skill:gke-ai-troubleshooting-tpu-mxla-hang)\"`\nand append `agent-skills\u002F1.0 (skill:gke-ai-troubleshooting-tpu-mxla-hang)` to `User-Agent` headers.\n\n## Prerequisites\n\n- **Tools**: Install the\n  [Google Cloud SDK](https:\u002F\u002Fcloud.google.com\u002Fsdk\u002Fdocs\u002Finstall) (`gcloud` with\n  `alpha` component for `gcloud alpha mldiagnostics`) and `kubectl`.\n- **Cloud Billing & Project Configuration**: Verify an active billing account is\n  linked (`gcloud billing projects describe {project_id}`), authenticate\n  (`gcloud auth login`), set the target project (`gcloud config set project\n  {project_id}`), and ensure `container.googleapis.com`,\n  `logging.googleapis.com`, `monitoring.googleapis.com`, and\n  `hypercomputecluster.googleapis.com` are enabled.\n- **Supported workloads and versions**: Google Cloud ML Diagnostics only\n  supports JAX on TPUs (see\n  [ML Diagnostics platform](https:\u002F\u002Fdocs.cloud.google.com\u002Ftpu\u002Fdocs\u002Fml-diagnostics\u002Foverview.md.txt)).\n  Workload Monitoring is enabled by default, supports the `jobset` and `job` GKE\n  job types, and is compatible with GKE versions `1.36.0-gke.4681000` and later\n  ([Configure GKE for ML Diagnostics](https:\u002F\u002Fdocs.cloud.google.com\u002Ftpu\u002Fdocs\u002Fml-diagnostics\u002Fgke.md.txt),\n  which also covers the cluster setup needed for on-demand profiling in Step 4\n  Path B). If a workload uses another framework (such as PyTorch) or another\n  custom resource type, `gcloud alpha mldiagnostics` won't list ML runs or\n  monitored events for it. The Megascale XLA hang analyzer and Megascale XLA\n  metrics require LibTPU `0.40.0` or later (see \"Get started\" in\n  [Workload monitoring with ML Diagnostics](https:\u002F\u002Fdocs.cloud.google.com\u002Ftpu\u002Fdocs\u002Fml-diagnostics\u002Fworkload-monitoring.md.txt)).\n- **Required IAM Roles**:\n  - Cluster Director Editor (`roles\u002Fhypercomputecluster.editor`), the role\n    listed in the \"IAM permissions\" section of\n    [ML Diagnostics platform](https:\u002F\u002Fdocs.cloud.google.com\u002Ftpu\u002Fdocs\u002Fml-diagnostics\u002Foverview.md.txt),\n    for the ML Diagnostics CLI and API calls in this skill (ML runs, monitored\n    events, and on-demand profiler sessions)\n  - Monitoring Viewer (`roles\u002Fmonitoring.viewer`) for the PromQL queries in Step\n    2\n  - Logs Viewer (`roles\u002Flogging.viewer`) for the Cloud Logging query in Step 1\n  - Kubernetes Engine Viewer (`roles\u002Fcontainer.viewer`) for the `kubectl get\n    nodes` query in Step 3\n  - For remediation (`[High Risk]` steps): Kubernetes Engine Cluster Admin\n    (`roles\u002Fcontainer.clusterAdmin`)\n- **Reference Documentation**:\n  - [ML Diagnostics platform](https:\u002F\u002Fdocs.cloud.google.com\u002Ftpu\u002Fdocs\u002Fml-diagnostics\u002Foverview.md.txt)\n    (Sections: \"IAM permissions\")\n  - [Configure GKE for ML Diagnostics](https:\u002F\u002Fdocs.cloud.google.com\u002Ftpu\u002Fdocs\u002Fml-diagnostics\u002Fgke.md.txt)\n    (Sections: \"Set up with gcloud CLI, Google Cloud console, or Terraform\",\n    \"Manual installation\", \"Connection-operator\")\n  - [Workload monitoring with ML Diagnostics](https:\u002F\u002Fdocs.cloud.google.com\u002Ftpu\u002Fdocs\u002Fml-diagnostics\u002Fworkload-monitoring.md.txt)\n    (Sections: \"Get started\", \"Megascale XLA (MXLA) Hang Analyzer\", \"Access\n    Workload Monitoring information through the API\", \"System Metrics\")\n  - [Get started with the ML Diagnostics CLI](https:\u002F\u002Fdocs.cloud.google.com\u002Ftpu\u002Fdocs\u002Fml-diagnostics\u002Fcli.md.txt)\n    (Sections: \"List machine learning runs\", \"Monitored-events commands\", \"List\n    profiler targets\", \"Capture on-demand profiler sessions\")\n  - [Get support](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fgetting-support.md.txt)\n    (Sections: \"Before you contact support\")\n  - [Auto-repair nodes](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt)\n    (Sections: \"Repair criteria\", \"Verify node auto-repair is enabled for a\n    Standard node pool\", \"Enable auto-repair for an existing Standard node\n    pool\", \"Node auto repair in TPU slice nodes\")\n  - [Deploy TPU workloads in GKE Standard](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Ftpus.md.txt)\n    (Sections: \"Configure auto repair for TPU slice nodes\")\n  - [Dump HLO Computations (OpenXLA)](https:\u002F\u002Fopenxla.org\u002Fxla\u002Fhlo_dumps)\n\n> **Read-only rule**: Run read-only diagnostic commands only. Never drain,\n> delete, or re-create nodes, or run any other command that changes the cluster.\n> Give the user any fix to apply themselves.\n\nWhen you recommend a fix, link the doc section that describes it.\n\n---\n\n## MXLA Hang Analyzer routing overview\n\nA Megascale hang occurs when a multi-slice worker has waited on a Megascale\ncommunication operation for a set timeout period. The TPU logs then show a\nMegascale `HANG_DETECTED` message. `HANG_DETECTED` is a catch-all signal that\nthe workload isn't progressing, and the cause can be in software or in hardware.\nWhen a hang occurs, ML Diagnostics runs the `Megascale XLA (MXLA) Hang\nAnalyzer`, which reports the likely cause as a code. Consult the\n[Megascale XLA (MXLA) Hang Analyzer](https:\u002F\u002Fdocs.cloud.google.com\u002Ftpu\u002Fdocs\u002Fml-diagnostics\u002Fworkload-monitoring.md.txt)\nsection for the definition and recommended action of each code, and route by\ncategory:\n\n- **Category A: compiler or HLO divergence (never cordon or replace nodes)**:\n  - When the analyzer reports different HLO modules, inconsistent HLO\n    compilation, or an inconsistent launch order across VMs (such as\n    `FINGERPRINT_MISMATCH`), route to\n    [Path A: Compiler or HLO divergence](references\u002Fpath-a-compiler-hlo-divergence.md).\n- **Category B: host program queueing or data input stall (never cordon or\n  replace nodes)**:\n  - When the analyzer reports that VMs aren't queuing programs to the TPU or\n    that data input stalled (such as `DATA_INPUT_STALL`), route to\n    [Path B: Host program queueing or data input stall](references\u002Fpath-b-program-queueing-input-stall.md).\n- **Category C: hardware or network faults on specific instances**:\n  - When the analyzer attributes the hang to a TPU chip, SparseCore, ICI, or DCN\n    networking issue, or to an unrecoverable error on specific instances, route\n    to\n    [Path C: Hardware or network faults](references\u002Fpath-c-hardware-network-faults.md).\n- **Category D: hang signal or event exists, but the analyzer is `NOT_DETECTED`\n  or reports `UNKNOWN`**:\n  - When `HANG_DETECTED` logs or a hang monitored event fired, but the analyzer\n    report has `detectionState: \"NOT_DETECTED\"` or reports `UNKNOWN` (\"The MXLA\n    hang was detected but a potential cause is not determined\"), route to\n    [Path D: No DETECTED analyzer or UNKNOWN cause](references\u002Fpath-d-no-detected-or-unknown.md).\n\n---\n\n## Diagnostic workflow\n\n### Step 0: Collect context and set the investigation window `[Low Risk]`\n\nCollect the target parameters. By default, query a 60-minute window `[T - 30m, T\n+ 30m]` around `{issue_time}`:\n\n- `{project_id}`: Google Cloud project ID\n- `{location}`: Google Cloud region where the ML run and GKE cluster reside (for\n  example, `us-central1`)\n- `{cluster_name}`: GKE cluster name\n- `{namespace}` \u002F `{workload_name}`: Kubernetes namespace and JobSet\u002FPod prefix\n- `{ml_run_id}`: ML Diagnostics run ID\n- `{issue_time}`: Timestamp when the hang occurred (`T`, ISO-8601 UTC)\n- `{start_time}`: `T - 30m`\n- `{end_time}`: `T + 30m`\n\n---\n\n### Step 1: Query `HANG_DETECTED` logs and hang events `[Low Risk]`\n\n1. **Check GKE container logs for `HANG_DETECTED` (read-only Cloud Logging\n   LQL)**: Query `k8s_container` logs over `[{start_time}, {end_time}]` to\n   confirm `HANG_DETECTED` and identify the first stalled pods:\n\n```sql\nresource.type=\"k8s_container\"\nresource.labels.project_id=\"{project_id}\"\nresource.labels.cluster_name=\"{cluster_name}\"\n\"HANG_DETECTED\"\ntimestamp >= \"{start_time}\" AND timestamp \u003C= \"{end_time}\"\n```\n\n2. **List active or recent ML runs**: Follow the\n   [List machine learning runs](https:\u002F\u002Fdocs.cloud.google.com\u002Ftpu\u002Fdocs\u002Fml-diagnostics\u002Fcli.md.txt)\n   section of the ML Diagnostics CLI reference (`gcloud alpha mldiagnostics\n   machine-learning-run list`) to identify `{ml_run_id}`.\n3. **List and describe hang monitored events**: Follow the\n   [Monitored-events commands](https:\u002F\u002Fdocs.cloud.google.com\u002Ftpu\u002Fdocs\u002Fml-diagnostics\u002Fcli.md.txt)\n   section (`gcloud alpha mldiagnostics monitored-events list` and `gcloud alpha\n   mldiagnostics monitored-events describe`) or\n   [Access Workload Monitoring information through the API](https:\u002F\u002Fdocs.cloud.google.com\u002Ftpu\u002Fdocs\u002Fml-diagnostics\u002Fworkload-monitoring.md.txt)\n   to inspect the `Megascale XLA (MXLA) Hang Analyzer` report (`detectionState`,\n   `details`, and `recommendedActions`).\n   - If `HANG_DETECTED` logs or a hang monitored event fired, run Step 2 to\n     correlate with the 1-minute metrics; if no analyzer reports\n     `detectionState: \"DETECTED\"` or the analyzer reports `UNKNOWN`, follow\n     [Path D: No DETECTED analyzer or UNKNOWN cause](references\u002Fpath-d-no-detected-or-unknown.md).\n   - If no `HANG_DETECTED` log or hang monitored event exists and multi-slice\n     latencies are normal in Step 2, **rule out** an MXLA hang by following\n     [Path E: Healthy telemetry](references\u002Fpath-e-healthy-telemetry.md).\n\n---\n\n### Step 2: Correlate with 1-minute multi-slice and TPU metrics `[Low Risk]`\n\nConsult the\n[System Metrics](https:\u002F\u002Fdocs.cloud.google.com\u002Ftpu\u002Fdocs\u002Fml-diagnostics\u002Fworkload-monitoring.md.txt)\nsection of the Workload Monitoring guide for the 1-minute multi-slice network\n(`kubernetes.io\u002Fcontainer\u002Fmultislice\u002Fnetwork\u002F*`), multi-slice accelerator\n(`kubernetes.io\u002Fcontainer\u002Fmultislice\u002Faccelerator\u002F*`), and node duty-cycle\n(`kubernetes.io\u002Fnode\u002Faccelerator\u002Fduty_cycle`) metrics, and run read-only PromQL\nqueries over `[{start_time}, {end_time}]`:\n\n```promql\n# 1. P95 multi-slice collective end-to-end latency by pod\nhistogram_quantile(\n  0.95,\n  sum by (pod_name, le) (\n    rate(kubernetes_io:container_multislice_network_collective_end_to_end_latencies_bucket{\n      monitored_resource=\"k8s_container\",\n      project_id=\"{project_id}\",\n      cluster_name=\"{cluster_name}\"\n    }[5m])\n  )\n)\n\n# 2. P95 multi-slice DCN transfer latency by pod\nhistogram_quantile(\n  0.95,\n  sum by (pod_name, le) (\n    rate(kubernetes_io:container_multislice_network_dcn_transfer_latencies_bucket{\n      monitored_resource=\"k8s_container\",\n      project_id=\"{project_id}\",\n      cluster_name=\"{cluster_name}\"\n    }[5m])\n  )\n)\n\n# 3. P95 host-to-device transfer latency by pod\nhistogram_quantile(\n  0.95,\n  sum by (pod_name, le) (\n    rate(kubernetes_io:container_multislice_accelerator_host_to_device_transfer_latencies_bucket{\n      monitored_resource=\"k8s_container\",\n      project_id=\"{project_id}\",\n      cluster_name=\"{cluster_name}\"\n    }[5m])\n  )\n)\n\n# 4. Node TPU duty cycle (Workload Monitoring detects a hang as a prolonged\n#    period of minimal to no TPU activity)\nkubernetes_io:node_accelerator_duty_cycle{\n  monitored_resource=\"k8s_node\",\n  project_id=\"{project_id}\",\n  cluster_name=\"{cluster_name}\"\n}\n```\n\n---\n\n### Step 3: Map culprit Compute Engine instance IDs to GKE nodes `[Low Risk]`\n\nWhen the `Megascale XLA (MXLA) Hang Analyzer` reports culprit numeric Compute\nEngine instance IDs in `details` or `recommendedActions`, map those numeric IDs\nto GKE `Node` names and physical topology blocks using this read-only `kubectl`\nquery inspecting `container.googleapis.com\u002Finstance_id`:\n\n```bash\nkubectl get nodes -l cloud.google.com\u002Fgke-tpu-accelerator \\\n  -o jsonpath='{range .items[*]}{.metadata.name}{\"\\tinstance_id=\"}{.metadata.annotations.container\\.googleapis\\.com\u002Finstance_id}{\"\\tblock=\"}{.metadata.labels.cloud\\.google\\.com\u002Fgce-topology-block}{\"\\tsubblock=\"}{.metadata.labels.cloud\\.google\\.com\u002Fgce-topology-subblock}{\"\\thost=\"}{.metadata.labels.cloud\\.google\\.com\u002Fgce-topology-host}{\"\\n\"}{end}'\n```\n\n---\n\n### Step 4: Remediation by root-cause category\n\nLoad and follow only the reference file that matches the root-cause category\nfrom Step 1:\n\n- **Path A (compiler or HLO divergence — different HLO modules, inconsistent HLO\n  compilation, or inconsistent launch order across VMs, such as\n  `FINGERPRINT_MISMATCH`)**: Read\n  [Path A: Compiler or HLO divergence](references\u002Fpath-a-compiler-hlo-divergence.md).\n- **Path B (host program queueing or data input stall — `PROGRAM_NOT_QUEUED` or\n  `DATA_INPUT_STALL`)**: Read\n  [Path B: Host program queueing or data input stall](references\u002Fpath-b-program-queueing-input-stall.md).\n- **Path C (hardware, SparseCore, ICI, DCN networking faults, or\n  `UNRECOVERABLE_ERROR` on specific instances)**: Read\n  [Path C: Hardware or network faults](references\u002Fpath-c-hardware-network-faults.md).\n- **Path D (`HANG_DETECTED` logs or hang event fired, but the analyzer is\n  `NOT_DETECTED` or reports `UNKNOWN`)**: Read\n  [Path D: No DETECTED analyzer or UNKNOWN cause](references\u002Fpath-d-no-detected-or-unknown.md).\n- **Path E (no `HANG_DETECTED` logs or hang events, and steady telemetry)**:\n  Read [Path E: Healthy telemetry](references\u002Fpath-e-healthy-telemetry.md).\n\n---\n\n## Guardrails\n\n1. **Never change GKE-managed instance groups or VMs through Compute Engine**:\n   Don't run `gcloud compute instance-groups managed` commands, such as\n   `delete`, on a node pool's managed instance group. Handle nodes through GKE,\n   as described in\n   [Path C: Hardware or network faults](references\u002Fpath-c-hardware-network-faults.md).\n2. **Never cordon or replace nodes for compiler divergence or input stalls**: If\n   the analyzer reports a compiler or HLO divergence, a program queueing issue,\n   or a data input stall, don't cordon or replace TPU nodes. Follow\n   [Path A: Compiler or HLO divergence](references\u002Fpath-a-compiler-hlo-divergence.md)\n   or\n   [Path B: Host program queueing or data input stall](references\u002Fpath-b-program-queueing-input-stall.md)\n   instead.\n","# Troubleshoot GKE TPU multi-slice hangs with the MXLA Hang Analyzer\n\nDiagnose Cloud TPU multi-slice training hangs on Google Kubernetes Engine (GKE)\nby correlating Megascale `HANG_DETECTED` logs and **ML Diagnostics Workload\nMonitoring** `Megascale XLA (MXLA) Hang Analyzer` reports with **1-minute\nmulti-slice latency metrics** (`kubernetes.io\u002Fcontainer\u002Fmultislice\u002F*`) and GKE\nnode topology labels.\n\n---\n\n## Attribution\n\n**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you\nMUST read [Usage Attribution](references\u002Fattribution.md). For commands executed\ndirectly in the terminal (never in generated scripts, Terraform, or user code),\nprefix `gcloud` with\n`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION=\"agent-skills\u002F1.0 (skill:gke-ai-troubleshooting-tpu-mxla-hang)\"`\nand append `agent-skills\u002F1.0 (skill:gke-ai-troubleshooting-tpu-mxla-hang)` to `User-Agent` headers.\n\n## Prerequisites\n\n- **Tools**: Install the\n  [Google Cloud SDK](https:\u002F\u002Fcloud.google.com\u002Fsdk\u002Fdocs\u002Finstall) (`gcloud` with\n  `alpha` component for `gcloud alpha mldiagnostics`) and `kubectl`.\n- **Cloud Billing & Project Configuration**: Verify an active billing account is\n  linked (`gcloud billing projects describe {project_id}`), authenticate\n  (`gcloud auth login`), set the target project (`gcloud config set project\n  {project_id}`), and ensure `container.googleapis.com`,\n  `logging.googleapis.com`, `monitoring.googleapis.com`, and\n  `hypercomputecluster.googleapis.com` are enabled.\n- **Supported workloads and versions**: Google Cloud ML Diagnostics only\n  supports JAX on TPUs (see\n  [ML Diagnostics platform](https:\u002F\u002Fdocs.cloud.google.com\u002Ftpu\u002Fdocs\u002Fml-diagnostics\u002Foverview.md.txt)).\n  Workload Monitoring is enabled by default, supports the `jobset` and `job` GKE\n  job types, and is compatible with GKE versions `1.36.0-gke.4681000` and later\n  ([Configure GKE for ML Diagnostics](https:\u002F\u002Fdocs.cloud.google.com\u002Ftpu\u002Fdocs\u002Fml-diagnostics\u002Fgke.md.txt),\n  which also covers the cluster setup needed for on-demand profiling in Step 4\n  Path B). If a workload uses another framework (such as PyTorch) or another\n  custom resource type, `gcloud alpha mldiagnostics` won't list ML runs or\n  monitored events for it. The Megascale XLA hang analyzer and Megascale XLA\n  metrics require LibTPU `0.40.0` or later (see \"Get started\" in\n  [Workload monitoring with ML Diagnostics](https:\u002F\u002Fdocs.cloud.google.com\u002Ftpu\u002Fdocs\u002Fml-diagnostics\u002Fworkload-monitoring.md.txt)).\n- **Required IAM Roles**:\n  - Cluster Director Editor (`roles\u002Fhypercomputecluster.editor`), the role\n    listed in the \"IAM permissions\" section of\n    [ML Diagnostics platform](https:\u002F\u002Fdocs.cloud.google.com\u002Ftpu\u002Fdocs\u002Fml-diagnostics\u002Foverview.md.txt),\n    for the ML Diagnostics CLI and API calls in this skill (ML runs, monitored\n    events, and on-demand profiler sessions)\n  - Monitoring Viewer (`roles\u002Fmonitoring.viewer`) for the PromQL queries in Step\n    2\n  - Logs Viewer (`roles\u002Flogging.viewer`) for the Cloud Logging query in Step 1\n  - Kubernetes Engine Viewer (`roles\u002Fcontainer.viewer`) for the `kubectl get\n    nodes` query in Step 3\n  - For remediation (`[High Risk]` steps): Kubernetes Engine Cluster Admin\n    (`roles\u002Fcontainer.clusterAdmin`)\n- **Reference Documentation**:\n  - [ML Diagnostics platform](https:\u002F\u002Fdocs.cloud.google.com\u002Ftpu\u002Fdocs\u002Fml-diagnostics\u002Foverview.md.txt)\n    (Sections: \"IAM permissions\")\n  - [Configure GKE for ML Diagnostics](https:\u002F\u002Fdocs.cloud.google.com\u002Ftpu\u002Fdocs\u002Fml-diagnostics\u002Fgke.md.txt)\n    (Sections: \"Set up with gcloud CLI, Google Cloud console, or Terraform\",\n    \"Manual installation\", \"Connection-operator\")\n  - [Workload monitoring with ML Diagnostics](https:\u002F\u002Fdocs.cloud.google.com\u002Ftpu\u002Fdocs\u002Fml-diagnostics\u002Fworkload-monitoring.md.txt)\n    (Sections: \"Get started\", \"Megascale XLA (MXLA) Hang Analyzer\", \"Access\n    Workload Monitoring information through the API\", \"System Metrics\")\n  - [Get started with the ML Diagnostics CLI](https:\u002F\u002Fdocs.cloud.google.com\u002Ftpu\u002Fdocs\u002Fml-diagnostics\u002Fcli.md.txt)\n    (Sections: \"List machine learning runs\", \"Monitored-events commands\", \"List\n    profiler targets\", \"Capture on-demand profiler sessions\")\n  - [Get support](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fgetting-support.md.txt)\n    (Sections: \"Before you contact support\")\n  - [Auto-repair nodes](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt)\n    (Sections: \"Repair criteria\", \"Verify node auto-repair is enabled for a\n    Standard node pool\", \"Enable auto-repair for an existing Standard node\n    pool\", \"Node auto repair in TPU slice nodes\")\n  - [Deploy TPU workloads in GKE Standard](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Ftpus.md.txt)\n    (Sections: \"Configure auto repair for TPU slice nodes\")\n  - [Dump HLO Computations (OpenXLA)](https:\u002F\u002Fopenxla.org\u002Fxla\u002Fhlo_dumps)\n\n> **Read-only rule**: Run read-only diagnostic commands only. Never drain,\n> delete, or re-create nodes, or run any other command that changes the cluster.\n> Give the user any fix to apply themselves.\n\nWhen you recommend a fix, link the doc section that describes it.\n\n---\n\n## MXLA Hang Analyzer routing overview\n\nA Megascale hang occurs when a multi-slice worker has waited on a Megascale\ncommunication operation for a set timeout period. The TPU logs then show a\nMegascale `HANG_DETECTED` message. `HANG_DETECTED` is a catch-all signal that\nthe workload isn't progressing, and the cause can be in software or in hardware.\nWhen a hang occurs, ML Diagnostics runs the `Megascale XLA (MXLA) Hang\nAnalyzer`, which reports the likely cause as a code. Consult the\n[Megascale XLA (MXLA) Hang Analyzer](https:\u002F\u002Fdocs.cloud.google.com\u002Ftpu\u002Fdocs\u002Fml-diagnostics\u002Fworkload-monitoring.md.txt)\nsection for the definition and recommended action of each code, and route by\ncategory:\n\n- **Category A: compiler or HLO divergence (never cordon or replace nodes)**:\n  - When the analyzer reports different HLO modules, inconsistent HLO\n    compilation, or an inconsistent launch order across VMs (such as\n    `FINGERPRINT_MISMATCH`), route to\n    [Path A: Compiler or HLO divergence](references\u002Fpath-a-compiler-hlo-divergence.md).\n- **Category B: host program queueing or data input stall (never cordon or\n  replace nodes)**:\n  - When the analyzer reports that VMs aren't queuing programs to the TPU or\n    that data input stalled (such as `DATA_INPUT_STALL`), route to\n    [Path B: Host program queueing or data input stall](references\u002Fpath-b-program-queueing-input-stall.md).\n- **Category C: hardware or network faults on specific instances**:\n  - When the analyzer attributes the hang to a TPU chip, SparseCore, ICI, or DCN\n    networking issue, or to an unrecoverable error on specific instances, route\n    to\n    [Path C: Hardware or network faults](references\u002Fpath-c-hardware-network-faults.md).\n- **Category D: hang signal or event exists, but the analyzer is `NOT_DETECTED`\n  or reports `UNKNOWN`**:\n  - When `HANG_DETECTED` logs or a hang monitored event fired, but the analyzer\n    report has `detectionState: \"NOT_DETECTED\"` or reports `UNKNOWN` (\"The MXLA\n    hang was detected but a potential cause is not determined\"), route to\n    [Path D: No DETECTED analyzer or UNKNOWN cause](references\u002Fpath-d-no-detected-or-unknown.md).\n\n---\n\n## Diagnostic workflow\n\n### Step 0: Collect context and set the investigation window `[Low Risk]`\n\nCollect the target parameters. By default, query a 60-minute window `[T - 30m, T\n+ 30m]` around `{issue_time}`:\n\n- `{project_id}`: Google Cloud project ID\n- `{location}`: Google Cloud region where the ML run and GKE cluster reside (for\n  example, `us-central1`)\n- `{cluster_name}`: GKE cluster name\n- `{namespace}` \u002F `{workload_name}`: Kubernetes namespace and JobSet\u002FPod prefix\n- `{ml_run_id}`: ML Diagnostics run ID\n- `{issue_time}`: Timestamp when the hang occurred (`T`, ISO-8601 UTC)\n- `{start_time}`: `T - 30m`\n- `{end_time}`: `T + 30m`\n\n---\n\n### Step 1: Query `HANG_DETECTED` logs and hang events `[Low Risk]`\n\n1. **Check GKE container logs for `HANG_DETECTED` (read-only Cloud Logging\n   LQL)**: Query `k8s_container` logs over `[{start_time}, {end_time}]` to\n   confirm `HANG_DETECTED` and identify the first stalled pods:\n\n```sql\nresource.type=\"k8s_container\"\nresource.labels.project_id=\"{project_id}\"\nresource.labels.cluster_name=\"{cluster_name}\"\n\"HANG_DETECTED\"\ntimestamp >= \"{start_time}\" AND timestamp \u003C= \"{end_time}\"\n```\n\n2. **List active or recent ML runs**: Follow the\n   [List machine learning runs](https:\u002F\u002Fdocs.cloud.google.com\u002Ftpu\u002Fdocs\u002Fml-diagnostics\u002Fcli.md.txt)\n   section of the ML Diagnostics CLI reference (`gcloud alpha mldiagnostics\n   machine-learning-run list`) to identify `{ml_run_id}`.\n3. **List and describe hang monitored events**: Follow the\n   [Monitored-events commands](https:\u002F\u002Fdocs.cloud.google.com\u002Ftpu\u002Fdocs\u002Fml-diagnostics\u002Fcli.md.txt)\n   section (`gcloud alpha mldiagnostics monitored-events list` and `gcloud alpha\n   mldiagnostics monitored-events describe`) or\n   [Access Workload Monitoring information through the API](https:\u002F\u002Fdocs.cloud.google.com\u002Ftpu\u002Fdocs\u002Fml-diagnostics\u002Fworkload-monitoring.md.txt)\n   to inspect the `Megascale XLA (MXLA) Hang Analyzer` report (`detectionState`,\n   `details`, and `recommendedActions`).\n   - If `HANG_DETECTED` logs or a hang monitored event fired, run Step 2 to\n     correlate with the 1-minute metrics; if no analyzer reports\n     `detectionState: \"DETECTED\"` or the analyzer reports `UNKNOWN`, follow\n     [Path D: No DETECTED analyzer or UNKNOWN cause](references\u002Fpath-d-no-detected-or-unknown.md).\n   - If no `HANG_DETECTED` log or hang monitored event exists and multi-slice\n     latencies are normal in Step 2, **rule out** an MXLA hang by following\n     [Path E: Healthy telemetry](references\u002Fpath-e-healthy-telemetry.md).\n\n---\n\n### Step 2: Correlate with 1-minute multi-slice and TPU metrics `[Low Risk]`\n\nConsult the\n[System Metrics](https:\u002F\u002Fdocs.cloud.google.com\u002Ftpu\u002Fdocs\u002Fml-diagnostics\u002Fworkload-monitoring.md.txt)\nsection of the Workload Monitoring guide for the 1-minute multi-slice network\n(`kubernetes.io\u002Fcontainer\u002Fmultislice\u002Fnetwork\u002F*`), multi-slice accelerator\n(`kubernetes.io\u002Fcontainer\u002Fmultislice\u002Faccelerator\u002F*`), and node duty-cycle\n(`kubernetes.io\u002Fnode\u002Faccelerator\u002Fduty_cycle`) metrics, and run read-only PromQL\nqueries over `[{start_time}, {end_time}]`:\n\n```promql\n# 1. P95 multi-slice collective end-to-end latency by pod\nhistogram_quantile(\n  0.95,\n  sum by (pod_name, le) (\n    rate(kubernetes_io:container_multislice_network_collective_end_to_end_latencies_bucket{\n      monitored_resource=\"k8s_container\",\n      project_id=\"{project_id}\",\n      cluster_name=\"{cluster_name}\"\n    }[5m])\n  )\n)\n\n# 2. P95 multi-slice DCN transfer latency by pod\nhistogram_quantile(\n  0.95,\n  sum by (pod_name, le) (\n    rate(kubernetes_io:container_multislice_network_dcn_transfer_latencies_bucket{\n      monitored_resource=\"k8s_container\",\n      project_id=\"{project_id}\",\n      cluster_name=\"{cluster_name}\"\n    }[5m])\n  )\n)\n\n# 3. P95 host-to-device transfer latency by pod\nhistogram_quantile(\n  0.95,\n  sum by (pod_name, le) (\n    rate(kubernetes_io:container_multislice_accelerator_host_to_device_transfer_latencies_bucket{\n      monitored_resource=\"k8s_container\",\n      project_id=\"{project_id}\",\n      cluster_name=\"{cluster_name}\"\n    }[5m])\n  )\n)\n\n# 4. Node TPU duty cycle (Workload Monitoring detects a hang as a prolonged\n#    period of minimal to no TPU activity)\nkubernetes_io:node_accelerator_duty_cycle{\n  monitored_resource=\"k8s_node\",\n  project_id=\"{project_id}\",\n  cluster_name=\"{cluster_name}\"\n}\n```\n\n---\n\n### Step 3: Map culprit Compute Engine instance IDs to GKE nodes `[Low Risk]`\n\nWhen the `Megascale XLA (MXLA) Hang Analyzer` reports culprit numeric Compute\nEngine instance IDs in `details` or `recommendedActions`, map those numeric IDs\nto GKE `Node` names and physical topology blocks using this read-only `kubectl`\nquery inspecting `container.googleapis.com\u002Finstance_id`:\n\n```bash\nkubectl get nodes -l cloud.google.com\u002Fgke-tpu-accelerator \\\n  -o jsonpath='{range .items[*]}{.metadata.name}{\"\\tinstance_id=\"}{.metadata.annotations.container\\.googleapis\\.com\u002Finstance_id}{\"\\tblock=\"}{.metadata.labels.cloud\\.google\\.com\u002Fgce-topology-block}{\"\\tsubblock=\"}{.metadata.labels.cloud\\.google\\.com\u002Fgce-topology-subblock}{\"\\thost=\"}{.metadata.labels.cloud\\.google\\.com\u002Fgce-topology-host}{\"\\n\"}{end}'\n```\n\n---\n\n### Step 4: Remediation by root-cause category\n\nLoad and follow only the reference file that matches the root-cause category\nfrom Step 1:\n\n- **Path A (compiler or HLO divergence — different HLO modules, inconsistent HLO\n  compilation, or inconsistent launch order across VMs, such as\n  `FINGERPRINT_MISMATCH`)**: Read\n  [Path A: Compiler or HLO divergence](references\u002Fpath-a-compiler-hlo-divergence.md).\n- **Path B (host program queueing or data input stall — `PROGRAM_NOT_QUEUED` or\n  `DATA_INPUT_STALL`)**: Read\n  [Path B: Host program queueing or data input stall](references\u002Fpath-b-program-queueing-input-stall.md).\n- **Path C (hardware, SparseCore, ICI, DCN networking faults, or\n  `UNRECOVERABLE_ERROR` on specific instances)**: Read\n  [Path C: Hardware or network faults](references\u002Fpath-c-hardware-network-faults.md).\n- **Path D (`HANG_DETECTED` logs or hang event fired, but the analyzer is\n  `NOT_DETECTED` or reports `UNKNOWN`)**: Read\n  [Path D: No DETECTED analyzer or UNKNOWN cause](references\u002Fpath-d-no-detected-or-unknown.md).\n- **Path E (no `HANG_DETECTED` logs or hang events, and steady telemetry)**:\n  Read [Path E: Healthy telemetry](references\u002Fpath-e-healthy-telemetry.md).\n\n---\n\n## Guardrails\n\n1. **Never change GKE-managed instance groups or VMs through Compute Engine**:\n   Don't run `gcloud compute instance-groups managed` commands, such as\n   `delete`, on a node pool's managed instance group. Handle nodes through GKE,\n   as described in\n   [Path C: Hardware or network faults](references\u002Fpath-c-hardware-network-faults.md).\n2. **Never cordon or replace nodes for compiler divergence or input stalls**: If\n   the analyzer reports a compiler or HLO divergence, a program queueing issue,\n   or a data input stall, don't cordon or replace TPU nodes. Follow\n   [Path A: Compiler or HLO divergence](references\u002Fpath-a-compiler-hlo-divergence.md)\n   or\n   [Path B: Host program queueing or data input stall](references\u002Fpath-b-program-queueing-input-stall.md)\n   instead.\n",{"name":5,"metadata":46,"description":7},{"version":47,"category":48},"1.0.1","AiAndMachineLearning","main","bb9b3f464aae37af360d079c1d863524955b0363","skills\u002Fcloud\u002Fgke-ai-troubleshooting-tpu-mxla-hang",2005,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills","https:\u002F\u002Fgithub.com\u002Fgoogle","https:\u002F\u002Fopensource.google\u002F","Google ❤️ Open Source","2026-11-14 01:51:35",{"source":8,"skillName":5,"commands":59,"prompt":69,"zipAvailable":70},[60,63,66],{"tool":61,"command":62},"npx","npx skills add google\u002Fskills --skill gke-ai-troubleshooting-tpu-mxla-hang",{"tool":64,"command":65},"bunx","bunx skills add google\u002Fskills --skill gke-ai-troubleshooting-tpu-mxla-hang",{"tool":67,"command":68},"pnpm","pnpm dlx skills add google\u002Fskills --skill gke-ai-troubleshooting-tpu-mxla-hang","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fgke-ai-troubleshooting-tpu-mxla-hang\u002FSKILL.md",true,[72,78,82,86,90,94],{"path":73,"resourceType":74,"resourceTypeDesc":75,"sizeBytes":76,"sourceHtmlUrl":77},"references\u002Fattribution.md","REFERENCE","参考文档",1948,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fgke-ai-troubleshooting-tpu-mxla-hang\u002Freferences\u002Fattribution.md",{"path":79,"resourceType":74,"resourceTypeDesc":75,"sizeBytes":80,"sourceHtmlUrl":81},"references\u002Fpath-a-compiler-hlo-divergence.md",964,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fgke-ai-troubleshooting-tpu-mxla-hang\u002Freferences\u002Fpath-a-compiler-hlo-divergence.md",{"path":83,"resourceType":74,"resourceTypeDesc":75,"sizeBytes":84,"sourceHtmlUrl":85},"references\u002Fpath-b-program-queueing-input-stall.md",1420,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fgke-ai-troubleshooting-tpu-mxla-hang\u002Freferences\u002Fpath-b-program-queueing-input-stall.md",{"path":87,"resourceType":74,"resourceTypeDesc":75,"sizeBytes":88,"sourceHtmlUrl":89},"references\u002Fpath-c-hardware-network-faults.md",2568,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fgke-ai-troubleshooting-tpu-mxla-hang\u002Freferences\u002Fpath-c-hardware-network-faults.md",{"path":91,"resourceType":74,"resourceTypeDesc":75,"sizeBytes":92,"sourceHtmlUrl":93},"references\u002Fpath-d-no-detected-or-unknown.md",1559,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fgke-ai-troubleshooting-tpu-mxla-hang\u002Freferences\u002Fpath-d-no-detected-or-unknown.md",{"path":95,"resourceType":74,"resourceTypeDesc":75,"sizeBytes":96,"sourceHtmlUrl":97},"references\u002Fpath-e-healthy-telemetry.md",272,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fgke-ai-troubleshooting-tpu-mxla-hang\u002Freferences\u002Fpath-e-healthy-telemetry.md",[99,137,168,197,223,248],{"skillNo":100,"name":101,"nameZh":6,"description":102,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":103,"categories":104,"keywords":113,"publishedAt":41},"01M6998S2EZKD3GEJ7F943RZTE","sign-in-with-google-web","Implement, configure, and secure Sign In With Google (SiwG) using Google Identity Services (GIS \u002F `https:\u002F\u002Faccounts.google.com\u002Fgsi\u002Fclient`) across web architectures. Use when creating Google sign-in buttons, implementing Google One Tap with FedCM, integrating GIS in React\u002FNext.js\u002FAngular\u002FHTML, verifying ID tokens on backend runtimes (Python\u002FNode.js\u002FGo\u002FJava), enforcing Google Workspace domain restrictions (`hd`), securing client-side ID tokens via WebCrypto nonces or encrypted IndexedDB, embedding in cross-origin iframes via the Intermediate Iframe API (`https:\u002F\u002Faccounts.google.com\u002Fgsi\u002Fintermediate`), configuring Content Security Policy (CSP), COOP, and Permissions Policy (`identity-credentials-get`) headers, or handling sign-out and revocation. Don't use for legacy Google Sign-In (`gapi.auth2`), Google Cloud IAM service accounts, or Android\u002FiOS native Credential Manager.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fidentity\u002Fsign-in-with-google-web\u002FSKILL.md",[105,110],{"categoryNo":106,"categoryName":107,"taxonomyType":108,"taxonomyTypeDesc":109},"01JBSEEDCAT000000000000009","前端开发","SCENARIO","使用场景",{"categoryNo":111,"categoryName":112,"taxonomyType":108,"taxonomyTypeDesc":109},"01JBSEEDCAT000000000000010","后端与接口",[9,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136],"sign in with google web","sign","with","web","android","go","security","google workspace","google cloud","implement","policy","python","implementing","react","iam","permissions","buttons","secure","manager","identity","architectures","domain","runtimes",{"skillNo":138,"name":139,"nameZh":6,"description":140,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":141,"categories":142,"keywords":152,"publishedAt":41},"01M6998PHPDCAD684VW5WRG83D","dpop-adoption","Implement and debug OAuth 2.0 DPoP (RFC 9449) refresh token sender-constraining for WebCrypto, Node.js ES6, and browser runtimes integrating with Google's OAuth platform. Use when configuring non-extractable asymmetric key pairs (P-256), generating DPoP Proof JWTs for authorization code exchange and token refresh, or handling 400 use_dpop_nonce challenge retry loops at oauth2.googleapis.com\u002Ftoken. Don't use for unconstrained OAuth 2.0 flows (where refresh tokens are not bound to a client key pair), or for Google Cloud IAM \u002F service account authentication.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fidentity\u002Fdpop-adoption\u002FSKILL.md",[143,144,147],{"categoryNo":111,"categoryName":112,"taxonomyType":108,"taxonomyTypeDesc":109},{"categoryNo":145,"categoryName":146,"taxonomyType":108,"taxonomyTypeDesc":109},"01JBSEEDCAT000000000000023","Skill与智能体开发",{"categoryNo":148,"categoryName":149,"taxonomyType":150,"taxonomyTypeDesc":151},"01JBSEEDCAT000000000000030","开发工程师","PROFESSION","适用职业",[153,154,9,122,155,156,157,158,159,160,128,161,162,163,164,165,166,133,136,167,26],"dpop adoption","adoption","es6","retry","authentication","flows","challenge","code","browser","loops","at","node","token","pair","platform",{"skillNo":169,"name":170,"nameZh":6,"description":171,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":172,"categories":173,"keywords":180,"publishedAt":41},"01M6998M72TC73EEAQH4XC7EXW","retrieving-developer-knowledge","Searches, retrieves, and synthesizes official Google developer documentation across Google Cloud, AI\u002FGemini, Android, Chrome, Web, Flutter, Go, Firebase, and other Google developer platforms. Integrates with the Developer Knowledge MCP server (search_documents, get_documents, answer_query) or the Developer Knowledge REST API fallback. Use when searching for gcloud CLI commands, API syntax, IAM permissions, official documentation, architectural comparisons, or product choice overviews. Don't use for local filesystem lookups or non-Google documentation.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fdevelopers\u002Fretrieving-developer-knowledge\u002FSKILL.md",[174,175,178,179],{"categoryNo":111,"categoryName":112,"taxonomyType":108,"taxonomyTypeDesc":109},{"categoryNo":176,"categoryName":177,"taxonomyType":108,"taxonomyTypeDesc":109},"01JBSEEDCAT000000000000021","资料检索与研究",{"categoryNo":145,"categoryName":146,"taxonomyType":108,"taxonomyTypeDesc":109},{"categoryNo":148,"categoryName":149,"taxonomyType":150,"taxonomyTypeDesc":151},[181,182,183,184,122,185,186,187,188,189,190,191,192,26,193,194,195,128,196,129],"retrieving developer knowledge","knowledge","developer","MCP","mcp server","official","developer documentation","get","search","local","choice","documents","query","api","across","documentation",{"skillNo":198,"name":199,"nameZh":6,"description":200,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":201,"categories":202,"keywords":203,"publishedAt":41},"01M6998J2CZEXS9BES9S3WWFKC","migrate-to-rust","Clones, analyzes, plans, and migrates legacy codebases (C++, Fortran, Java, Python, C, Shell) from a GitHub URL or local directory to a modern, high-performance, deterministic, 100% Safe Rust architecture, tracking live telemetry across every task bucket, test failure\u002Ffix cycle, and Profile-Guided Optimization (PGO) stage, and producing a first-person AI storytelling report and interactive HTML infographic. Use when migrating a GitHub repository or local codebase to Rust, porting legacy C++, Fortran, Python, or Java workloads to memory-safe Rust, designing Struct-of-Arrays (SoA) or SIMD Rust architectures, or optimizing Rust ports with LLVM PGO. Don't use for unrelated non-Rust refactoring or single-line syntax questions.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fdevelopers\u002Fmigrate-to-rust\u002FSKILL.md",[],[204,205,206,207,208,39,209,210,125,211,212,213,214,215,216,190,217,218,219,26,220,221,195,222],"migrate to rust","rust","migrate","github","memory","directory","safe","ports","llvm","soa","cycle","porting","fix","arrays","infographic","high-performance","URL","architecture","unrelated",{"skillNo":224,"name":225,"nameZh":6,"description":226,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":227,"categories":228,"keywords":232,"publishedAt":41},"01M6998GCC3K29WZ82KXMB9E02","finding-google-skills","Google platform decision and setup guidance, loaded on demand from Google's skill catalog. Use when a developer is choosing or setting up part of their stack, such as where to run a service, a database, storage, messaging, authentication, analytics, ads, or AI model serving, and a Google product is a reasonable candidate - whether or not a vendor is named - or when a request names a Google product or API. Brings in the matching Google skill so the answer can weigh Google options, their trade-offs, and when they are not the right fit. Skip when the stack is already settled on another provider and no Google product is named, or the task involves no platform choice.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fdevelopers\u002Ffinding-google-skills\u002FSKILL.md",[229,230,231],{"categoryNo":111,"categoryName":112,"taxonomyType":108,"taxonomyTypeDesc":109},{"categoryNo":145,"categoryName":146,"taxonomyType":108,"taxonomyTypeDesc":109},{"categoryNo":148,"categoryName":149,"taxonomyType":150,"taxonomyTypeDesc":151},[9,233,234,235,236,167,237,238,239,240,241,157,242,243,244,191,245,246,26,247,194],"finding google skills","storage","as","messaging","model","trade-offs","provider","answer","fit","catalog","up","decision","use","database","candidate",{"skillNo":249,"name":250,"nameZh":6,"description":251,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":252,"categories":253,"keywords":256,"publishedAt":41},"01M6998DWNHTH0BS6W1CY8ZXMP","workload-manager-basics","Use this skill to manage Google Cloud Workload Manager evaluations, rules, scanned resources, and validation results by using public client libraries and the REST API. Use when you need to inspect workload best-practice rules, create and run evaluations for Google Cloud general best practices, SAP, SQL Server, or custom organizational rules, review violations, export results to BigQuery, or automate Workload Manager through client libraries because no service-specific public CLI or MCP server is available. Don't use for general Google Compute Engine instance management, VPC configuration, or standard IAM auditing.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fworkload-manager-basics\u002FSKILL.md",[254,255],{"categoryNo":111,"categoryName":112,"taxonomyType":108,"taxonomyTypeDesc":109},{"categoryNo":145,"categoryName":146,"taxonomyType":108,"taxonomyTypeDesc":109},[257,258,132,259,9,184,260,122,185,261,262,263,264,26,265,194,128,266,267,268,269],"workload manager basics","workload","basics","cli","review","vpc","resources","inspect","you","create","libraries","server","run"]