[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6993C2VAXF46BT2DRTNXCJ2":3,"skill-same:01M6993C2VAXF46BT2DRTNXCJ2":77},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":40,"linkOnly":41,"markdownContent":42,"markdownBody":43,"frontMatter":44,"refName":48,"commitSha":49,"skillDirectory":50,"forksCount":51,"repositoryHtmlUrl":52,"repositoryHomepageUrl":6,"ownerHtmlUrl":53,"ownerBlogUrl":54,"ownerDescription":55,"updatedAt":56,"install":57,"resources":70},"01M6993C2VAXF46BT2DRTNXCJ2","gke-ai-troubleshooting-tpu-vm-repair-stuck",null,"Diagnose GKE TPU VM instances stuck in Compute Engine repair (REPAIRING state) after host errors, maintenance events, or faulty host reports in single-host, multi-host, dynamic slicing, and TPU All Capacity mode node pools. Use when TPU instances in a slice node pool remain in REPAIRING, host error events appear in audit logs, or TPU slices are unschedulable due to a VM in repair. Don't use for nodes reporting Ready: Unknown when no VM is in REPAIRING (use gke-ai-troubleshooting-node-unresponsive-timeout), other NotReady nodes when no VM is in REPAIRING (use gke-node-notready), Pending Pods when all nodes are Ready (use gke-workload-troubleshooting), JobSet preemption events (use gke-ai-troubleshooting-jobset-interruption), or Slice custom resource lifecycle failures (use gke-ai-troubleshooting-tpu-dynamic-slices-monitoring).","google\u002Fskills","google","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F1342004?v=4","01M698VYEFDM0VJBZDYYJ2T83S","Google","01M698WR0H76ENHE587T9EJT89",24122,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fgke-ai-troubleshooting-tpu-vm-repair-stuck\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39],"gke ai troubleshooting tpu vm repair stuck","tpu","vm","gke","troubleshooting","repair","when","containers","unknown","mode","errors","error","host","instances","reports","ready","node","reporting","monitoring","logs","audit","2026-11-10 10:12:30",false,"---\nname: gke-ai-troubleshooting-tpu-vm-repair-stuck\ndescription: >-\n  Diagnose GKE TPU VM instances stuck in Compute Engine repair (REPAIRING state)\n  after host errors, maintenance events, or faulty host reports in single-host,\n  multi-host, dynamic slicing, and TPU All Capacity mode node pools. Use when\n  TPU instances in a slice node pool remain in REPAIRING, host error events\n  appear in audit logs, or TPU slices are unschedulable due to a VM in repair.\n  Don't use for nodes reporting Ready: Unknown when no VM is in REPAIRING (use\n  gke-ai-troubleshooting-node-unresponsive-timeout), other NotReady nodes when\n  no VM is in REPAIRING (use gke-node-notready), Pending Pods when all nodes are\n  Ready (use gke-workload-troubleshooting), JobSet preemption events (use\n  gke-ai-troubleshooting-jobset-interruption), or Slice custom resource\n  lifecycle failures (use gke-ai-troubleshooting-tpu-dynamic-slices-monitoring).\nmetadata:\n  version: \"1.0.1\"\n  category: Containers\n---\n\n# Troubleshoot GKE TPU VMs stuck in Compute Engine repair (`REPAIRING`)\n\nThis skill checks the VM status, the host events, and the GKE repair status, and\nroutes recovery through GKE instead of direct Compute Engine changes.\n\n## Attribution\n\n**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you\nMUST read [Usage Attribution](references\u002Fattribution.md). For commands executed\ndirectly in the terminal (never in generated scripts, Terraform, or user code),\nprefix `gcloud` with\n`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION=\"agent-skills\u002F1.0 (skill:gke-ai-troubleshooting-tpu-vm-repair-stuck)\"`\nand append `agent-skills\u002F1.0 (skill:gke-ai-troubleshooting-tpu-vm-repair-stuck)` to `User-Agent` headers.\n\n## Prerequisites\n\n- **Tools**:\n  - For installing and initializing `gcloud` (which covers authentication and\n    project selection), follow\n    [Before you begin](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Ftpus.md.txt).\n  - For installing `kubectl` and required plugins, follow\n    [Install kubectl](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcluster-access-for-kubectl.md.txt)\n    and\n    [Install required plugins](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcluster-access-for-kubectl.md.txt).\n- **Cloud Billing & Project Configuration**: Ensure the project has an active\n  Cloud Billing account. Follow\n  [Before you begin](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Ftpus.md.txt)\n  to initialize `gcloud` and enable the Google Kubernetes Engine API\n  (`container.googleapis.com`), ensure the Compute Engine API\n  (`compute.googleapis.com`) and Cloud Logging API (`logging.googleapis.com`)\n  are enabled, and follow\n  [Store cluster information for kubectl](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcluster-access-for-kubectl.md.txt)\n  to configure `kubectl` cluster credentials.\n- **Required IAM Roles**:\n  - Kubernetes Engine Viewer (`roles\u002Fcontainer.viewer`)\n  - Compute Viewer (`roles\u002Fcompute.viewer`)\n  - Logs Viewer (`roles\u002Flogging.viewer`)\n  - For the user who enables node auto-repair in Step 5, item 1: Kubernetes\n    Engine Cluster Admin (`roles\u002Fcontainer.clusterAdmin`)\n- **Documentation**:\n  - [View information about MIGs and managed instances](https:\u002F\u002Fdocs.cloud.google.com\u002Fcompute\u002Fdocs\u002Finstance-groups\u002Fgetting-info-about-migs.md.txt)\n    (Sections: \"List MIGs\", \"Check the status of managed instances\")\n  - [Compute Engine instance lifecycle](https:\u002F\u002Fdocs.cloud.google.com\u002Fcompute\u002Fdocs\u002Finstances\u002Finstance-lifecycle.md.txt)\n    (Sections: \"`REPAIRING` state\")\n  - [Troubleshooting VM shutdowns and reboots](https:\u002F\u002Fdocs.cloud.google.com\u002Fcompute\u002Fdocs\u002Ftroubleshooting\u002Ftroubleshooting-reboots.md.txt)\n    (Sections: \"Querying Cloud Audit Logs\", \"Reviewing Cloud Audit Logs\")\n  - [Configure cluster access for kubectl](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcluster-access-for-kubectl.md.txt)\n    (Sections: \"Install kubectl\", \"Install required plugins\", \"Store cluster\n    information for kubectl\")\n  - [Add and manage node pools](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-pools.md.txt)\n    (Sections: \"View node pools in a Standard cluster\", \"Deploy a Pod to a\n    specific node pool\")\n  - [Troubleshoot NodeNotReady state](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Ftroubleshooting\u002Fnode-notready.md.txt)\n    (Sections: \"Check the node's status and conditions\", \"Confirm node\n    preemption\")\n  - [About TPUs in GKE](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fconcepts\u002Ftpus.md.txt)\n    (Sections: \"Multi-host TPU slice node pools\")\n  - [Use dynamic slicing with a custom scheduler](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt)\n    (Sections: \"Requirements\", \"Verify the status of the nodes and the\n    partitions\", \"Monitor the status of the Slice custom resource\", \"Hardware\n    maintenance and failures\")\n  - [About GKE dynamic slicing](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fconcepts\u002Fdynamic-slicing.md.txt)\n    (Sections: \"Key characteristics of dynamic slicing configurations\")\n  - [Deploy TPUs in All Capacity mode in GKE](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Ftpu-all-capacity-mode.md.txt)\n    (Sections: \"Manage node failures\", \"Node failure and recovery\")\n  - [Report and repair faulty TPU hosts in All Capacity mode](https:\u002F\u002Fdocs.cloud.google.com\u002Ftpu\u002Fdocs\u002Fall-capacity-report-faulty-hosts.md.txt)\n    (Sections: \"Understand the faulty host report process\", \"Review report\n    faulty host operations\")\n  - [Auto-repair nodes](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt)\n    (Sections: \"Settings for Autopilot and Standard\", \"Repair criteria\",\n    \"Node repair process\", \"Verify node auto-repair is enabled for a Standard\n    node pool\", \"Enable auto-repair for an existing Standard node pool\", \"Get\n    information about recent automated repair events\", \"Node auto repair in TPU\n    slice nodes\")\n  - [Deploy TPU workloads in GKE Standard](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Ftpus.md.txt)\n    (Sections: \"Before you begin\", \"Configure auto repair for TPU slice nodes\")\n  - [Get support](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fgetting-support.md.txt)\n    (Sections: \"Before you contact support\", \"Recommendations for efficient\n    support\")\n  - [REST Resource: projects.locations.clusters.nodePools](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Freference\u002Frest\u002Fv1\u002Fprojects.locations.clusters.nodePools.md.txt)\n    (Sections: \"Resource: NodePool\")\n\n> **Read-only rule**: Run read-only diagnostic commands only. Never drain,\n> delete, or re-create nodes, or run any other command that changes the cluster.\n> Give the user any fix to apply themselves.\n\nWhen you recommend a fix, link the doc section that describes it.\n\n---\n\n## Diagnostic workflow\n\n### Step 0: Collect context `[Low Risk]`\n\nCollect the target parameters:\n\n- `{project_id}`: Google Cloud project ID\n- `{cluster_name}`: GKE cluster name\n- `{location}`: Cluster region or zone\n- `{cluster_mode}`: Cluster mode (`Autopilot` or `Standard`)\n- `{tpu_version}`: TPU version (such as Ironwood `tpu7x`, TPU v6e, v5p, v5e, or\n  v4) and whether the node pool is a single-host or multi-host TPU slice node\n  pool\n- `{nodepool_name}`: TPU node pool name\n- `{group_name}` and `{zone}`: Name and zone of the managed instance group (MIG)\n  that backs the node pool:\n  - In a **Standard** cluster, follow\n    [View node pools in a Standard cluster](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-pools.md.txt)\n    to describe `{nodepool_name}` and read its `instanceGroupUrls` field (see\n    [Resource: NodePool](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Freference\u002Frest\u002Fv1\u002Fprojects.locations.clusters.nodePools.md.txt)).\n  - In an **Autopilot** cluster (or if `{nodepool_name}` isn't known), follow\n    [List MIGs](https:\u002F\u002Fdocs.cloud.google.com\u002Fcompute\u002Fdocs\u002Finstance-groups\u002Fgetting-info-about-migs.md.txt)\n    to list the managed instance groups in `{project_id}` and find\n    `{group_name}` and `{zone}`.\n- `{instance_name}`: Affected TPU VM name, if known. If the user doesn't know\n  `{instance_name}`, Step 1 finds the instances in `REPAIRING`.\n- `{issue_time}`: Incident timestamp in RFC3339 UTC, if known\n\n---\n\n### Step 1: Check the VM status in the managed instance group `[Low Risk]`\n\nTo list the instances in `{group_name}` with their status (`STATUS` \u002F\n`instanceStatus`) and current action (`ACTION` \u002F `currentAction`), follow the\ninstructions in\n[Check the status of managed instances](https:\u002F\u002Fdocs.cloud.google.com\u002Fcompute\u002Fdocs\u002Finstance-groups\u002Fgetting-info-about-migs.md.txt).\n\n- **Analysis**:\n  - **`STATUS` (`instanceStatus`) is `REPAIRING`**: Compute Engine is repairing\n    the VM because it encountered an internal error or its host server is\n    unavailable due to maintenance. The VM can't be used during the repair. See\n    [`REPAIRING` state](https:\u002F\u002Fdocs.cloud.google.com\u002Fcompute\u002Fdocs\u002Finstances\u002Finstance-lifecycle.md.txt).\n    Record `{instance_name}` and proceed to Step 2.\n  - **No instance is in `REPAIRING`** (every instance reports `RUNNING` with\n    `ACTION` `NONE`, or an instance reports another status or action such as\n    `ACTION` `RECREATING`, which means \"The instance is being replaced\" per\n    [Check the status of managed instances](https:\u002F\u002Fdocs.cloud.google.com\u002Fcompute\u002Fdocs\u002Finstance-groups\u002Fgetting-info-about-migs.md.txt)):\n    Check each node's status and conditions following\n    [Check the node's status and conditions](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Ftroubleshooting\u002Fnode-notready.md.txt)\n    and hand off to the matching sibling skill:\n    - If a node's `Ready` condition is `Unknown` (`NodeStatusUnknown`), hand off\n      to `gke-ai-troubleshooting-node-unresponsive-timeout`.\n    - If a node isn't `Ready` for another reason, hand off to\n      `gke-node-notready`.\n    - If every node is `Ready` (per\n      [Repair criteria](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt),\n      \"A node reporting a Ready status is considered healthy\") but the\n      workload's Pods are still `Pending`, hand off to\n      `gke-workload-troubleshooting`.\n\n---\n\n### Step 2: Check Compute Engine system events `[Low Risk]`\n\nTo list the system events for `{instance_name}` around `{issue_time}`, follow\nthe instructions in the section\n[Querying Cloud Audit Logs](https:\u002F\u002Fdocs.cloud.google.com\u002Fcompute\u002Fdocs\u002Ftroubleshooting\u002Ftroubleshooting-reboots.md.txt).\nCompare the `method` field with the table in the\n[Reviewing Cloud Audit Logs](https:\u002F\u002Fdocs.cloud.google.com\u002Fcompute\u002Fdocs\u002Ftroubleshooting\u002Ftroubleshooting-reboots.md.txt)\nsection of the same document:\n\n- **`compute.instances.hostError`,\n  `compute.instances.terminateOnHostMaintenance`, or\n  `compute.instances.automaticRestart`**: Record the matching system event log\n  entries and timestamps, and continue to Steps 3 and 4.\n- **`compute.instances.preempted`**: Compute Engine preempted a Spot VM or\n  preemptible VM. If the affected workload is a `JobSet`, hand off to\n  `gke-ai-troubleshooting-jobset-interruption`. Otherwise, follow\n  [Confirm node preemption](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Ftroubleshooting\u002Fnode-notready.md.txt).\n- **No method from the table, or another method**: Per the note in\n  [Reviewing Cloud Audit Logs](https:\u002F\u002Fdocs.cloud.google.com\u002Fcompute\u002Fdocs\u002Ftroubleshooting\u002Ftroubleshooting-reboots.md.txt),\n  if a VM rebooted and none of the listed methods appears in the query results,\n  the reboot likely happened because the VM's operating system initiated it. If\n  the VM is still in `REPAIRING` in Step 1 (or a method such as\n  `compute.instances.reportHostAsFaulty` appears), continue to Steps 3 and 4.\n\n---\n\n### Step 3: Check the node status and slice impact `[Low Risk]`\n\nFollow\n[Check the node's status and conditions](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Ftroubleshooting\u002Fnode-notready.md.txt)\nto check the status and conditions of the nodes in the cluster and match\n`{instance_name}` using the node's `ProviderID` field. To filter nodes to\n`{nodepool_name}`, use the `cloud.google.com\u002Fgke-nodepool` node label documented\nin\n[Deploy a Pod to a specific node pool](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-pools.md.txt).\n\n- If the node for `{instance_name}` reports `Ready` (per\n  [Repair criteria](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt),\n  \"A node reporting a Ready status is considered healthy\") and the VM is no\n  longer in `REPAIRING`, proceed to Step 5, item 4.\n- Otherwise, determine how the unhealthy node affects the node pool based on\n  the node pool configuration:\n  - **Single-host TPU slice node pool**: Per\n    [Configure auto repair for TPU slice nodes](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Ftpus.md.txt),\n    \"In a single-host TPU slice node pool, only the unhealthy TPU node is\n    auto-repaired.\"\n  - **Dynamic slicing node pool**: To check whether `{nodepool_name}` uses\n    dynamic slicing, see\n    [Requirements](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt):\n    dynamic slicing requires a Standard cluster, Ironwood (`tpu7x`), and All\n    Capacity mode reservations. For a Standard cluster with Ironwood (`tpu7x`),\n    check whether the node has `cloud.google.com\u002Fgke-accelerator-topology-mode:\n    PROVISION_ONLY` as shown in step 2 of\n    [Verify the status of the nodes and the partitions](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt)\n    (per\n    [Key characteristics of dynamic slicing configurations](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fconcepts\u002Fdynamic-slicing.md.txt),\n    \"dynamic slicing uses incremental provisioning\"). Per\n    [Hardware maintenance and failures](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt),\n    \"If you trigger customer-initiated maintenance or if a hardware failover\n    occurs, only the targeted nodes are affected rather than the entire node\n    pool.\" For an active dynamic sub-slice, the `Slice` custom resource\n    transitions to `FAILED` and the scheduler attempts to re-create the slice on\n    other available healthy nodes. Check the `Slice` status following\n    [Monitor the status of the Slice custom resource](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt);\n    if the `Slice` custom resource is in `FAILED` or another non-active state,\n    hand off `Slice` troubleshooting to\n    `gke-ai-troubleshooting-tpu-dynamic-slices-monitoring` alongside checking\n    the VM repair in Step 4.\n  - **TPU All Capacity mode node pool**: Per\n    [Manage node failures](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Ftpu-all-capacity-mode.md.txt)\n    and\n    [Node failure and recovery](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Ftpu-all-capacity-mode.md.txt),\n    in TPU All Capacity mode, if a VM fails due to issues like hardware\n    failure, Google Cloud \"attempts to recover the VM on the same host\n    (in-place repair)\", failed nodes have a status of `NotReady`, and the node\n    pool's status doesn't change to `ERROR`.\n  - **Multi-host TPU slice node pool (without dynamic slicing or All Capacity\n    mode)**: Per the section\n    [Multi-host TPU slice node pools](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fconcepts\u002Ftpus.md.txt),\n    the VMs in a multi-host TPU slice node pool are treated as a single atomic\n    unit. When a node in the slice requires repairing, GKE shuts down all VMs in\n    the TPU slice, forcing eviction of all the Kubernetes Pods in the workload,\n    and the Pods can be scheduled again only after all VMs in the slice are up\n    and running.\n\n---\n\n### Step 4: Check node auto-repair status and faulty host reports `[Low Risk]`\n\nFind out whether GKE or Compute Engine is repairing the node:\n\n1. **Check whether node auto-repair is enabled**:\n   - **Autopilot clusters**: Autopilot always repairs nodes, and you can't turn\n     this off (see\n     [Settings for Autopilot and Standard](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt)).\n   - **Standard clusters**: Follow\n     [Verify node auto-repair is enabled for a Standard node pool](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt)\n     to check whether `autoRepair` is enabled on `{nodepool_name}`.\n2. **Check the GKE repair operation history**: Per\n   [Configure auto repair for TPU slice nodes](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Ftpus.md.txt),\n   check the repair status, including any failure reason, in the operation\n   history described in\n   [Get information about recent automated repair events](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt):\n   - If a repair operation failed because of insufficient quota, tell the user\n     to contact their Google Cloud account representative to increase the quota.\n   - If `autoRepair` is enabled and no repair operation has appeared yet, check\n     how long the node has been unhealthy (from `{issue_time}` or the `Ready`\n     condition's `LastTransitionTime` in Step 3). Per\n     [Repair criteria](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt),\n     GKE initiates a repair when a node reports a `NotReady` status, or doesn't\n     report any status at all, over a given time threshold of approximately 10\n     minutes. If the node has reported `NotReady` (or no status) for less than\n     approximately 10 minutes, check the operation history again after that\n     threshold has passed. If it has been longer than approximately 10 minutes\n     and there is still no repair operation (and no All Capacity mode faulty\n     host report in item 3), escalate in Step 5, item 3.\n3. **For TPU All Capacity mode node pools, check for a faulty host report**:\n   Per\n   [Understand the faulty host report process](https:\u002F\u002Fdocs.cloud.google.com\u002Ftpu\u002Fdocs\u002Fall-capacity-report-faulty-hosts.md.txt),\n   after a user reports a faulty host, the TPU VM is stopped, its state changes\n   to `REPAIRING`, and \"The host repair process can take between 3 and 14 days,\n   or longer.\" Before escalating a VM in `REPAIRING` in an All Capacity mode\n   reservation, follow\n   [Review report faulty host operations](https:\u002F\u002Fdocs.cloud.google.com\u002Ftpu\u002Fdocs\u002Fall-capacity-report-faulty-hosts.md.txt)\n   to check for a `compute.instances.reportHostAsFaulty` operation on\n   `{instance_name}`.\n\n---\n\n### Step 5: Resolution `[High Risk]`\n\n> **Guardrails**:\n> - **Never** change GKE-managed instance groups or VMs directly through Compute\n>   Engine, for example with `gcloud compute instances delete` or `gcloud\n>   compute instance-groups managed` commands that change the group. Manage the\n>   node pool through GKE.\n> - **Never** run `kubectl delete node` on nodes in the node pool. Per\n>   [Settings for Autopilot and Standard](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt),\n>   if you use `kubectl` to delete an unhealthy node when auto-repair is\n>   disabled, the underlying VM might become orphaned from the cluster, and you\n>   would continue to be billed for it.\n\n1. **Enable node auto-repair (Standard clusters only)**:\n   - Autopilot clusters always auto-repair nodes, so skip this step.\n   - Before the user enables auto-repair on a multi-host TPU slice node pool\n     that doesn't use dynamic slicing or All Capacity mode, tell them that GKE\n     re-creates the entire node pool when a node in it needs repair (see item 2\n     of this step).\n   - If `autoRepair` is disabled on `{nodepool_name}`, link the user (who needs\n     the `Kubernetes Engine Cluster Admin` role, `roles\u002Fcontainer.clusterAdmin`)\n     to\n     [Enable auto-repair for an existing Standard node pool](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt)\n     and\n     [Configure auto repair for TPU slice nodes](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Ftpus.md.txt)\n     so they can apply the change themselves.\n2. **Let GKE or All Capacity mode repair the node or slice**:\n   - **Single-host TPU slice node pool**: Per\n     [Configure auto repair for TPU slice nodes](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Ftpus.md.txt),\n     \"In a single-host TPU slice node pool, only the unhealthy TPU node is\n     auto-repaired.\" Per\n     [Node repair process](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt),\n     for a single node repair, GKE drains the node and waits one hour for the\n     drain to complete; if the drain doesn't complete, GKE shuts the node down\n     and creates a new node.\n   - **Multi-host TPU slice node pool (without dynamic slicing or All Capacity\n     mode)**: Per\n     [Node auto repair in TPU slice nodes](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt)\n     and\n     [Configure auto repair for TPU slice nodes](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Ftpus.md.txt),\n     if a TPU slice node in a multi-host TPU slice node pool is unhealthy and\n     requires auto repair, the entire node pool is re-created.\n   - **Dynamic slicing node pool**: Per\n     [Hardware maintenance and failures](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt),\n     only the targeted nodes are affected rather than the entire node pool, and\n     for dynamic sub-slicing the scheduler attempts to re-create failed slices\n     on other available healthy nodes. If `Slice` custom resources fail to form\n     or remain `FAILED`, hand off to\n     `gke-ai-troubleshooting-tpu-dynamic-slices-monitoring`.\n   - **TPU All Capacity mode with a faulty host report**: If Step 4, item 3\n     found a `compute.instances.reportHostAsFaulty` operation for\n     `{instance_name}` (tracked via\n     [Review report faulty host operations](https:\u002F\u002Fdocs.cloud.google.com\u002Ftpu\u002Fdocs\u002Fall-capacity-report-faulty-hosts.md.txt)),\n     do not escalate the VM as stuck. Tell the user that per\n     [Understand the faulty host report process](https:\u002F\u002Fdocs.cloud.google.com\u002Ftpu\u002Fdocs\u002Fall-capacity-report-faulty-hosts.md.txt),\n     the VM stops during the repair and restarts on the same host after the\n     repair is complete, and \"The host repair process can take between 3 and 14\n     days, or longer.\"\n3. **Contact Cloud Customer Care if the repair doesn't start or complete**:\n   - If the VM stays in `REPAIRING` (and isn't waiting on a 3-to-14-day All\n     Capacity mode faulty host repair), if the node has been `NotReady` (or\n     reported no status) for longer than the ~10-minute threshold in\n     [Repair criteria](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt)\n     with `autoRepair` enabled and no repair operation in Step 4, or if the\n     operation history shows a failed repair that isn't caused by quota, direct\n     the user to\n     [Before you contact support](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fgetting-support.md.txt)\n     and\n     [Recommendations for efficient support](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fgetting-support.md.txt)\n     to contact Cloud Customer Care. In addition to the details listed in\n     [Recommendations for efficient support](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fgetting-support.md.txt),\n     include the node pool name, instance name, the system event log entries\n     from Step 2, and the repair operation name from Step 4.\n4. **Verify recovery**:\n   - Repeat Step 1 and Step 3. When recovery is complete, every instance in\n     `{group_name}` reports `RUNNING` with action `NONE`, and every node in\n     `{nodepool_name}` is `Ready`.\n   - If the instances report `RUNNING` with action `NONE` (or another status or\n     action) and a node still isn't `Ready`, follow the handoffs in Step 1\n     (`gke-ai-troubleshooting-node-unresponsive-timeout` or\n     `gke-node-notready`) or escalate to Cloud Customer Care in item 3 of this\n     step.\n","# Troubleshoot GKE TPU VMs stuck in Compute Engine repair (`REPAIRING`)\n\nThis skill checks the VM status, the host events, and the GKE repair status, and\nroutes recovery through GKE instead of direct Compute Engine changes.\n\n## Attribution\n\n**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you\nMUST read [Usage Attribution](references\u002Fattribution.md). For commands executed\ndirectly in the terminal (never in generated scripts, Terraform, or user code),\nprefix `gcloud` with\n`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION=\"agent-skills\u002F1.0 (skill:gke-ai-troubleshooting-tpu-vm-repair-stuck)\"`\nand append `agent-skills\u002F1.0 (skill:gke-ai-troubleshooting-tpu-vm-repair-stuck)` to `User-Agent` headers.\n\n## Prerequisites\n\n- **Tools**:\n  - For installing and initializing `gcloud` (which covers authentication and\n    project selection), follow\n    [Before you begin](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Ftpus.md.txt).\n  - For installing `kubectl` and required plugins, follow\n    [Install kubectl](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcluster-access-for-kubectl.md.txt)\n    and\n    [Install required plugins](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcluster-access-for-kubectl.md.txt).\n- **Cloud Billing & Project Configuration**: Ensure the project has an active\n  Cloud Billing account. Follow\n  [Before you begin](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Ftpus.md.txt)\n  to initialize `gcloud` and enable the Google Kubernetes Engine API\n  (`container.googleapis.com`), ensure the Compute Engine API\n  (`compute.googleapis.com`) and Cloud Logging API (`logging.googleapis.com`)\n  are enabled, and follow\n  [Store cluster information for kubectl](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcluster-access-for-kubectl.md.txt)\n  to configure `kubectl` cluster credentials.\n- **Required IAM Roles**:\n  - Kubernetes Engine Viewer (`roles\u002Fcontainer.viewer`)\n  - Compute Viewer (`roles\u002Fcompute.viewer`)\n  - Logs Viewer (`roles\u002Flogging.viewer`)\n  - For the user who enables node auto-repair in Step 5, item 1: Kubernetes\n    Engine Cluster Admin (`roles\u002Fcontainer.clusterAdmin`)\n- **Documentation**:\n  - [View information about MIGs and managed instances](https:\u002F\u002Fdocs.cloud.google.com\u002Fcompute\u002Fdocs\u002Finstance-groups\u002Fgetting-info-about-migs.md.txt)\n    (Sections: \"List MIGs\", \"Check the status of managed instances\")\n  - [Compute Engine instance lifecycle](https:\u002F\u002Fdocs.cloud.google.com\u002Fcompute\u002Fdocs\u002Finstances\u002Finstance-lifecycle.md.txt)\n    (Sections: \"`REPAIRING` state\")\n  - [Troubleshooting VM shutdowns and reboots](https:\u002F\u002Fdocs.cloud.google.com\u002Fcompute\u002Fdocs\u002Ftroubleshooting\u002Ftroubleshooting-reboots.md.txt)\n    (Sections: \"Querying Cloud Audit Logs\", \"Reviewing Cloud Audit Logs\")\n  - [Configure cluster access for kubectl](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcluster-access-for-kubectl.md.txt)\n    (Sections: \"Install kubectl\", \"Install required plugins\", \"Store cluster\n    information for kubectl\")\n  - [Add and manage node pools](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-pools.md.txt)\n    (Sections: \"View node pools in a Standard cluster\", \"Deploy a Pod to a\n    specific node pool\")\n  - [Troubleshoot NodeNotReady state](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Ftroubleshooting\u002Fnode-notready.md.txt)\n    (Sections: \"Check the node's status and conditions\", \"Confirm node\n    preemption\")\n  - [About TPUs in GKE](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fconcepts\u002Ftpus.md.txt)\n    (Sections: \"Multi-host TPU slice node pools\")\n  - [Use dynamic slicing with a custom scheduler](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt)\n    (Sections: \"Requirements\", \"Verify the status of the nodes and the\n    partitions\", \"Monitor the status of the Slice custom resource\", \"Hardware\n    maintenance and failures\")\n  - [About GKE dynamic slicing](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fconcepts\u002Fdynamic-slicing.md.txt)\n    (Sections: \"Key characteristics of dynamic slicing configurations\")\n  - [Deploy TPUs in All Capacity mode in GKE](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Ftpu-all-capacity-mode.md.txt)\n    (Sections: \"Manage node failures\", \"Node failure and recovery\")\n  - [Report and repair faulty TPU hosts in All Capacity mode](https:\u002F\u002Fdocs.cloud.google.com\u002Ftpu\u002Fdocs\u002Fall-capacity-report-faulty-hosts.md.txt)\n    (Sections: \"Understand the faulty host report process\", \"Review report\n    faulty host operations\")\n  - [Auto-repair nodes](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt)\n    (Sections: \"Settings for Autopilot and Standard\", \"Repair criteria\",\n    \"Node repair process\", \"Verify node auto-repair is enabled for a Standard\n    node pool\", \"Enable auto-repair for an existing Standard node pool\", \"Get\n    information about recent automated repair events\", \"Node auto repair in TPU\n    slice nodes\")\n  - [Deploy TPU workloads in GKE Standard](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Ftpus.md.txt)\n    (Sections: \"Before you begin\", \"Configure auto repair for TPU slice nodes\")\n  - [Get support](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fgetting-support.md.txt)\n    (Sections: \"Before you contact support\", \"Recommendations for efficient\n    support\")\n  - [REST Resource: projects.locations.clusters.nodePools](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Freference\u002Frest\u002Fv1\u002Fprojects.locations.clusters.nodePools.md.txt)\n    (Sections: \"Resource: NodePool\")\n\n> **Read-only rule**: Run read-only diagnostic commands only. Never drain,\n> delete, or re-create nodes, or run any other command that changes the cluster.\n> Give the user any fix to apply themselves.\n\nWhen you recommend a fix, link the doc section that describes it.\n\n---\n\n## Diagnostic workflow\n\n### Step 0: Collect context `[Low Risk]`\n\nCollect the target parameters:\n\n- `{project_id}`: Google Cloud project ID\n- `{cluster_name}`: GKE cluster name\n- `{location}`: Cluster region or zone\n- `{cluster_mode}`: Cluster mode (`Autopilot` or `Standard`)\n- `{tpu_version}`: TPU version (such as Ironwood `tpu7x`, TPU v6e, v5p, v5e, or\n  v4) and whether the node pool is a single-host or multi-host TPU slice node\n  pool\n- `{nodepool_name}`: TPU node pool name\n- `{group_name}` and `{zone}`: Name and zone of the managed instance group (MIG)\n  that backs the node pool:\n  - In a **Standard** cluster, follow\n    [View node pools in a Standard cluster](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-pools.md.txt)\n    to describe `{nodepool_name}` and read its `instanceGroupUrls` field (see\n    [Resource: NodePool](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Freference\u002Frest\u002Fv1\u002Fprojects.locations.clusters.nodePools.md.txt)).\n  - In an **Autopilot** cluster (or if `{nodepool_name}` isn't known), follow\n    [List MIGs](https:\u002F\u002Fdocs.cloud.google.com\u002Fcompute\u002Fdocs\u002Finstance-groups\u002Fgetting-info-about-migs.md.txt)\n    to list the managed instance groups in `{project_id}` and find\n    `{group_name}` and `{zone}`.\n- `{instance_name}`: Affected TPU VM name, if known. If the user doesn't know\n  `{instance_name}`, Step 1 finds the instances in `REPAIRING`.\n- `{issue_time}`: Incident timestamp in RFC3339 UTC, if known\n\n---\n\n### Step 1: Check the VM status in the managed instance group `[Low Risk]`\n\nTo list the instances in `{group_name}` with their status (`STATUS` \u002F\n`instanceStatus`) and current action (`ACTION` \u002F `currentAction`), follow the\ninstructions in\n[Check the status of managed instances](https:\u002F\u002Fdocs.cloud.google.com\u002Fcompute\u002Fdocs\u002Finstance-groups\u002Fgetting-info-about-migs.md.txt).\n\n- **Analysis**:\n  - **`STATUS` (`instanceStatus`) is `REPAIRING`**: Compute Engine is repairing\n    the VM because it encountered an internal error or its host server is\n    unavailable due to maintenance. The VM can't be used during the repair. See\n    [`REPAIRING` state](https:\u002F\u002Fdocs.cloud.google.com\u002Fcompute\u002Fdocs\u002Finstances\u002Finstance-lifecycle.md.txt).\n    Record `{instance_name}` and proceed to Step 2.\n  - **No instance is in `REPAIRING`** (every instance reports `RUNNING` with\n    `ACTION` `NONE`, or an instance reports another status or action such as\n    `ACTION` `RECREATING`, which means \"The instance is being replaced\" per\n    [Check the status of managed instances](https:\u002F\u002Fdocs.cloud.google.com\u002Fcompute\u002Fdocs\u002Finstance-groups\u002Fgetting-info-about-migs.md.txt)):\n    Check each node's status and conditions following\n    [Check the node's status and conditions](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Ftroubleshooting\u002Fnode-notready.md.txt)\n    and hand off to the matching sibling skill:\n    - If a node's `Ready` condition is `Unknown` (`NodeStatusUnknown`), hand off\n      to `gke-ai-troubleshooting-node-unresponsive-timeout`.\n    - If a node isn't `Ready` for another reason, hand off to\n      `gke-node-notready`.\n    - If every node is `Ready` (per\n      [Repair criteria](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt),\n      \"A node reporting a Ready status is considered healthy\") but the\n      workload's Pods are still `Pending`, hand off to\n      `gke-workload-troubleshooting`.\n\n---\n\n### Step 2: Check Compute Engine system events `[Low Risk]`\n\nTo list the system events for `{instance_name}` around `{issue_time}`, follow\nthe instructions in the section\n[Querying Cloud Audit Logs](https:\u002F\u002Fdocs.cloud.google.com\u002Fcompute\u002Fdocs\u002Ftroubleshooting\u002Ftroubleshooting-reboots.md.txt).\nCompare the `method` field with the table in the\n[Reviewing Cloud Audit Logs](https:\u002F\u002Fdocs.cloud.google.com\u002Fcompute\u002Fdocs\u002Ftroubleshooting\u002Ftroubleshooting-reboots.md.txt)\nsection of the same document:\n\n- **`compute.instances.hostError`,\n  `compute.instances.terminateOnHostMaintenance`, or\n  `compute.instances.automaticRestart`**: Record the matching system event log\n  entries and timestamps, and continue to Steps 3 and 4.\n- **`compute.instances.preempted`**: Compute Engine preempted a Spot VM or\n  preemptible VM. If the affected workload is a `JobSet`, hand off to\n  `gke-ai-troubleshooting-jobset-interruption`. Otherwise, follow\n  [Confirm node preemption](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Ftroubleshooting\u002Fnode-notready.md.txt).\n- **No method from the table, or another method**: Per the note in\n  [Reviewing Cloud Audit Logs](https:\u002F\u002Fdocs.cloud.google.com\u002Fcompute\u002Fdocs\u002Ftroubleshooting\u002Ftroubleshooting-reboots.md.txt),\n  if a VM rebooted and none of the listed methods appears in the query results,\n  the reboot likely happened because the VM's operating system initiated it. If\n  the VM is still in `REPAIRING` in Step 1 (or a method such as\n  `compute.instances.reportHostAsFaulty` appears), continue to Steps 3 and 4.\n\n---\n\n### Step 3: Check the node status and slice impact `[Low Risk]`\n\nFollow\n[Check the node's status and conditions](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Ftroubleshooting\u002Fnode-notready.md.txt)\nto check the status and conditions of the nodes in the cluster and match\n`{instance_name}` using the node's `ProviderID` field. To filter nodes to\n`{nodepool_name}`, use the `cloud.google.com\u002Fgke-nodepool` node label documented\nin\n[Deploy a Pod to a specific node pool](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-pools.md.txt).\n\n- If the node for `{instance_name}` reports `Ready` (per\n  [Repair criteria](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt),\n  \"A node reporting a Ready status is considered healthy\") and the VM is no\n  longer in `REPAIRING`, proceed to Step 5, item 4.\n- Otherwise, determine how the unhealthy node affects the node pool based on\n  the node pool configuration:\n  - **Single-host TPU slice node pool**: Per\n    [Configure auto repair for TPU slice nodes](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Ftpus.md.txt),\n    \"In a single-host TPU slice node pool, only the unhealthy TPU node is\n    auto-repaired.\"\n  - **Dynamic slicing node pool**: To check whether `{nodepool_name}` uses\n    dynamic slicing, see\n    [Requirements](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt):\n    dynamic slicing requires a Standard cluster, Ironwood (`tpu7x`), and All\n    Capacity mode reservations. For a Standard cluster with Ironwood (`tpu7x`),\n    check whether the node has `cloud.google.com\u002Fgke-accelerator-topology-mode:\n    PROVISION_ONLY` as shown in step 2 of\n    [Verify the status of the nodes and the partitions](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt)\n    (per\n    [Key characteristics of dynamic slicing configurations](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fconcepts\u002Fdynamic-slicing.md.txt),\n    \"dynamic slicing uses incremental provisioning\"). Per\n    [Hardware maintenance and failures](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt),\n    \"If you trigger customer-initiated maintenance or if a hardware failover\n    occurs, only the targeted nodes are affected rather than the entire node\n    pool.\" For an active dynamic sub-slice, the `Slice` custom resource\n    transitions to `FAILED` and the scheduler attempts to re-create the slice on\n    other available healthy nodes. Check the `Slice` status following\n    [Monitor the status of the Slice custom resource](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt);\n    if the `Slice` custom resource is in `FAILED` or another non-active state,\n    hand off `Slice` troubleshooting to\n    `gke-ai-troubleshooting-tpu-dynamic-slices-monitoring` alongside checking\n    the VM repair in Step 4.\n  - **TPU All Capacity mode node pool**: Per\n    [Manage node failures](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Ftpu-all-capacity-mode.md.txt)\n    and\n    [Node failure and recovery](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Ftpu-all-capacity-mode.md.txt),\n    in TPU All Capacity mode, if a VM fails due to issues like hardware\n    failure, Google Cloud \"attempts to recover the VM on the same host\n    (in-place repair)\", failed nodes have a status of `NotReady`, and the node\n    pool's status doesn't change to `ERROR`.\n  - **Multi-host TPU slice node pool (without dynamic slicing or All Capacity\n    mode)**: Per the section\n    [Multi-host TPU slice node pools](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fconcepts\u002Ftpus.md.txt),\n    the VMs in a multi-host TPU slice node pool are treated as a single atomic\n    unit. When a node in the slice requires repairing, GKE shuts down all VMs in\n    the TPU slice, forcing eviction of all the Kubernetes Pods in the workload,\n    and the Pods can be scheduled again only after all VMs in the slice are up\n    and running.\n\n---\n\n### Step 4: Check node auto-repair status and faulty host reports `[Low Risk]`\n\nFind out whether GKE or Compute Engine is repairing the node:\n\n1. **Check whether node auto-repair is enabled**:\n   - **Autopilot clusters**: Autopilot always repairs nodes, and you can't turn\n     this off (see\n     [Settings for Autopilot and Standard](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt)).\n   - **Standard clusters**: Follow\n     [Verify node auto-repair is enabled for a Standard node pool](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt)\n     to check whether `autoRepair` is enabled on `{nodepool_name}`.\n2. **Check the GKE repair operation history**: Per\n   [Configure auto repair for TPU slice nodes](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Ftpus.md.txt),\n   check the repair status, including any failure reason, in the operation\n   history described in\n   [Get information about recent automated repair events](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt):\n   - If a repair operation failed because of insufficient quota, tell the user\n     to contact their Google Cloud account representative to increase the quota.\n   - If `autoRepair` is enabled and no repair operation has appeared yet, check\n     how long the node has been unhealthy (from `{issue_time}` or the `Ready`\n     condition's `LastTransitionTime` in Step 3). Per\n     [Repair criteria](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt),\n     GKE initiates a repair when a node reports a `NotReady` status, or doesn't\n     report any status at all, over a given time threshold of approximately 10\n     minutes. If the node has reported `NotReady` (or no status) for less than\n     approximately 10 minutes, check the operation history again after that\n     threshold has passed. If it has been longer than approximately 10 minutes\n     and there is still no repair operation (and no All Capacity mode faulty\n     host report in item 3), escalate in Step 5, item 3.\n3. **For TPU All Capacity mode node pools, check for a faulty host report**:\n   Per\n   [Understand the faulty host report process](https:\u002F\u002Fdocs.cloud.google.com\u002Ftpu\u002Fdocs\u002Fall-capacity-report-faulty-hosts.md.txt),\n   after a user reports a faulty host, the TPU VM is stopped, its state changes\n   to `REPAIRING`, and \"The host repair process can take between 3 and 14 days,\n   or longer.\" Before escalating a VM in `REPAIRING` in an All Capacity mode\n   reservation, follow\n   [Review report faulty host operations](https:\u002F\u002Fdocs.cloud.google.com\u002Ftpu\u002Fdocs\u002Fall-capacity-report-faulty-hosts.md.txt)\n   to check for a `compute.instances.reportHostAsFaulty` operation on\n   `{instance_name}`.\n\n---\n\n### Step 5: Resolution `[High Risk]`\n\n> **Guardrails**:\n> - **Never** change GKE-managed instance groups or VMs directly through Compute\n>   Engine, for example with `gcloud compute instances delete` or `gcloud\n>   compute instance-groups managed` commands that change the group. Manage the\n>   node pool through GKE.\n> - **Never** run `kubectl delete node` on nodes in the node pool. Per\n>   [Settings for Autopilot and Standard](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt),\n>   if you use `kubectl` to delete an unhealthy node when auto-repair is\n>   disabled, the underlying VM might become orphaned from the cluster, and you\n>   would continue to be billed for it.\n\n1. **Enable node auto-repair (Standard clusters only)**:\n   - Autopilot clusters always auto-repair nodes, so skip this step.\n   - Before the user enables auto-repair on a multi-host TPU slice node pool\n     that doesn't use dynamic slicing or All Capacity mode, tell them that GKE\n     re-creates the entire node pool when a node in it needs repair (see item 2\n     of this step).\n   - If `autoRepair` is disabled on `{nodepool_name}`, link the user (who needs\n     the `Kubernetes Engine Cluster Admin` role, `roles\u002Fcontainer.clusterAdmin`)\n     to\n     [Enable auto-repair for an existing Standard node pool](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt)\n     and\n     [Configure auto repair for TPU slice nodes](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Ftpus.md.txt)\n     so they can apply the change themselves.\n2. **Let GKE or All Capacity mode repair the node or slice**:\n   - **Single-host TPU slice node pool**: Per\n     [Configure auto repair for TPU slice nodes](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Ftpus.md.txt),\n     \"In a single-host TPU slice node pool, only the unhealthy TPU node is\n     auto-repaired.\" Per\n     [Node repair process](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt),\n     for a single node repair, GKE drains the node and waits one hour for the\n     drain to complete; if the drain doesn't complete, GKE shuts the node down\n     and creates a new node.\n   - **Multi-host TPU slice node pool (without dynamic slicing or All Capacity\n     mode)**: Per\n     [Node auto repair in TPU slice nodes](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt)\n     and\n     [Configure auto repair for TPU slice nodes](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Ftpus.md.txt),\n     if a TPU slice node in a multi-host TPU slice node pool is unhealthy and\n     requires auto repair, the entire node pool is re-created.\n   - **Dynamic slicing node pool**: Per\n     [Hardware maintenance and failures](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt),\n     only the targeted nodes are affected rather than the entire node pool, and\n     for dynamic sub-slicing the scheduler attempts to re-create failed slices\n     on other available healthy nodes. If `Slice` custom resources fail to form\n     or remain `FAILED`, hand off to\n     `gke-ai-troubleshooting-tpu-dynamic-slices-monitoring`.\n   - **TPU All Capacity mode with a faulty host report**: If Step 4, item 3\n     found a `compute.instances.reportHostAsFaulty` operation for\n     `{instance_name}` (tracked via\n     [Review report faulty host operations](https:\u002F\u002Fdocs.cloud.google.com\u002Ftpu\u002Fdocs\u002Fall-capacity-report-faulty-hosts.md.txt)),\n     do not escalate the VM as stuck. Tell the user that per\n     [Understand the faulty host report process](https:\u002F\u002Fdocs.cloud.google.com\u002Ftpu\u002Fdocs\u002Fall-capacity-report-faulty-hosts.md.txt),\n     the VM stops during the repair and restarts on the same host after the\n     repair is complete, and \"The host repair process can take between 3 and 14\n     days, or longer.\"\n3. **Contact Cloud Customer Care if the repair doesn't start or complete**:\n   - If the VM stays in `REPAIRING` (and isn't waiting on a 3-to-14-day All\n     Capacity mode faulty host repair), if the node has been `NotReady` (or\n     reported no status) for longer than the ~10-minute threshold in\n     [Repair criteria](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt)\n     with `autoRepair` enabled and no repair operation in Step 4, or if the\n     operation history shows a failed repair that isn't caused by quota, direct\n     the user to\n     [Before you contact support](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fgetting-support.md.txt)\n     and\n     [Recommendations for efficient support](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fgetting-support.md.txt)\n     to contact Cloud Customer Care. In addition to the details listed in\n     [Recommendations for efficient support](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fgetting-support.md.txt),\n     include the node pool name, instance name, the system event log entries\n     from Step 2, and the repair operation name from Step 4.\n4. **Verify recovery**:\n   - Repeat Step 1 and Step 3. When recovery is complete, every instance in\n     `{group_name}` reports `RUNNING` with action `NONE`, and every node in\n     `{nodepool_name}` is `Ready`.\n   - If the instances report `RUNNING` with action `NONE` (or another status or\n     action) and a node still isn't `Ready`, follow the handoffs in Step 1\n     (`gke-ai-troubleshooting-node-unresponsive-timeout` or\n     `gke-node-notready`) or escalate to Cloud Customer Care in item 3 of this\n     step.\n",{"name":5,"metadata":45,"description":7},{"version":46,"category":47},"1.0.1","Containers","main","bb9b3f464aae37af360d079c1d863524955b0363","skills\u002Fcloud\u002Fgke-ai-troubleshooting-tpu-vm-repair-stuck",2005,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills","https:\u002F\u002Fgithub.com\u002Fgoogle","https:\u002F\u002Fopensource.google\u002F","Google ❤️ Open Source","2026-11-14 01:51:35",{"source":8,"skillName":5,"commands":58,"prompt":68,"zipAvailable":69},[59,62,65],{"tool":60,"command":61},"npx","npx skills add google\u002Fskills --skill gke-ai-troubleshooting-tpu-vm-repair-stuck",{"tool":63,"command":64},"bunx","bunx skills add google\u002Fskills --skill gke-ai-troubleshooting-tpu-vm-repair-stuck",{"tool":66,"command":67},"pnpm","pnpm dlx skills add google\u002Fskills --skill gke-ai-troubleshooting-tpu-vm-repair-stuck","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fgke-ai-troubleshooting-tpu-vm-repair-stuck\u002FSKILL.md",true,[71],{"path":72,"resourceType":73,"resourceTypeDesc":74,"sizeBytes":75,"sourceHtmlUrl":76},"references\u002Fattribution.md","REFERENCE","参考文档",1970,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fgke-ai-troubleshooting-tpu-vm-repair-stuck\u002Freferences\u002Fattribution.md",[78,116,146,175,202,227],{"skillNo":79,"name":80,"nameZh":6,"description":81,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":82,"categories":83,"keywords":92,"publishedAt":40},"01M6998S2EZKD3GEJ7F943RZTE","sign-in-with-google-web","Implement, configure, and secure Sign In With Google (SiwG) using Google Identity Services (GIS \u002F `https:\u002F\u002Faccounts.google.com\u002Fgsi\u002Fclient`) across web architectures. Use when creating Google sign-in buttons, implementing Google One Tap with FedCM, integrating GIS in React\u002FNext.js\u002FAngular\u002FHTML, verifying ID tokens on backend runtimes (Python\u002FNode.js\u002FGo\u002FJava), enforcing Google Workspace domain restrictions (`hd`), securing client-side ID tokens via WebCrypto nonces or encrypted IndexedDB, embedding in cross-origin iframes via the Intermediate Iframe API (`https:\u002F\u002Faccounts.google.com\u002Fgsi\u002Fintermediate`), configuring Content Security Policy (CSP), COOP, and Permissions Policy (`identity-credentials-get`) headers, or handling sign-out and revocation. Don't use for legacy Google Sign-In (`gapi.auth2`), Google Cloud IAM service accounts, or Android\u002FiOS native Credential Manager.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fidentity\u002Fsign-in-with-google-web\u002FSKILL.md",[84,89],{"categoryNo":85,"categoryName":86,"taxonomyType":87,"taxonomyTypeDesc":88},"01JBSEEDCAT000000000000009","前端开发","SCENARIO","使用场景",{"categoryNo":90,"categoryName":91,"taxonomyType":87,"taxonomyTypeDesc":88},"01JBSEEDCAT000000000000010","后端与接口",[9,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115],"sign in with google web","sign","with","web","android","go","security","google workspace","google cloud","implement","policy","python","implementing","react","iam","permissions","buttons","secure","manager","identity","architectures","domain","runtimes",{"skillNo":117,"name":118,"nameZh":6,"description":119,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":120,"categories":121,"keywords":131,"publishedAt":40},"01M6998PHPDCAD684VW5WRG83D","dpop-adoption","Implement and debug OAuth 2.0 DPoP (RFC 9449) refresh token sender-constraining for WebCrypto, Node.js ES6, and browser runtimes integrating with Google's OAuth platform. Use when configuring non-extractable asymmetric key pairs (P-256), generating DPoP Proof JWTs for authorization code exchange and token refresh, or handling 400 use_dpop_nonce challenge retry loops at oauth2.googleapis.com\u002Ftoken. Don't use for unconstrained OAuth 2.0 flows (where refresh tokens are not bound to a client key pair), or for Google Cloud IAM \u002F service account authentication.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fidentity\u002Fdpop-adoption\u002FSKILL.md",[122,123,126],{"categoryNo":90,"categoryName":91,"taxonomyType":87,"taxonomyTypeDesc":88},{"categoryNo":124,"categoryName":125,"taxonomyType":87,"taxonomyTypeDesc":88},"01JBSEEDCAT000000000000023","Skill与智能体开发",{"categoryNo":127,"categoryName":128,"taxonomyType":129,"taxonomyTypeDesc":130},"01JBSEEDCAT000000000000030","开发工程师","PROFESSION","适用职业",[132,133,9,101,134,135,136,137,138,139,107,140,141,142,35,143,144,112,115,145,25],"dpop adoption","adoption","es6","retry","authentication","flows","challenge","code","browser","loops","at","token","pair","platform",{"skillNo":147,"name":148,"nameZh":6,"description":149,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":150,"categories":151,"keywords":158,"publishedAt":40},"01M6998M72TC73EEAQH4XC7EXW","retrieving-developer-knowledge","Searches, retrieves, and synthesizes official Google developer documentation across Google Cloud, AI\u002FGemini, Android, Chrome, Web, Flutter, Go, Firebase, and other Google developer platforms. Integrates with the Developer Knowledge MCP server (search_documents, get_documents, answer_query) or the Developer Knowledge REST API fallback. Use when searching for gcloud CLI commands, API syntax, IAM permissions, official documentation, architectural comparisons, or product choice overviews. Don't use for local filesystem lookups or non-Google documentation.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fdevelopers\u002Fretrieving-developer-knowledge\u002FSKILL.md",[152,153,156,157],{"categoryNo":90,"categoryName":91,"taxonomyType":87,"taxonomyTypeDesc":88},{"categoryNo":154,"categoryName":155,"taxonomyType":87,"taxonomyTypeDesc":88},"01JBSEEDCAT000000000000021","资料检索与研究",{"categoryNo":124,"categoryName":125,"taxonomyType":87,"taxonomyTypeDesc":88},{"categoryNo":127,"categoryName":128,"taxonomyType":129,"taxonomyTypeDesc":130},[159,160,161,162,101,163,164,165,166,167,168,169,170,25,171,172,173,107,174,108],"retrieving developer knowledge","knowledge","developer","MCP","mcp server","official","developer documentation","get","search","local","choice","documents","query","api","across","documentation",{"skillNo":176,"name":177,"nameZh":6,"description":178,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":179,"categories":180,"keywords":181,"publishedAt":40},"01M6998J2CZEXS9BES9S3WWFKC","migrate-to-rust","Clones, analyzes, plans, and migrates legacy codebases (C++, Fortran, Java, Python, C, Shell) from a GitHub URL or local directory to a modern, high-performance, deterministic, 100% Safe Rust architecture, tracking live telemetry across every task bucket, test failure\u002Ffix cycle, and Profile-Guided Optimization (PGO) stage, and producing a first-person AI storytelling report and interactive HTML infographic. Use when migrating a GitHub repository or local codebase to Rust, porting legacy C++, Fortran, Python, or Java workloads to memory-safe Rust, designing Struct-of-Arrays (SoA) or SIMD Rust architectures, or optimizing Rust ports with LLVM PGO. Don't use for unrelated non-Rust refactoring or single-line syntax questions.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fdevelopers\u002Fmigrate-to-rust\u002FSKILL.md",[],[182,183,184,185,186,187,188,189,104,190,191,192,193,194,195,168,196,197,198,25,199,200,173,201],"migrate to rust","rust","migrate","github","memory","performance","directory","safe","ports","llvm","soa","cycle","porting","fix","arrays","infographic","high-performance","URL","architecture","unrelated",{"skillNo":203,"name":204,"nameZh":6,"description":205,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":206,"categories":207,"keywords":211,"publishedAt":40},"01M6998GCC3K29WZ82KXMB9E02","finding-google-skills","Google platform decision and setup guidance, loaded on demand from Google's skill catalog. Use when a developer is choosing or setting up part of their stack, such as where to run a service, a database, storage, messaging, authentication, analytics, ads, or AI model serving, and a Google product is a reasonable candidate - whether or not a vendor is named - or when a request names a Google product or API. Brings in the matching Google skill so the answer can weigh Google options, their trade-offs, and when they are not the right fit. Skip when the stack is already settled on another provider and no Google product is named, or the task involves no platform choice.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fdevelopers\u002Ffinding-google-skills\u002FSKILL.md",[208,209,210],{"categoryNo":90,"categoryName":91,"taxonomyType":87,"taxonomyTypeDesc":88},{"categoryNo":124,"categoryName":125,"taxonomyType":87,"taxonomyTypeDesc":88},{"categoryNo":127,"categoryName":128,"taxonomyType":129,"taxonomyTypeDesc":130},[9,212,213,214,215,145,216,217,218,219,220,136,221,222,223,169,224,225,25,226,172],"finding google skills","storage","as","messaging","model","trade-offs","provider","answer","fit","catalog","up","decision","use","database","candidate",{"skillNo":228,"name":229,"nameZh":6,"description":230,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":231,"categories":232,"keywords":235,"publishedAt":40},"01M6998DWNHTH0BS6W1CY8ZXMP","workload-manager-basics","Use this skill to manage Google Cloud Workload Manager evaluations, rules, scanned resources, and validation results by using public client libraries and the REST API. Use when you need to inspect workload best-practice rules, create and run evaluations for Google Cloud general best practices, SAP, SQL Server, or custom organizational rules, review violations, export results to BigQuery, or automate Workload Manager through client libraries because no service-specific public CLI or MCP server is available. Don't use for general Google Compute Engine instance management, VPC configuration, or standard IAM auditing.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fworkload-manager-basics\u002FSKILL.md",[233,234],{"categoryNo":90,"categoryName":91,"taxonomyType":87,"taxonomyTypeDesc":88},{"categoryNo":124,"categoryName":125,"taxonomyType":87,"taxonomyTypeDesc":88},[236,237,111,238,9,162,239,101,163,240,241,242,243,25,244,172,107,245,246,247,248],"workload manager basics","workload","basics","cli","review","vpc","resources","inspect","you","create","libraries","server","run"]