[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6992WJD7EMBMR6AAFZS8FTM":3,"skill-same:01M6992WJD7EMBMR6AAFZS8FTM":76},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":47,"commitSha":48,"skillDirectory":49,"forksCount":50,"repositoryHtmlUrl":51,"repositoryHomepageUrl":6,"ownerHtmlUrl":52,"ownerBlogUrl":53,"ownerDescription":54,"updatedAt":55,"install":56,"resources":69},"01M6992WJD7EMBMR6AAFZS8FTM","gke-ai-troubleshooting-node-unresponsive-timeout",null,"Diagnose and mitigate GKE TPU or GPU nodes stuck in NotReady \u002F NodeStatusUnknown (\"Kubelet stopped posting node status\") due to host kernel panics, hardware lockups, or disabled node auto-repair. Use when nodes stop heartbeating beyond the node auto-repair threshold and pods remain stuck in Terminating. Don't use for healthy nodes, pod-only application crashes, or routine GKE upgrades.","google\u002Fskills","google","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F1342004?v=4","01M698VYEFDM0VJBZDYYJ2T83S","Google","01M698WR0H76ENHE587T9EJT89",24122,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fgke-ai-troubleshooting-node-unresponsive-timeout\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"gke ai troubleshooting node unresponsive timeout","timeout","gke","troubleshooting","node","auto","when","routine","for","host","posting","repair","stop","only","containers","diagnose","status","tpu","gke upgrades","kernel","2026-11-10 10:12:30",false,"---\nname: gke-ai-troubleshooting-node-unresponsive-timeout\ndescription: >-\n  Diagnose and mitigate GKE TPU or GPU nodes stuck in NotReady \u002F\n  NodeStatusUnknown (\"Kubelet stopped posting node status\") due to host kernel\n  panics, hardware lockups, or disabled node auto-repair. Use when nodes stop\n  heartbeating beyond the node auto-repair threshold and pods remain stuck in\n  Terminating. Don't use for healthy nodes, pod-only application crashes, or\n  routine GKE upgrades.\nmetadata:\n  version: \"1.0.1\"\n  category: Containers\n---\n\n# Troubleshoot unresponsive GKE TPU and GPU nodes (`NodeStatusUnknown`)\n\nWhen the Compute Engine host of a TPU or GPU node has a fatal hardware error,\nkernel panic, or non-maskable interrupt (NMI) lockup, the guest OS stops\nresponding. The kubelet can no longer send heartbeats, so the node `Ready`\ncondition becomes `Unknown` with `Reason: NodeStatusUnknown` (`Kubelet stopped\nposting node status.`). If node auto-repair is disabled on the node pool, GKE\ndoesn't repair the node. The node can stay `NotReady`, and pods on it can stay\nin `Terminating`, which blocks multi-host `JobSet` workloads from recovering.\n\n## Attribution\n\n**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you\nMUST read [Usage Attribution](references\u002Fattribution.md). For commands executed\ndirectly in the terminal (never in generated scripts, Terraform, or user code),\nprefix `gcloud` with\n`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION=\"agent-skills\u002F1.0 (skill:gke-ai-troubleshooting-node-unresponsive-timeout)\"`\nand append `agent-skills\u002F1.0 (skill:gke-ai-troubleshooting-node-unresponsive-timeout)` to `User-Agent` headers.\n\n## Prerequisites\n\n- **Tools**: Install the\n  [Google Cloud SDK](https:\u002F\u002Fcloud.google.com\u002Fsdk\u002Fdocs\u002Finstall) (`gcloud`) and\n  `kubectl`.\n- **Cloud Billing & Project Configuration**: Verify an active billing account is\n  linked (`gcloud billing projects describe {project_id}`), authenticate\n  (`gcloud auth login`), set the target project (`gcloud config set project\n  {project_id}`), and ensure `container.googleapis.com`,\n  `compute.googleapis.com`, `logging.googleapis.com`, and\n  `monitoring.googleapis.com` are enabled.\n- **Required IAM Roles**:\n  - Kubernetes Engine Viewer (`roles\u002Fcontainer.viewer`)\n  - Compute Viewer (`roles\u002Fcompute.viewer`)\n  - Logs Viewer (`roles\u002Flogging.viewer`)\n  - Monitoring Viewer (`roles\u002Fmonitoring.viewer`)\n  - For remediation (`[High Risk]` steps): Kubernetes Engine Cluster Admin\n    (`roles\u002Fcontainer.clusterAdmin`)\n- **Documentation**:\n  - [Troubleshoot nodes with the NotReady status in GKE](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Ftroubleshooting\u002Fnode-notready.md.txt)\n    (Sections: \"Check the node's status and conditions\", \"Confirm node\n    preemption\", \"Verify that the node has recovered\")\n  - [Deploy TPU workloads in GKE Standard](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Ftpus.md.txt)\n    (Sections: \"Monitor health metrics for TPU nodes and node pools\", \"Configure\n    auto repair for TPU slice nodes\")\n  - [Troubleshoot OOM events](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Ftroubleshooting\u002Foom-events.md.txt)\n  - [View GKE logs](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fview-logs.md.txt)\n    (Sections: \"System logs\")\n  - [Viewing serial port output](https:\u002F\u002Fdocs.cloud.google.com\u002Fcompute\u002Fdocs\u002Ftroubleshooting\u002Fviewing-serial-port-output.md.txt)\n  - [Troubleshoot Linux VM boot issues due to kernel panic](https:\u002F\u002Fdocs.cloud.google.com\u002Fcompute\u002Fdocs\u002Ftroubleshooting\u002Fkernel-panic.md.txt)\n  - [Auto-repair nodes](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt)\n    (Sections: \"Settings for Autopilot and Standard\", \"Verify node auto-repair\n    is enabled for a Standard node pool\", \"Get information about recent\n    automated repair events\", \"Enable auto-repair for an existing Standard node\n    pool\", \"Repair criteria\", \"Node repair process\", \"Node auto repair in TPU\n    slice nodes\")\n  - [Troubleshooting VM shutdowns and reboots](https:\u002F\u002Fdocs.cloud.google.com\u002Fcompute\u002Fdocs\u002Ftroubleshooting\u002Ftroubleshooting-reboots.md.txt)\n    (Sections: \"Querying Cloud Audit Logs\", \"Reviewing Cloud Audit Logs\")\n\n> **Read-only rule**: Run read-only diagnostic commands only. Never drain,\n> delete, or re-create nodes, or run any other command that changes the cluster.\n> Give the user any fix to apply themselves.\n\nWhen you recommend a fix, link the doc section that describes it.\n\n---\n\n## Diagnostic workflow\n\n### Step 0: Collect context and set the investigation window `[Low Risk]`\n\nCollect the target parameters. By default, query a 60-minute window `[T - 30m, T\n+ 30m]` around `{issue_time}`:\n\n- `{project_id}`: Google Cloud project ID\n- `{cluster_name}`: GKE cluster name\n- `{location}`: Cluster region or zone\n- `{nodepool_name}`: Target TPU or GPU node pool name\n- `{node_name}`: Unresponsive GKE node name (and its Compute Engine `{zone}`)\n- `{issue_time}`: Incident timestamp in RFC3339 UTC\n- `{start_time}`: `{issue_time} - 30m`\n- `{end_time}`: `{issue_time} + 30m`\n\n---\n\n### Step 1: Verify the `NodeStatusUnknown` heartbeat timeout `[Low Risk]`\n\n1. **Check Kubernetes node conditions**: To inspect the node status and verify\n   whether the `Ready` condition is `Unknown` with `Reason: NodeStatusUnknown`\n   (`Kubelet stopped posting node status.`), follow the instructions in the\n   section\n   [Check the node's status and conditions](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Ftroubleshooting\u002Fnode-notready.md.txt).\n2. **Query Cloud Logging (read-only LQL)**: Query `k8s_node` and `k8s_cluster`\n   logs across `[{start_time}, {end_time}]` to confirm when the control plane\n   lost heartbeat contact with `{node_name}`:\n\n```\n(resource.type=\"k8s_node\" OR resource.type=\"k8s_cluster\")\nresource.labels.cluster_name=\"{cluster_name}\"\n(\"{node_name}\" AND (\"NodeNotReady\" OR \"NodeStatusUnknown\" OR \"Kubelet stopped posting node status\"))\ntimestamp >= \"{start_time}\" AND timestamp \u003C= \"{end_time}\"\n```\n\n3. **Query Cloud Monitoring (read-only PromQL)**: Correlate the duration of the\n   `Unknown` state using the GKE system metric\n   `kubernetes.io\u002Fnode\u002Fstatus_condition` (`kubernetes_io:node_status_condition`,\n   GKE `1.32.1-gke.1357001+`) documented in\n   [Monitor health metrics for TPU nodes and node pools](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Ftpus.md.txt),\n   filtered by `condition=\"Ready\"` and `status=\"Unknown\"`:\n\n```promql\nkubernetes_io:node_status_condition{\n  monitored_resource=\"k8s_node\",\n  cluster_name=\"{cluster_name}\",\n  node_name=\"{node_name}\",\n  condition=\"Ready\",\n  status=\"Unknown\"\n}\n```\n\n- **Decision logic**:\n  - If the node `Ready` condition is `True` and `NodeStatusUnknown` is absent,\n    **rule out** an unresponsive node timeout and pivot to workload-level\n    troubleshooting (for example,\n    [Troubleshoot OOM events](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Ftroubleshooting\u002Foom-events.md.txt))\n    rather than repairing or draining the node.\n  - If `Ready` is `Unknown` (`NodeStatusUnknown`), proceed to Step 2.\n\n---\n\n### Step 2: Inspect serial port output for a kernel panic `[Low Risk]`\n\nThe guest OS can't send logs after a fatal kernel freeze, so check the serial\nport output of the node's VM:\n\n- **Cloud Logging**: If serial port logging is enabled (i.e. VM metadata\n  `serial-port-logging-enable` is set to `true`), GKE system logs in Cloud\n  Logging include the node's serial port output. See the section\n  [System logs](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fview-logs.md.txt).\n  Use Cloud Logging when the VM is stopped or has already been replaced by\n  auto-repair, or when you need more than the most recent output.\n- **Running VM**: Follow\n  [Viewing serial port output](https:\u002F\u002Fdocs.cloud.google.com\u002Fcompute\u002Fdocs\u002Ftroubleshooting\u002Fviewing-serial-port-output.md.txt)\n  to retrieve the serial port 1 output (`gcloud compute instances\n  get-serial-port-output` with `--port=1`) for `{node_name}` in `{zone}`. This\n  method returns only the most recent 1 MB of output per port.\n\nConsult\n[Troubleshoot Linux VM boot issues due to kernel panic](https:\u002F\u002Fdocs.cloud.google.com\u002Fcompute\u002Fdocs\u002Ftroubleshooting\u002Fkernel-panic.md.txt)\nto identify documented kernel panic and hardware crash patterns (such as `Fatal\nMachine check`, `hung_task: blocked tasks`, or `NMI: Not continuing`) in the\nserial port output.\n\n---\n\n### Step 3: Check node auto-repair status `[Low Risk]`\n\nFind out why GKE hasn't repaired the unresponsive node:\n\n- **Autopilot clusters**: Autopilot always repairs nodes, and you can't turn\n  this off (see\n  [Settings for Autopilot and Standard](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt)).\n  Skip the configuration check and check the repair history.\n- **Standard clusters**: Follow\n  [Verify node auto-repair is enabled for a Standard node pool](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt)\n  to check whether `autoRepair` is enabled on `{nodepool_name}`.\n- **Repair history (both modes)**: Auto-repair can be enabled and the repair can\n  still fail. Per\n  [Configure auto repair for TPU slice nodes](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Ftpus.md.txt),\n  check the repair status, including the failure reason, in the operation\n  history described in\n  [Get information about recent automated repair events](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt).\n  If the failure is caused by insufficient quota, tell the user to contact their\n  Google Cloud account representative to increase the quota.\n\n---\n\n### Step 4: Check Compute Engine system events `[Low Risk]`\n\nTo list the system events for `{node_name}` around `{issue_time}`, follow the\ninstructions in the section\n[Querying Cloud Audit Logs](https:\u002F\u002Fdocs.cloud.google.com\u002Fcompute\u002Fdocs\u002Ftroubleshooting\u002Ftroubleshooting-reboots.md.txt).\nCompare the `method` field with the table in the \"Reviewing Cloud Audit Logs\"\nsection of the same document, and look for:\n\n- `compute.instances.hostError`: a hardware or software issue on the physical\n  host caused the VM to crash.\n- `compute.instances.preempted`: Compute Engine preempted a Spot VM or\n  preemptible VM. For preempted nodes, also see\n  [Confirm node preemption](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Ftroubleshooting\u002Fnode-notready.md.txt).\n- `compute.instances.automaticRestart`: Compute Engine restarted the VM after a\n  `hostError` or `terminateOnHostMaintenance` event.\n- `compute.instances.guestTerminate`: the VM's operating system initiated the\n  shutdown.\n\n---\n\n### Step 5: Resolution `[High Risk]`\n\n> **Guardrails**:\n> - **Never** force-delete stuck `Terminating` pods on an unresponsive node.\n>   Force deletion doesn't wait for the kubelet to confirm that the pod has\n>   stopped, so a replacement pod can start while the old one is still running.\n>   See\n>   [Force Delete StatefulSet Pods](https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Ftasks\u002Frun-application\u002Fforce-delete-stateful-set-pod\u002F#force-deletion).\n> - **Never** delete GKE-managed Compute Engine VM instances directly (`gcloud\n>   compute instances delete`). Instead, check how long the node has reported\n>   `NodeStatusUnknown` (Step 1), check the serial console output (Step 2), and\n>   rely on node auto-repair, as described in the following steps.\n\n1. **Enable node auto-repair (Standard clusters only)**:\n   - Autopilot clusters always auto-repair nodes, so skip this step.\n   - Before the user enables auto-repair on a multi-host TPU slice node pool,\n     tell them that GKE re-creates the entire node pool when a node in it needs\n     repair (see step 2).\n   - If `autoRepair` is disabled on `{nodepool_name}`, link the user to\n     [Enable auto-repair for an existing Standard node pool](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt)\n     and\n     [Configure auto repair for TPU slice nodes](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Ftpus.md.txt)\n     so they can apply the change themselves.\n2. **Let GKE repair the node**:\n   - Explain that GKE repairs a node that reports `NotReady` or no status for\n     the documented time threshold by draining and re-creating it, and link the\n     \"Repair criteria\" and \"Node repair process\" sections of\n     [Auto-repair nodes](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt).\n     GKE waits one hour for the drain to complete. If the drain doesn't\n     complete, GKE shuts the node down and creates a new node. Tell the user to\n     expect this one-hour drain wait before GKE re-creates the node.\n   - For multi-host TPU slice node pools, note per\n     [Node auto repair in TPU slice nodes](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt)\n     that the entire node pool is re-created.\n3. **Verify node recovery**:\n   - After the repair completes, follow the instructions in the section\n     [Verify that the node has recovered](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Ftroubleshooting\u002Fnode-notready.md.txt)\n     to verify that the repaired node returns to `Ready` status, then confirm\n     that the `JobSet` pods are running again.\n","# Troubleshoot unresponsive GKE TPU and GPU nodes (`NodeStatusUnknown`)\n\nWhen the Compute Engine host of a TPU or GPU node has a fatal hardware error,\nkernel panic, or non-maskable interrupt (NMI) lockup, the guest OS stops\nresponding. The kubelet can no longer send heartbeats, so the node `Ready`\ncondition becomes `Unknown` with `Reason: NodeStatusUnknown` (`Kubelet stopped\nposting node status.`). If node auto-repair is disabled on the node pool, GKE\ndoesn't repair the node. The node can stay `NotReady`, and pods on it can stay\nin `Terminating`, which blocks multi-host `JobSet` workloads from recovering.\n\n## Attribution\n\n**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you\nMUST read [Usage Attribution](references\u002Fattribution.md). For commands executed\ndirectly in the terminal (never in generated scripts, Terraform, or user code),\nprefix `gcloud` with\n`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION=\"agent-skills\u002F1.0 (skill:gke-ai-troubleshooting-node-unresponsive-timeout)\"`\nand append `agent-skills\u002F1.0 (skill:gke-ai-troubleshooting-node-unresponsive-timeout)` to `User-Agent` headers.\n\n## Prerequisites\n\n- **Tools**: Install the\n  [Google Cloud SDK](https:\u002F\u002Fcloud.google.com\u002Fsdk\u002Fdocs\u002Finstall) (`gcloud`) and\n  `kubectl`.\n- **Cloud Billing & Project Configuration**: Verify an active billing account is\n  linked (`gcloud billing projects describe {project_id}`), authenticate\n  (`gcloud auth login`), set the target project (`gcloud config set project\n  {project_id}`), and ensure `container.googleapis.com`,\n  `compute.googleapis.com`, `logging.googleapis.com`, and\n  `monitoring.googleapis.com` are enabled.\n- **Required IAM Roles**:\n  - Kubernetes Engine Viewer (`roles\u002Fcontainer.viewer`)\n  - Compute Viewer (`roles\u002Fcompute.viewer`)\n  - Logs Viewer (`roles\u002Flogging.viewer`)\n  - Monitoring Viewer (`roles\u002Fmonitoring.viewer`)\n  - For remediation (`[High Risk]` steps): Kubernetes Engine Cluster Admin\n    (`roles\u002Fcontainer.clusterAdmin`)\n- **Documentation**:\n  - [Troubleshoot nodes with the NotReady status in GKE](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Ftroubleshooting\u002Fnode-notready.md.txt)\n    (Sections: \"Check the node's status and conditions\", \"Confirm node\n    preemption\", \"Verify that the node has recovered\")\n  - [Deploy TPU workloads in GKE Standard](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Ftpus.md.txt)\n    (Sections: \"Monitor health metrics for TPU nodes and node pools\", \"Configure\n    auto repair for TPU slice nodes\")\n  - [Troubleshoot OOM events](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Ftroubleshooting\u002Foom-events.md.txt)\n  - [View GKE logs](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fview-logs.md.txt)\n    (Sections: \"System logs\")\n  - [Viewing serial port output](https:\u002F\u002Fdocs.cloud.google.com\u002Fcompute\u002Fdocs\u002Ftroubleshooting\u002Fviewing-serial-port-output.md.txt)\n  - [Troubleshoot Linux VM boot issues due to kernel panic](https:\u002F\u002Fdocs.cloud.google.com\u002Fcompute\u002Fdocs\u002Ftroubleshooting\u002Fkernel-panic.md.txt)\n  - [Auto-repair nodes](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt)\n    (Sections: \"Settings for Autopilot and Standard\", \"Verify node auto-repair\n    is enabled for a Standard node pool\", \"Get information about recent\n    automated repair events\", \"Enable auto-repair for an existing Standard node\n    pool\", \"Repair criteria\", \"Node repair process\", \"Node auto repair in TPU\n    slice nodes\")\n  - [Troubleshooting VM shutdowns and reboots](https:\u002F\u002Fdocs.cloud.google.com\u002Fcompute\u002Fdocs\u002Ftroubleshooting\u002Ftroubleshooting-reboots.md.txt)\n    (Sections: \"Querying Cloud Audit Logs\", \"Reviewing Cloud Audit Logs\")\n\n> **Read-only rule**: Run read-only diagnostic commands only. Never drain,\n> delete, or re-create nodes, or run any other command that changes the cluster.\n> Give the user any fix to apply themselves.\n\nWhen you recommend a fix, link the doc section that describes it.\n\n---\n\n## Diagnostic workflow\n\n### Step 0: Collect context and set the investigation window `[Low Risk]`\n\nCollect the target parameters. By default, query a 60-minute window `[T - 30m, T\n+ 30m]` around `{issue_time}`:\n\n- `{project_id}`: Google Cloud project ID\n- `{cluster_name}`: GKE cluster name\n- `{location}`: Cluster region or zone\n- `{nodepool_name}`: Target TPU or GPU node pool name\n- `{node_name}`: Unresponsive GKE node name (and its Compute Engine `{zone}`)\n- `{issue_time}`: Incident timestamp in RFC3339 UTC\n- `{start_time}`: `{issue_time} - 30m`\n- `{end_time}`: `{issue_time} + 30m`\n\n---\n\n### Step 1: Verify the `NodeStatusUnknown` heartbeat timeout `[Low Risk]`\n\n1. **Check Kubernetes node conditions**: To inspect the node status and verify\n   whether the `Ready` condition is `Unknown` with `Reason: NodeStatusUnknown`\n   (`Kubelet stopped posting node status.`), follow the instructions in the\n   section\n   [Check the node's status and conditions](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Ftroubleshooting\u002Fnode-notready.md.txt).\n2. **Query Cloud Logging (read-only LQL)**: Query `k8s_node` and `k8s_cluster`\n   logs across `[{start_time}, {end_time}]` to confirm when the control plane\n   lost heartbeat contact with `{node_name}`:\n\n```\n(resource.type=\"k8s_node\" OR resource.type=\"k8s_cluster\")\nresource.labels.cluster_name=\"{cluster_name}\"\n(\"{node_name}\" AND (\"NodeNotReady\" OR \"NodeStatusUnknown\" OR \"Kubelet stopped posting node status\"))\ntimestamp >= \"{start_time}\" AND timestamp \u003C= \"{end_time}\"\n```\n\n3. **Query Cloud Monitoring (read-only PromQL)**: Correlate the duration of the\n   `Unknown` state using the GKE system metric\n   `kubernetes.io\u002Fnode\u002Fstatus_condition` (`kubernetes_io:node_status_condition`,\n   GKE `1.32.1-gke.1357001+`) documented in\n   [Monitor health metrics for TPU nodes and node pools](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Ftpus.md.txt),\n   filtered by `condition=\"Ready\"` and `status=\"Unknown\"`:\n\n```promql\nkubernetes_io:node_status_condition{\n  monitored_resource=\"k8s_node\",\n  cluster_name=\"{cluster_name}\",\n  node_name=\"{node_name}\",\n  condition=\"Ready\",\n  status=\"Unknown\"\n}\n```\n\n- **Decision logic**:\n  - If the node `Ready` condition is `True` and `NodeStatusUnknown` is absent,\n    **rule out** an unresponsive node timeout and pivot to workload-level\n    troubleshooting (for example,\n    [Troubleshoot OOM events](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Ftroubleshooting\u002Foom-events.md.txt))\n    rather than repairing or draining the node.\n  - If `Ready` is `Unknown` (`NodeStatusUnknown`), proceed to Step 2.\n\n---\n\n### Step 2: Inspect serial port output for a kernel panic `[Low Risk]`\n\nThe guest OS can't send logs after a fatal kernel freeze, so check the serial\nport output of the node's VM:\n\n- **Cloud Logging**: If serial port logging is enabled (i.e. VM metadata\n  `serial-port-logging-enable` is set to `true`), GKE system logs in Cloud\n  Logging include the node's serial port output. See the section\n  [System logs](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fview-logs.md.txt).\n  Use Cloud Logging when the VM is stopped or has already been replaced by\n  auto-repair, or when you need more than the most recent output.\n- **Running VM**: Follow\n  [Viewing serial port output](https:\u002F\u002Fdocs.cloud.google.com\u002Fcompute\u002Fdocs\u002Ftroubleshooting\u002Fviewing-serial-port-output.md.txt)\n  to retrieve the serial port 1 output (`gcloud compute instances\n  get-serial-port-output` with `--port=1`) for `{node_name}` in `{zone}`. This\n  method returns only the most recent 1 MB of output per port.\n\nConsult\n[Troubleshoot Linux VM boot issues due to kernel panic](https:\u002F\u002Fdocs.cloud.google.com\u002Fcompute\u002Fdocs\u002Ftroubleshooting\u002Fkernel-panic.md.txt)\nto identify documented kernel panic and hardware crash patterns (such as `Fatal\nMachine check`, `hung_task: blocked tasks`, or `NMI: Not continuing`) in the\nserial port output.\n\n---\n\n### Step 3: Check node auto-repair status `[Low Risk]`\n\nFind out why GKE hasn't repaired the unresponsive node:\n\n- **Autopilot clusters**: Autopilot always repairs nodes, and you can't turn\n  this off (see\n  [Settings for Autopilot and Standard](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt)).\n  Skip the configuration check and check the repair history.\n- **Standard clusters**: Follow\n  [Verify node auto-repair is enabled for a Standard node pool](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt)\n  to check whether `autoRepair` is enabled on `{nodepool_name}`.\n- **Repair history (both modes)**: Auto-repair can be enabled and the repair can\n  still fail. Per\n  [Configure auto repair for TPU slice nodes](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Ftpus.md.txt),\n  check the repair status, including the failure reason, in the operation\n  history described in\n  [Get information about recent automated repair events](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt).\n  If the failure is caused by insufficient quota, tell the user to contact their\n  Google Cloud account representative to increase the quota.\n\n---\n\n### Step 4: Check Compute Engine system events `[Low Risk]`\n\nTo list the system events for `{node_name}` around `{issue_time}`, follow the\ninstructions in the section\n[Querying Cloud Audit Logs](https:\u002F\u002Fdocs.cloud.google.com\u002Fcompute\u002Fdocs\u002Ftroubleshooting\u002Ftroubleshooting-reboots.md.txt).\nCompare the `method` field with the table in the \"Reviewing Cloud Audit Logs\"\nsection of the same document, and look for:\n\n- `compute.instances.hostError`: a hardware or software issue on the physical\n  host caused the VM to crash.\n- `compute.instances.preempted`: Compute Engine preempted a Spot VM or\n  preemptible VM. For preempted nodes, also see\n  [Confirm node preemption](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Ftroubleshooting\u002Fnode-notready.md.txt).\n- `compute.instances.automaticRestart`: Compute Engine restarted the VM after a\n  `hostError` or `terminateOnHostMaintenance` event.\n- `compute.instances.guestTerminate`: the VM's operating system initiated the\n  shutdown.\n\n---\n\n### Step 5: Resolution `[High Risk]`\n\n> **Guardrails**:\n> - **Never** force-delete stuck `Terminating` pods on an unresponsive node.\n>   Force deletion doesn't wait for the kubelet to confirm that the pod has\n>   stopped, so a replacement pod can start while the old one is still running.\n>   See\n>   [Force Delete StatefulSet Pods](https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Ftasks\u002Frun-application\u002Fforce-delete-stateful-set-pod\u002F#force-deletion).\n> - **Never** delete GKE-managed Compute Engine VM instances directly (`gcloud\n>   compute instances delete`). Instead, check how long the node has reported\n>   `NodeStatusUnknown` (Step 1), check the serial console output (Step 2), and\n>   rely on node auto-repair, as described in the following steps.\n\n1. **Enable node auto-repair (Standard clusters only)**:\n   - Autopilot clusters always auto-repair nodes, so skip this step.\n   - Before the user enables auto-repair on a multi-host TPU slice node pool,\n     tell them that GKE re-creates the entire node pool when a node in it needs\n     repair (see step 2).\n   - If `autoRepair` is disabled on `{nodepool_name}`, link the user to\n     [Enable auto-repair for an existing Standard node pool](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt)\n     and\n     [Configure auto repair for TPU slice nodes](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Ftpus.md.txt)\n     so they can apply the change themselves.\n2. **Let GKE repair the node**:\n   - Explain that GKE repairs a node that reports `NotReady` or no status for\n     the documented time threshold by draining and re-creating it, and link the\n     \"Repair criteria\" and \"Node repair process\" sections of\n     [Auto-repair nodes](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt).\n     GKE waits one hour for the drain to complete. If the drain doesn't\n     complete, GKE shuts the node down and creates a new node. Tell the user to\n     expect this one-hour drain wait before GKE re-creates the node.\n   - For multi-host TPU slice node pools, note per\n     [Node auto repair in TPU slice nodes](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fnode-auto-repair.md.txt)\n     that the entire node pool is re-created.\n3. **Verify node recovery**:\n   - After the repair completes, follow the instructions in the section\n     [Verify that the node has recovered](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Ftroubleshooting\u002Fnode-notready.md.txt)\n     to verify that the repaired node returns to `Ready` status, then confirm\n     that the `JobSet` pods are running again.\n",{"name":5,"metadata":44,"description":7},{"version":45,"category":46},"1.0.1","Containers","main","bb9b3f464aae37af360d079c1d863524955b0363","skills\u002Fcloud\u002Fgke-ai-troubleshooting-node-unresponsive-timeout",2005,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills","https:\u002F\u002Fgithub.com\u002Fgoogle","https:\u002F\u002Fopensource.google\u002F","Google ❤️ Open Source","2026-11-14 01:51:35",{"source":8,"skillName":5,"commands":57,"prompt":67,"zipAvailable":68},[58,61,64],{"tool":59,"command":60},"npx","npx skills add google\u002Fskills --skill gke-ai-troubleshooting-node-unresponsive-timeout",{"tool":62,"command":63},"bunx","bunx skills add google\u002Fskills --skill gke-ai-troubleshooting-node-unresponsive-timeout",{"tool":65,"command":66},"pnpm","pnpm dlx skills add google\u002Fskills --skill gke-ai-troubleshooting-node-unresponsive-timeout","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fgke-ai-troubleshooting-node-unresponsive-timeout\u002FSKILL.md",true,[70],{"path":71,"resourceType":72,"resourceTypeDesc":73,"sizeBytes":74,"sourceHtmlUrl":75},"references\u002Fattribution.md","REFERENCE","参考文档",1984,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fgke-ai-troubleshooting-node-unresponsive-timeout\u002Freferences\u002Fattribution.md",[77,115,145,174,201,226],{"skillNo":78,"name":79,"nameZh":6,"description":80,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":81,"categories":82,"keywords":91,"publishedAt":39},"01M6998S2EZKD3GEJ7F943RZTE","sign-in-with-google-web","Implement, configure, and secure Sign In With Google (SiwG) using Google Identity Services (GIS \u002F `https:\u002F\u002Faccounts.google.com\u002Fgsi\u002Fclient`) across web architectures. Use when creating Google sign-in buttons, implementing Google One Tap with FedCM, integrating GIS in React\u002FNext.js\u002FAngular\u002FHTML, verifying ID tokens on backend runtimes (Python\u002FNode.js\u002FGo\u002FJava), enforcing Google Workspace domain restrictions (`hd`), securing client-side ID tokens via WebCrypto nonces or encrypted IndexedDB, embedding in cross-origin iframes via the Intermediate Iframe API (`https:\u002F\u002Faccounts.google.com\u002Fgsi\u002Fintermediate`), configuring Content Security Policy (CSP), COOP, and Permissions Policy (`identity-credentials-get`) headers, or handling sign-out and revocation. Don't use for legacy Google Sign-In (`gapi.auth2`), Google Cloud IAM service accounts, or Android\u002FiOS native Credential Manager.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fidentity\u002Fsign-in-with-google-web\u002FSKILL.md",[83,88],{"categoryNo":84,"categoryName":85,"taxonomyType":86,"taxonomyTypeDesc":87},"01JBSEEDCAT000000000000009","前端开发","SCENARIO","使用场景",{"categoryNo":89,"categoryName":90,"taxonomyType":86,"taxonomyTypeDesc":87},"01JBSEEDCAT000000000000010","后端与接口",[9,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114],"sign in with google web","sign","with","web","android","go","security","google workspace","google cloud","implement","policy","python","implementing","react","iam","permissions","buttons","secure","manager","identity","architectures","domain","runtimes",{"skillNo":116,"name":117,"nameZh":6,"description":118,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":119,"categories":120,"keywords":130,"publishedAt":39},"01M6998PHPDCAD684VW5WRG83D","dpop-adoption","Implement and debug OAuth 2.0 DPoP (RFC 9449) refresh token sender-constraining for WebCrypto, Node.js ES6, and browser runtimes integrating with Google's OAuth platform. Use when configuring non-extractable asymmetric key pairs (P-256), generating DPoP Proof JWTs for authorization code exchange and token refresh, or handling 400 use_dpop_nonce challenge retry loops at oauth2.googleapis.com\u002Ftoken. Don't use for unconstrained OAuth 2.0 flows (where refresh tokens are not bound to a client key pair), or for Google Cloud IAM \u002F service account authentication.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fidentity\u002Fdpop-adoption\u002FSKILL.md",[121,122,125],{"categoryNo":89,"categoryName":90,"taxonomyType":86,"taxonomyTypeDesc":87},{"categoryNo":123,"categoryName":124,"taxonomyType":86,"taxonomyTypeDesc":87},"01JBSEEDCAT000000000000023","Skill与智能体开发",{"categoryNo":126,"categoryName":127,"taxonomyType":128,"taxonomyTypeDesc":129},"01JBSEEDCAT000000000000030","开发工程师","PROFESSION","适用职业",[131,132,9,100,133,134,135,136,137,138,106,139,140,141,23,142,143,111,114,144,25],"dpop adoption","adoption","es6","retry","authentication","flows","challenge","code","browser","loops","at","token","pair","platform",{"skillNo":146,"name":147,"nameZh":6,"description":148,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":149,"categories":150,"keywords":157,"publishedAt":39},"01M6998M72TC73EEAQH4XC7EXW","retrieving-developer-knowledge","Searches, retrieves, and synthesizes official Google developer documentation across Google Cloud, AI\u002FGemini, Android, Chrome, Web, Flutter, Go, Firebase, and other Google developer platforms. Integrates with the Developer Knowledge MCP server (search_documents, get_documents, answer_query) or the Developer Knowledge REST API fallback. Use when searching for gcloud CLI commands, API syntax, IAM permissions, official documentation, architectural comparisons, or product choice overviews. Don't use for local filesystem lookups or non-Google documentation.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fdevelopers\u002Fretrieving-developer-knowledge\u002FSKILL.md",[151,152,155,156],{"categoryNo":89,"categoryName":90,"taxonomyType":86,"taxonomyTypeDesc":87},{"categoryNo":153,"categoryName":154,"taxonomyType":86,"taxonomyTypeDesc":87},"01JBSEEDCAT000000000000021","资料检索与研究",{"categoryNo":123,"categoryName":124,"taxonomyType":86,"taxonomyTypeDesc":87},{"categoryNo":126,"categoryName":127,"taxonomyType":128,"taxonomyTypeDesc":129},[158,159,160,161,100,162,163,164,165,166,167,168,169,25,170,171,172,106,173,107],"retrieving developer knowledge","knowledge","developer","MCP","mcp server","official","developer documentation","get","search","local","choice","documents","query","api","across","documentation",{"skillNo":175,"name":176,"nameZh":6,"description":177,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":178,"categories":179,"keywords":180,"publishedAt":39},"01M6998J2CZEXS9BES9S3WWFKC","migrate-to-rust","Clones, analyzes, plans, and migrates legacy codebases (C++, Fortran, Java, Python, C, Shell) from a GitHub URL or local directory to a modern, high-performance, deterministic, 100% Safe Rust architecture, tracking live telemetry across every task bucket, test failure\u002Ffix cycle, and Profile-Guided Optimization (PGO) stage, and producing a first-person AI storytelling report and interactive HTML infographic. Use when migrating a GitHub repository or local codebase to Rust, porting legacy C++, Fortran, Python, or Java workloads to memory-safe Rust, designing Struct-of-Arrays (SoA) or SIMD Rust architectures, or optimizing Rust ports with LLVM PGO. Don't use for unrelated non-Rust refactoring or single-line syntax questions.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fdevelopers\u002Fmigrate-to-rust\u002FSKILL.md",[],[181,182,183,184,185,186,187,188,103,189,190,191,192,193,194,167,195,196,197,25,198,199,172,200],"migrate to rust","rust","migrate","github","memory","performance","directory","safe","ports","llvm","soa","cycle","porting","fix","arrays","infographic","high-performance","URL","architecture","unrelated",{"skillNo":202,"name":203,"nameZh":6,"description":204,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":205,"categories":206,"keywords":210,"publishedAt":39},"01M6998GCC3K29WZ82KXMB9E02","finding-google-skills","Google platform decision and setup guidance, loaded on demand from Google's skill catalog. Use when a developer is choosing or setting up part of their stack, such as where to run a service, a database, storage, messaging, authentication, analytics, ads, or AI model serving, and a Google product is a reasonable candidate - whether or not a vendor is named - or when a request names a Google product or API. Brings in the matching Google skill so the answer can weigh Google options, their trade-offs, and when they are not the right fit. Skip when the stack is already settled on another provider and no Google product is named, or the task involves no platform choice.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fdevelopers\u002Ffinding-google-skills\u002FSKILL.md",[207,208,209],{"categoryNo":89,"categoryName":90,"taxonomyType":86,"taxonomyTypeDesc":87},{"categoryNo":123,"categoryName":124,"taxonomyType":86,"taxonomyTypeDesc":87},{"categoryNo":126,"categoryName":127,"taxonomyType":128,"taxonomyTypeDesc":129},[9,211,212,213,214,144,215,216,217,218,219,135,220,221,222,168,223,224,25,225,171],"finding google skills","storage","as","messaging","model","trade-offs","provider","answer","fit","catalog","up","decision","use","database","candidate",{"skillNo":227,"name":228,"nameZh":6,"description":229,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":230,"categories":231,"keywords":234,"publishedAt":39},"01M6998DWNHTH0BS6W1CY8ZXMP","workload-manager-basics","Use this skill to manage Google Cloud Workload Manager evaluations, rules, scanned resources, and validation results by using public client libraries and the REST API. Use when you need to inspect workload best-practice rules, create and run evaluations for Google Cloud general best practices, SAP, SQL Server, or custom organizational rules, review violations, export results to BigQuery, or automate Workload Manager through client libraries because no service-specific public CLI or MCP server is available. Don't use for general Google Compute Engine instance management, VPC configuration, or standard IAM auditing.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fworkload-manager-basics\u002FSKILL.md",[232,233],{"categoryNo":89,"categoryName":90,"taxonomyType":86,"taxonomyTypeDesc":87},{"categoryNo":123,"categoryName":124,"taxonomyType":86,"taxonomyTypeDesc":87},[235,236,110,237,9,161,238,100,162,239,240,241,242,25,243,171,106,244,245,246,247],"workload manager basics","workload","basics","cli","review","vpc","resources","inspect","you","create","libraries","server","run"]