gke-node-notready
Diagnoses GKE nodes reporting NotReady or Unknown status by inspecting node conditions, events, kubelet/containerd logs, and node metrics, then proposing safe remediations. Use when nodes show NotReady, when the kubelet stops posting node status, or when workloads are evicted or stuck Pending due to node health. Don't use for pod-level application failures (use gke-workload-troubleshooting), autoscaler scale-up/scale-down decisions (use gke-cluster-autoscaler), or non-GKE compute.
- License 许可
- Apache-2.0
- 所属仓库
- google/skills
- 最近更新
- 2026-11-14
- Stars
- ★ 2.4w
- Forks
- 2.0k
01 前言 Front Matter
- name
- gke-node-notready
- metadata
- { "version": "1.0.1", "category": "Containers" }
- description
- Diagnoses GKE nodes reporting NotReady or Unknown status by inspecting node conditions, events, kubelet/containerd logs, and node metrics, then proposing safe remediations. Use when nodes show NotReady, when the kubelet stops posting node status, or when workloads are evicted or stuck Pending due to node health. Don't use for pod-level application failures (use gke-workload-troubleshooting), autoscaler scale-up/scale-down decisions (use gke-cluster-autoscaler), or non-GKE compute.
02 资源文件 1
- 参考文档references/attribution.md1.8 KB
03 SKILL.md 正文
GKE Node NotReady Troubleshooting Skill
Use this skill to systematically diagnose why one or more GKE nodes report a
NotReady (or Ready: Unknown) status and to propose safe remediations. A
NotReady status means the node's kubelet is not reporting to the control plane
correctly, so Kubernetes stops scheduling new Pods on the node, which can reduce
application capacity and cause downtime.
This skill operates non-interactively and enforces a read-only diagnostics
boundary: gather evidence first, then propose a fix (a kubectl/gcloud
command or a GitOps manifest change) for a human to apply. Never mutate the
cluster, drain, delete, or recreate nodes automatically.
[!IMPORTANT] First rule out an expected
NotReady: a node that is newly provisioning, upgrading, being repaired, cordoned, or scaling down will transiently reportNotReady. Only treat it as a fault if it persists beyond the expected window.
Attribution
MUST READ: Before executing any gcloud, REST API, or MCP command, you
MUST read Usage Attribution. For commands executed
directly in the terminal (never in generated scripts, Terraform, or user code),
prefix gcloud with
CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-node-notready)"
and append agent-skills/1.0 (skill:gke-node-notready) to User-Agent headers.
? Diagnostic Workflow
Step 0: Context discovery & time window
- Parameter extraction — obtain
project_id,cluster_name,cluster_location, andnode_namenon-interactively from the user prompt, activeSETTINGS.md, or environment defaults (kubectl config current-context,gcloud config get-value project). - Credentials & fallback — attempt
gcloud container clusters get-credentials {cluster_name} --location {cluster_location} --project {project_id}. If the cluster is unreachable or commands fail (sandbox/dry-run/offline), present the exact diagnostic commands for a human to run and continue the analysis from the reported symptoms. - Time window — determine
{issue_time}(explicit, relative, or now) and center a 1-hour window around it (start = issue_time - 30m,end = issue_time + 30m) for all log/metric queries.
Step 1: Identify NotReady nodes and gather initial status
# List nodes and spot NotReady status, node IPs, and container-runtime version.
kubectl get nodes -o wide
# Inspect the affected node's Conditions and Events (the primary clues).
kubectl describe node "{node_name}"
Equivalent via Cloud Logging (preferred when kubectl access is limited or for
historical events). Open it as a Logs Explorer deep link — URL-encode the
query and append the project and Step 0 time window:
https://console.cloud.google.com/logs/query;query={URL_ENCODED_QUERY};timeRange={start}%2F{end}?project={project_id}
(encode / as %2F, or use ;duration=PT1H for a rolling hour):
resource.type="k8s_node"
log_id("events")
resource.labels.node_name="{node_name}"
resource.labels.cluster_name="{cluster_name}"
resource.labels.location="{cluster_location}"
Interpret the Conditions table:
Ready: False/Ready: Unknownwith reasonKubeletNotReady/NodeStatusUnknown("Kubelet stopped posting node status") → kubelet or runtime problem; continue to Step 2.MemoryPressure: True,DiskPressure: True,PIDPressure: True→ resource exhaustion; go to Step 4b.NetworkUnavailable: True→ networking/CNI problem; go to Step 4d.
Step 2: Scan kubelet logs for error signatures
Open these kubelet logs as a Logs Explorer deep link using the same
logs/query;query={URL_ENCODED_QUERY};timeRange=...?project=... pattern as Step 1.
resource.type="k8s_node"
resource.labels.node_name="{node_name}"
resource.labels.cluster_name="{cluster_name}"
resource.labels.location="{cluster_location}"
log_id("kubelet")
severity>=WARNING
Also review the node's serial-console logs (log_id("serialconsole.googleapis.com/serial_port_1_output")
or the resource.type="gce_instance" serial logs) for kernel TaskHung,
OOM-killer, or disk I/O errors that correlate with the kubelet failures.
Step 3: Map the signature to a root cause (decision table)
| Kubelet / event signature | Likely root cause | Go to |
|---|---|---|
runtime is down, Container runtime not ready, errors on /run/containerd/containerd.sock (connection refused / DeadlineExceeded) |
Container runtime (containerd) down or unresponsive |
Step 4a |
Got sys oom event from cadvisor / kernel OOM-killer in serial logs |
System (node-level) OOM killed critical processes | Step 4b |
PLEG is not healthy |
PLEG stalled, usually node overload (CPU/disk) | Step 4c |
TaskHung for containerd/kubelet, high disk latency |
Disk throttling / I/O starvation | Step 4b |
failed to ensure lease, leases.coordination.k8s.io ... namespace kube-node-lease ... terminating |
kube-node-lease termination → NotReady flapping |
Step 4f |
| Kubelet cannot reach API server, TLS/dial timeouts | Kubelet ↔ control-plane connectivity | Step 4d |
NetworkPluginNotReady, cni plugin not initialized, NetworkUnavailable |
CNI plugin failure | Step 4d |
| Node-critical DaemonSet Pods (CNI, kube-proxy, metadata) blocked from admission | Admission webhook interference | Step 4e |
Only generic NodeNotReady, no other signature |
Cause unclear — widen to Step 4d, then escalate | Escalation |
Step 4: Branch investigations
4a. Container runtime (containerd) down
Confirm the kubelet cannot talk to containerd (socket errors above). Check for
containerd restarts/crashes in serial logs. Remediation (propose, don't run):
recreate/repair the node (kubectl drain then let the node pool recreate it, or
gcloud container clusters upgrade/node auto-repair); if it recurs across nodes,
suspect a node image or custom DaemonSet interfering with containerd.
4b. Resource pressure & OOM
# Node allocatable vs. usage.
kubectl describe node "{node_name}" | sed -n '/Allocated resources/,/Events/p'
Cloud Monitoring metrics to inspect (read-only): kubernetes.io/node/memory/used_bytes,
kubernetes.io/node/cpu/core_usage_time, kubernetes.io/node/ephemeral_storage/used_bytes.
- DiskPressure / disk throttling: full boot disk or slow PD → increase disk size / use a faster PD type; reduce image/log churn.
- System OOM: node memory exhausted → set/raise Pod memory
requests/limits, reduce over-commit, or use larger machine types. Distinguish system OOM (node-wide, kills kubelet/runtime) from cgroup OOM (single container). - PIDPressure: too many processes → cap Pod PIDs / reduce workload density.
4c. PLEG is not healthy
PLEG is not healthy almost always means the node is overloaded (CPU saturation,
disk latency, or too many Pods/containers per node) so the runtime can't relist
in time. Correlate with 4b metrics. Remediation: reduce node density, add
CPU/disk headroom, or spread workloads.
4d. Networking
# Are node-critical networking Pods healthy on this node?
kubectl get pods -n kube-system -o wide --field-selector spec.nodeName={node_name}
- Kubelet ↔ control-plane: dial/TLS timeouts to the API server → check firewall rules, Private Google Access, authorized networks, and route/NAT changes.
- CNI failure (
NetworkPluginNotReady): the CNI DaemonSet (netd/calico/dataplane) is not running on the node → inspect those Pods' logs/events.
4e. Admission webhook interference
A misconfigured/failing validating or mutating webhook with a broad scope can block node-critical system Pods from being admitted, keeping the node NotReady.
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations
Look for webhooks that intercept kube-system / node-critical objects with
failurePolicy: Fail. Remediation (propose): scope the webhook out of
kube-system/node-critical namespaces or set an appropriate namespaceSelector.
4f. kube-node-lease termination flapping
If the node flaps NotReady with leases.coordination.k8s.io ... namespace kube-node-lease ... is being terminated, the kube-node-lease namespace was
deleted/terminating. Remediation (propose): do not delete the
kube-node-lease namespace; if terminating, identify the finalizer/actor holding
it and restore the namespace.
Step 5: Remediation boundary & escalation
- Present the root cause + evidence (the exact conditions, events, log lines, or metrics observed). Provide Cloud Logging deep links (and Cloud Monitoring links for the Step 4b metrics) to the supporting entries — using the deep-link pattern from Steps 1-2 — so a human can open the evidence directly.
- Propose the fix as a command or GitOps manifest change for a human to apply — never apply, drain, or recreate nodes automatically. When to escalate (do this instead of proposing more self-service diagnostics):
Escalate when either:
- the relevant logs are unavailable — excluded by a logging filter, or older
than the log bucket's retention (the
_Defaultbucket defaults to 30 days, so incidents older than that are permanently deleted); or - the kubelet/event signature is not in the Step 3 table and the root cause remains undetermined after the branch investigations.
In those cases, do all three:
- State the limitation plainly (for example, "kubelet logs for that date are
past the 30-day
_Defaultretention window and are permanently deleted"). - Summarize the findings you did gather (node conditions, events, metrics,
and any Admin Activity audit logs still in the
_Requiredbucket, default 400-day retention). - Route to GKE support / engineering escalation with those findings. Do not keep proposing further self-service investigation, and do not fabricate a diagnosis when the evidence is missing.
References
This skill is derived from public Google Cloud documentation:
- Troubleshoot nodes with the NotReady status
— node conditions and the kubelet / PLEG / system-OOM / containerd /
kube-node-lease/ CNI / admission-webhook signatures and their remediations. - Troubleshoot node registration — Node Registration Checker for nodes that never finish registering.
- View GKE logs
and Cloud Logging routing overview
— log queries (
resource.type="k8s_node",log_id("kubelet")) and log-bucket retention (_Default30 days,_Required400 days). - Logs Explorer interface
— building and sharing a query by URL (the
logs/query;query=...deep-link format used above).
04 同仓库更多 Skills
cloud-operator-share-context
Shares Google Cloud operating context the user or their team already have — environments, projects, services and regions, product preferences, best practices, architecture guidance, runbooks — found in agent instruction files, local memory, skills and workflows, and the repository's GCP docs, with the operator agent's memory after the user reviews the list. Use when the user runs this skill, accepts the cloud-operator or cloud-operator-setup skill's offer to share their existing Google Cloud context, or asks to share, import or save that context or a GCP fact for the operator agent. Don't use for ordinary GCP requests, which go to the cloud-operator skill.
cloud-operator-setup
Sets up the operator agent in a Google Cloud project via the autocloud MCP server: picks the operator project, creates or checks the user's operator agent, saves the project, and offers to share the user's existing Google Cloud operating context with the operator agent's memory. Use when the user runs this skill, asks to set up the operator or create their operator agent, or chooses setup when the cloud-operator skill asks. Don't use for ordinary GCP requests or installing gcloud.
developer-device-platform-device-run
Runs automated, non-interactive jobs on remote Android devices with Developer Device Platform (DDP) Device Run using `gcloud beta device-run sessions`: native Android executables or shell scripts (for example CLI tools, custom runners, test harnesses or benchmarks) and Android instrumentation tests, on one or many devices, then waits for completion and collects per-device outputs and logs. Use when running a binary or test suite on remote devices without an interactive connection. Don't use it for interactive device reservations, adb tunnels or screen streaming (use developer-device-platform-basics), or for local devices.
compute-vm-creation
Guides the planning, configuration, and provisioning of standalone Google Compute Engine virtual machines (VMs) and bare metal (-metal) instances using gcloud CLI commands or direct MCP tool execution. Use ONLY when the user wants to create, configure, or provision a new standalone Compute Engine VM or bare metal instance from conversational intents (hardware sizing, region selection, OS image, networking, security defaults). DO NOT read or use for Google Kubernetes Engine (GKE) Autopilot or Standard clusters, Kubernetes node pools or Pods, container orchestration, managing existing VMs, Managed Instance Groups (MIGs), or Instance Templates.
agent-platform-autorater-customization
Customizes an autorater (evaluation metric) for an AI agent so its grades match the user's own judgment, starting from the agent's real trajectories. Use when a user wants a metric tailored to their agent, wants to turn failing trajectories into rubrics, says an existing autorater grades their agent wrong, or wants an autorater to agree with human labels (autorater hillclimbing, judge calibration). Don't use for quick evals with built-in metrics or for improving the agent itself (use agent-platform-eval-flywheel).
iap-agent-platform-troubleshooting
Diagnoses Identity-Aware Proxy (IAP) denials of outbound requests from Gemini Enterprise Agent Platform agents that run on Agent Runtime and send traffic through Agent Gateway. Use when an agent request fails with "Egress request is not authorized", when Cloud Audit Logs show an IAP egressViaIAP decision with granted false, when IAP audit entries are missing while debugging agent egress, or when Access policies, IAP allow policies, policy bindings, or Agent Registry entries might block agent egress. Don't use for Gemini model IAM permission errors (such as aiplatform.endpoints.predict), Agent Platform API errors, or Agent Runtime deployment or 500 errors (use agent-platform-troubleshooting instead). Don't use for setting up, creating, or connecting an Agent Gateway, or for IAP web apps, IAP TCP forwarding, or SSH.