[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6994E7EDJRN5X4QHZP55QXN":3,"skill-same:01M6994E7EDJRN5X4QHZP55QXN":90},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":41,"linkOnly":42,"markdownContent":43,"markdownBody":44,"frontMatter":45,"refName":49,"commitSha":50,"skillDirectory":51,"forksCount":52,"repositoryHtmlUrl":53,"repositoryHomepageUrl":6,"ownerHtmlUrl":54,"ownerBlogUrl":55,"ownerDescription":56,"updatedAt":57,"install":58,"resources":71},"01M6994E7EDJRN5X4QHZP55QXN","gke-observability",null,"Configures GKE observability, including Cloud Logging, Cloud Monitoring, and managed Prometheus. Use when configuring GKE monitoring, setting up GKE logging, or configuring Prometheus metrics collection, to diagnose GKE control plane, API server, admission webhook, etcd storage, node registration or label, and pod eviction, preemption, or scheduling issues using GKE metrics, Kubernetes events, and Cloud Logging, and to troubleshoot Managed Service for Prometheus (GMP) issues such as missing metrics, unhealthy scrape targets, PodMonitoring misconfiguration, rule\u002Falert evaluation failures, and monitoring permission errors. Don't use to configure local application logging frameworks or external APMs outside GKE.","google\u002Fskills","google","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F1342004?v=4","01M698VYEFDM0VJBZDYYJ2T83S","Google","01M698WR0H76ENHE587T9EJT89",24122,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fgke-observability\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40],"gke","observability","gke observability","when","managed","plane","errors","Evaluation","node","storage","etcd","server","monitoring","api","scrape","collection","configure","local","issues","up","webhook","Managed Service","2026-11-10 10:12:30",false,"---\nname: gke-observability\ndescription: >-\n  Configures GKE observability, including Cloud Logging, Cloud Monitoring, and\n  managed Prometheus. Use when configuring GKE monitoring, setting up GKE\n  logging, or configuring Prometheus metrics collection, to diagnose GKE control\n  plane, API server, admission webhook, etcd storage, node registration or\n  label, and pod eviction, preemption, or scheduling issues using GKE metrics,\n  Kubernetes events, and Cloud Logging, and to troubleshoot Managed Service for\n  Prometheus (GMP) issues such as missing metrics, unhealthy scrape targets,\n  PodMonitoring misconfiguration, rule\u002Falert evaluation failures, and monitoring\n  permission errors. Don't use to configure local application logging frameworks\n  or external APMs outside GKE.\nmetadata:\n  version: \"1.2.1\"\n  category: CloudObservabilityAndMonitoring\n---\n\n# GKE Observability\n\nThis reference covers monitoring, logging, and metrics configuration and\ntroubleshooting for Google Kubernetes Engine (GKE). The golden path enables\ncomprehensive observability including control-plane metrics and Managed Service\nfor Prometheus (GMP).\n\n> **MCP Tools:** `get_cluster`, `list_k8s_events`, `get_k8s_logs`,\n> `get_k8s_cluster_info`, `describe_k8s_resource`. **CLI-only:** `gcloud\n> container clusters update --monitoring=...`, `gcloud logging read`\n\n## Usage Attribution\n\nDo not run live `kubectl`, `gcloud`, or `curl` commands unless the user names a\ntarget cluster\u002Fproject or asks to inspect a live cluster (output commands and\nqueries with placeholders, and skip [Usage Attribution](references\u002Fattribution.md)).\nBefore running any live command, read [Usage Attribution](references\u002Fattribution.md),\nprefix `gcloud` with `CLOUDSDK_METRICS_REQUEST_ATTRIBUTION=\"agent-skills\u002F1.0 (skill:gke-observability)\"`,\nand append `agent-skills\u002F1.0 (skill:gke-observability)` to `User-Agent` headers\n(never in generated scripts, Terraform, or user-facing code).\n\n## Sub-Skill Reference Routing (Progressive Disclosure)\n\n> **Complete Reference Index** (no `scripts\u002F` directory or unlisted reference\n> files exist — do not run `ls` on the skill folder; read only the matching\n> reference below when answering deep diagnostic, metric, log, or GMP questions):\n\n1.  **[Control-Plane Metrics, Storage, Webhooks & Capacity](references\u002Fcontrol-plane-metrics-and-storage.md)**:\n    Read for GKE control-plane (KCP) VM CPU or RAM\u002Fmemory usage questions (not\n    publicly exposed) and customer-visible alternatives; cluster etcd \u002F\n    etcd-compatible storage left, quota & per-resource scalability limits;\n    admission webhook latency, system namespace\u002FLease\u002Fupgrade exclusions,\n    `APIService` discovery bloat, client discovery caching bursts\n    (`\u002Fapis\u002Fcustom.metrics.k8s.io`, `\u002Fapis\u002Fmetrics.k8s.io`) & proactive GKE\n    Recommender insights; watch storms; controller-manager workqueue & scheduler\n    lag; or control-plane headroom, VM autoscaling & node-count pre-scaling,\n    network egress, connections (`65k`, no per-replica metric) & Protobuf.\n2.  **[Cloud Logging, Audit Attribution, APF & API Server Traces](references\u002Flogging-audit-and-apiserver-traces.md)**:\n    Read for `kube-apiserver` slow-request `Trace[` logs (`trace.go` & KEP-647);\n    `httplog.go` client load (`user_agent`, `verb`, `apf_iseats`); Cloud Audit\n    Log noisy-neighbor, heavy `LIST` & unindexed CRD `fieldSelector` scan\n    tail-latency attribution; API Priority and Fairness (APF) `429`\u002F`504`\u002F`404`\n    throttling; `409 Conflict` hot objects (`ConfigMap` -> `Lease` heartbeats) &\n    `RenewDeadline \u002F 2` leader-election timeouts; `pods\u002Fexec` saturation,\n    Konnectivity (`No agent available`), disabled Cloud KMS keys, or in-cluster\n    GCP IAM OAuth token webhook authorizer saturation.\n3.  **[Cluster\u002FNode\u002FPod Health, Maintenance & Managed Prometheus (GMP)](references\u002Fcluster-node-pod-health-and-gmp.md)**:\n    Read for control-plane availability (`kubernetes.io\u002Fcluster\u002Fkcp_healthchecks`\n    rate ratio & ruling out KCP outages); maintenance windows\n    (`in_maintenance_window`), zonal resize downtime & cluster access errors\n    (`EOF` \u002F `masterAuthorizedNetworksConfig`); node readiness, registration\n    (`AUTO_REPAIR_NODES`, Private Google Access) & persistent node-pool labels;\n    pod startup timelines, crashing pod logs, `tolerationSeconds: 0` mass\n    eviction & Autopilot preemption; or multi-burn-rate SLO alerting\n    (`GlobalRules`) & GMP \u002F Grafana \u002F `PodMonitoring` manifests & troubleshooting.\n\n## Golden Path Observability Defaults\n\nSetting                                             | Golden Path Value                                                                                                                                   | Notes\n--------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | -----\n`loggingConfig` components                          | SYSTEM_COMPONENTS, WORKLOADS                                                                                                                        | Full workload logging\n`monitoringConfig` components                       | SYSTEM_COMPONENTS, STORAGE, POD, DEPLOYMENT, STATEFULSET, DAEMONSET, HPA, JOBSET, CADVISOR, KUBELET, DCGM, APISERVER, SCHEDULER, CONTROLLER_MANAGER | Full suite including control-plane\n`managedPrometheusConfig.enabled`                   | `true`                                                                                                                                              | Google-managed Prometheus\n`advancedDatapathObservabilityConfig.enableMetrics` | `true`                                                                                                                                              | Dataplane V2 flow metrics\n`loggingService`                                    | `logging.googleapis.com\u002Fkubernetes`                                                                                                                 | Cloud Logging\n`monitoringService`                                 | `monitoring.googleapis.com\u002Fkubernetes`                                                                                                              | Cloud Monitoring\n\n## Enabling Full Monitoring & Logging\n\nInclude all four caveats whenever providing a `--monitoring` or `--logging`\ncommand so users avoid accidental metric loss, surprise billing, or silent IAM\nfailures:\n\n1.  **Control-plane metrics and logs are NOT enabled by default.** State this\n    outright: `API_SERVER`, `SCHEDULER`, and `CONTROLLER_MANAGER` are off on\n    every new cluster (as are `DCGM`, `CADVISOR`, `KUBELET`, and kube-state\n    `POD`, `DEPLOYMENT`, `STATEFULSET`, `DAEMONSET`, `HPA`, `STORAGE`,\n    `JOBSET`). Only `SYSTEM` is enabled by default. Link to\n    https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcontrol-plane-metrics\n    and\n    https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fview-logs#control_plane_logs.\n2.  **The flag replaces, it does not append.** Passing `--monitoring` or\n    `--logging` overrides the existing component list; pass the full desired\n    list including `SYSTEM`.\n3.  **Control-plane and kube-state metrics bill per sample ingested** via\n    Managed Service for Prometheus (GMP). Mention the cost impact on large\n    clusters; use `--monitoring=SYSTEM` in non-production to reduce cost.\n4.  **Verify APIs and IAM permissions (`PermissionDenied`).** Check `gcloud\n    container clusters describe {cluster_name} --region {region}\n    --format=\"yaml(loggingConfig,monitoringConfig)\"`, ensure\n    `logging.googleapis.com` and `monitoring.googleapis.com` are enabled, and\n    verify the node service account \u002F GKE service agent has\n    `roles\u002Fmonitoring.metricWriter` and\n    `roles\u002Fcontainer.defaultNodeServiceAgent`.\n\n> **CLI flag vs. API enum spelling (mixing them fails the command):**\n>\n> Component      | `gcloud --monitoring=` | `monitoringConfig` API enum\n> -------------- | ---------------------- | ---------------------------\n> System         | `SYSTEM`               | `SYSTEM_COMPONENTS`\n> API server     | `API_SERVER`           | `APISERVER`\n> Controller mgr | `CONTROLLER_MANAGER`   | `CONTROLLER_MANAGER`\n\n```bash\n# Enable golden path monitoring and control-plane logging suite\ngcloud container clusters update {cluster_name} --region {region} \\\n  --monitoring=SYSTEM,API_SERVER,SCHEDULER,CONTROLLER_MANAGER,STORAGE,POD,DEPLOYMENT,STATEFULSET,DAEMONSET,HPA,JOBSET,CADVISOR,KUBELET,DCGM \\\n  --logging=SYSTEM,WORKLOAD,API_SERVER,SCHEDULER,CONTROLLER_MANAGER \\\n  --quiet\n\n# Enable Managed Prometheus and Dataplane V2 flow\u002Fpolicy observability\ngcloud container clusters update {cluster_name} --region {region} \\\n  --enable-managed-prometheus \\\n  --enable-dataplane-v2-flow-observability \\\n  --quiet\n```\n\n## Core GKE Metrics & Alerting Baseline\n\nExplicitly name **Google Cloud Monitoring** when proposing dashboards and\nalerts, and cover cluster\u002Fnode CPU and memory capacity, pod restarts\n(`CrashLoopBackOff`), volume space exhaustion, and API server latency\n(`apiserver_request_duration_seconds`). Direct control-plane VM CPU\u002FRAM metrics\nare not publicly exposed — read\n[Control-Plane Metrics, Storage, Webhooks & Capacity](references\u002Fcontrol-plane-metrics-and-storage.md)\nwhenever asked about control-plane VM CPU\u002Fmemory or deep control-plane metrics.\n\nMetric                                           | Source             | Use \u002F Alert Threshold\n------------------------------------------------ | ------------------ | ---------------------\n`apiserver_request_duration_seconds`             | API Server (GMP)   | Control-plane latency (Alert: P99 > 5s)\n`scheduler_scheduling_attempt_duration_seconds`  | Scheduler (GMP)    | Current GKE scheduling latency (`scheduler_schedule_attempts_total{result=\"error\"} > 0`)\n`container_cpu_usage_seconds_total`              | cAdvisor (Node)    | Worker-node container CPU usage\n`container_memory_working_set_bytes`             | cAdvisor (Node)    | Worker-node container memory usage\n`kubernetes.io\u002Fnode\u002Fcpu\u002Fcore_usage_time`         | Cloud Monitoring   | Node CPU usage (`kubernetes.io\u002Fnode\u002Fmemory\u002Fused_bytes` for memory)\n`kube_pod_status_phase`                          | kube-state-metrics | Pod lifecycle phase (requires deploying GMP or self-managed kube-state-metrics)\n`kube_pod_container_status_restarts_total`       | kube-state-metrics | Pod crash loops (Alert: > 5 in 10m; requires kube-state-metrics)\n`kube_node_status_condition`                     | kube-state-metrics | Node readiness (`condition=\"Ready\", status!=\"True\"`; or `kubernetes.io\u002Fnode\u002Fstatus_condition`)\n`kubelet_volume_stats_used_bytes \u002F capacity`     | Kubelet            | PVC capacity utilization (Alert: > 85%)\n`DCGM_FI_DEV_GPU_UTIL`                           | DCGM (GMP)         | GPU utilization (Alert: > 95% sustained)\n\n## Live Resource Usage & Cloud Logging Queries\n\nUse `kubectl top` for live usage (no MCP or `gcloud` equivalent):\n\n```bash\nkubectl top pods --all-namespaces --sort-by=cpu\nkubectl top nodes\nkubectl top pods --containers -n {namespace}\n```\n\nUse `gcloud logging read` (or Cloud Logging Log Explorer) with LQL filters:\n\n```bash\n# Control-plane component logs (requires API_SERVER \u002F SCHEDULER \u002F CONTROLLER_MANAGER enabled)\ngcloud logging read 'resource.type=\"k8s_control_plane_component\" AND resource.labels.cluster_name=\"{cluster_name}\"' --project {project_id} --limit 50 --quiet\n\n# Container errors: resource.type=\"k8s_container\" AND resource.labels.container_name=\"{container_name}\" AND severity>=ERROR\n# OOMKilled events: resource.type=\"k8s_event\" AND jsonPayload.reason=\"OOMKilling\"\n# FailedScheduling: resource.type=\"k8s_event\" AND jsonPayload.reason=\"FailedScheduling\"\n# Cloud Audit Logs: resource.type=\"k8s_cluster\" AND logName:\"cloudaudit.googleapis.com\"\n# Dataplane V2 network policy \u002F packet drop logs: logName:\"policy_action\" AND jsonPayload.disposition=\"DENY\"\n```\n\n## Distributed Tracing, Profiling & Recent Additions\n\n-   **Cloud Trace & OpenTelemetry**: Deploy the OpenTelemetry Collector\n    (`googlecloud` exporter with Workload Identity) or enable **Managed\n    OpenTelemetry for GKE (Preview)** on GKE 1.34.1-gke.2178000+ (`gcloud beta\n    container clusters update {cluster_name} --region {region}\n    --managed-otel-scope=COLLECTION_AND_INSTRUMENTATION_COMPONENTS`).\n-   **Cloud Profiler**: Continuous low-overhead production CPU\u002Fmemory profiling.\n-   **PSI (Pressure Stall Information) metrics**: cAdvisor\n    `container_pressure_{cpu,memory,io}_{waiting,stalled}_seconds_total` via GMP\n    `ClusterNodeMonitoring` (GKE 1.35+).\n\n## Supporting Links\n\n-   [GKE System Metrics](https:\u002F\u002Fdocs.cloud.google.com\u002Fmonitoring\u002Fapi\u002Fmetrics_kubernetes)\n-   [GKE Control-Plane Metrics](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcontrol-plane-metrics)\n-   [GKE Control-Plane Logs](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fview-logs#control_plane_logs)\n-   [Google Cloud Managed Service for Prometheus](https:\u002F\u002Fcloud.google.com\u002Fstackdriver\u002Fdocs\u002Fmanaged-prometheus)\n-   [Troubleshoot Managed Service for Prometheus](https:\u002F\u002Fdocs.cloud.google.com\u002Fstackdriver\u002Fdocs\u002Fmanaged-prometheus\u002Ftroubleshooting.md.txt)\n-   [Rule Evaluation (Rules \u002F ClusterRules \u002F GlobalRules)](https:\u002F\u002Fdocs.cloud.google.com\u002Fstackdriver\u002Fdocs\u002Fmanaged-prometheus\u002Frules-managed.md.txt)\n-   [Cloud Logging Query Language (LQL)](https:\u002F\u002Fcloud.google.com\u002Flogging\u002Fdocs\u002Fview\u002Flogging-query-language)\n-   [Google Cloud Monitoring Alerts](https:\u002F\u002Fcloud.google.com\u002Fmonitoring\u002Falerts)\n","# GKE Observability\n\nThis reference covers monitoring, logging, and metrics configuration and\ntroubleshooting for Google Kubernetes Engine (GKE). The golden path enables\ncomprehensive observability including control-plane metrics and Managed Service\nfor Prometheus (GMP).\n\n> **MCP Tools:** `get_cluster`, `list_k8s_events`, `get_k8s_logs`,\n> `get_k8s_cluster_info`, `describe_k8s_resource`. **CLI-only:** `gcloud\n> container clusters update --monitoring=...`, `gcloud logging read`\n\n## Usage Attribution\n\nDo not run live `kubectl`, `gcloud`, or `curl` commands unless the user names a\ntarget cluster\u002Fproject or asks to inspect a live cluster (output commands and\nqueries with placeholders, and skip [Usage Attribution](references\u002Fattribution.md)).\nBefore running any live command, read [Usage Attribution](references\u002Fattribution.md),\nprefix `gcloud` with `CLOUDSDK_METRICS_REQUEST_ATTRIBUTION=\"agent-skills\u002F1.0 (skill:gke-observability)\"`,\nand append `agent-skills\u002F1.0 (skill:gke-observability)` to `User-Agent` headers\n(never in generated scripts, Terraform, or user-facing code).\n\n## Sub-Skill Reference Routing (Progressive Disclosure)\n\n> **Complete Reference Index** (no `scripts\u002F` directory or unlisted reference\n> files exist — do not run `ls` on the skill folder; read only the matching\n> reference below when answering deep diagnostic, metric, log, or GMP questions):\n\n1.  **[Control-Plane Metrics, Storage, Webhooks & Capacity](references\u002Fcontrol-plane-metrics-and-storage.md)**:\n    Read for GKE control-plane (KCP) VM CPU or RAM\u002Fmemory usage questions (not\n    publicly exposed) and customer-visible alternatives; cluster etcd \u002F\n    etcd-compatible storage left, quota & per-resource scalability limits;\n    admission webhook latency, system namespace\u002FLease\u002Fupgrade exclusions,\n    `APIService` discovery bloat, client discovery caching bursts\n    (`\u002Fapis\u002Fcustom.metrics.k8s.io`, `\u002Fapis\u002Fmetrics.k8s.io`) & proactive GKE\n    Recommender insights; watch storms; controller-manager workqueue & scheduler\n    lag; or control-plane headroom, VM autoscaling & node-count pre-scaling,\n    network egress, connections (`65k`, no per-replica metric) & Protobuf.\n2.  **[Cloud Logging, Audit Attribution, APF & API Server Traces](references\u002Flogging-audit-and-apiserver-traces.md)**:\n    Read for `kube-apiserver` slow-request `Trace[` logs (`trace.go` & KEP-647);\n    `httplog.go` client load (`user_agent`, `verb`, `apf_iseats`); Cloud Audit\n    Log noisy-neighbor, heavy `LIST` & unindexed CRD `fieldSelector` scan\n    tail-latency attribution; API Priority and Fairness (APF) `429`\u002F`504`\u002F`404`\n    throttling; `409 Conflict` hot objects (`ConfigMap` -> `Lease` heartbeats) &\n    `RenewDeadline \u002F 2` leader-election timeouts; `pods\u002Fexec` saturation,\n    Konnectivity (`No agent available`), disabled Cloud KMS keys, or in-cluster\n    GCP IAM OAuth token webhook authorizer saturation.\n3.  **[Cluster\u002FNode\u002FPod Health, Maintenance & Managed Prometheus (GMP)](references\u002Fcluster-node-pod-health-and-gmp.md)**:\n    Read for control-plane availability (`kubernetes.io\u002Fcluster\u002Fkcp_healthchecks`\n    rate ratio & ruling out KCP outages); maintenance windows\n    (`in_maintenance_window`), zonal resize downtime & cluster access errors\n    (`EOF` \u002F `masterAuthorizedNetworksConfig`); node readiness, registration\n    (`AUTO_REPAIR_NODES`, Private Google Access) & persistent node-pool labels;\n    pod startup timelines, crashing pod logs, `tolerationSeconds: 0` mass\n    eviction & Autopilot preemption; or multi-burn-rate SLO alerting\n    (`GlobalRules`) & GMP \u002F Grafana \u002F `PodMonitoring` manifests & troubleshooting.\n\n## Golden Path Observability Defaults\n\nSetting                                             | Golden Path Value                                                                                                                                   | Notes\n--------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | -----\n`loggingConfig` components                          | SYSTEM_COMPONENTS, WORKLOADS                                                                                                                        | Full workload logging\n`monitoringConfig` components                       | SYSTEM_COMPONENTS, STORAGE, POD, DEPLOYMENT, STATEFULSET, DAEMONSET, HPA, JOBSET, CADVISOR, KUBELET, DCGM, APISERVER, SCHEDULER, CONTROLLER_MANAGER | Full suite including control-plane\n`managedPrometheusConfig.enabled`                   | `true`                                                                                                                                              | Google-managed Prometheus\n`advancedDatapathObservabilityConfig.enableMetrics` | `true`                                                                                                                                              | Dataplane V2 flow metrics\n`loggingService`                                    | `logging.googleapis.com\u002Fkubernetes`                                                                                                                 | Cloud Logging\n`monitoringService`                                 | `monitoring.googleapis.com\u002Fkubernetes`                                                                                                              | Cloud Monitoring\n\n## Enabling Full Monitoring & Logging\n\nInclude all four caveats whenever providing a `--monitoring` or `--logging`\ncommand so users avoid accidental metric loss, surprise billing, or silent IAM\nfailures:\n\n1.  **Control-plane metrics and logs are NOT enabled by default.** State this\n    outright: `API_SERVER`, `SCHEDULER`, and `CONTROLLER_MANAGER` are off on\n    every new cluster (as are `DCGM`, `CADVISOR`, `KUBELET`, and kube-state\n    `POD`, `DEPLOYMENT`, `STATEFULSET`, `DAEMONSET`, `HPA`, `STORAGE`,\n    `JOBSET`). Only `SYSTEM` is enabled by default. Link to\n    https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcontrol-plane-metrics\n    and\n    https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fview-logs#control_plane_logs.\n2.  **The flag replaces, it does not append.** Passing `--monitoring` or\n    `--logging` overrides the existing component list; pass the full desired\n    list including `SYSTEM`.\n3.  **Control-plane and kube-state metrics bill per sample ingested** via\n    Managed Service for Prometheus (GMP). Mention the cost impact on large\n    clusters; use `--monitoring=SYSTEM` in non-production to reduce cost.\n4.  **Verify APIs and IAM permissions (`PermissionDenied`).** Check `gcloud\n    container clusters describe {cluster_name} --region {region}\n    --format=\"yaml(loggingConfig,monitoringConfig)\"`, ensure\n    `logging.googleapis.com` and `monitoring.googleapis.com` are enabled, and\n    verify the node service account \u002F GKE service agent has\n    `roles\u002Fmonitoring.metricWriter` and\n    `roles\u002Fcontainer.defaultNodeServiceAgent`.\n\n> **CLI flag vs. API enum spelling (mixing them fails the command):**\n>\n> Component      | `gcloud --monitoring=` | `monitoringConfig` API enum\n> -------------- | ---------------------- | ---------------------------\n> System         | `SYSTEM`               | `SYSTEM_COMPONENTS`\n> API server     | `API_SERVER`           | `APISERVER`\n> Controller mgr | `CONTROLLER_MANAGER`   | `CONTROLLER_MANAGER`\n\n```bash\n# Enable golden path monitoring and control-plane logging suite\ngcloud container clusters update {cluster_name} --region {region} \\\n  --monitoring=SYSTEM,API_SERVER,SCHEDULER,CONTROLLER_MANAGER,STORAGE,POD,DEPLOYMENT,STATEFULSET,DAEMONSET,HPA,JOBSET,CADVISOR,KUBELET,DCGM \\\n  --logging=SYSTEM,WORKLOAD,API_SERVER,SCHEDULER,CONTROLLER_MANAGER \\\n  --quiet\n\n# Enable Managed Prometheus and Dataplane V2 flow\u002Fpolicy observability\ngcloud container clusters update {cluster_name} --region {region} \\\n  --enable-managed-prometheus \\\n  --enable-dataplane-v2-flow-observability \\\n  --quiet\n```\n\n## Core GKE Metrics & Alerting Baseline\n\nExplicitly name **Google Cloud Monitoring** when proposing dashboards and\nalerts, and cover cluster\u002Fnode CPU and memory capacity, pod restarts\n(`CrashLoopBackOff`), volume space exhaustion, and API server latency\n(`apiserver_request_duration_seconds`). Direct control-plane VM CPU\u002FRAM metrics\nare not publicly exposed — read\n[Control-Plane Metrics, Storage, Webhooks & Capacity](references\u002Fcontrol-plane-metrics-and-storage.md)\nwhenever asked about control-plane VM CPU\u002Fmemory or deep control-plane metrics.\n\nMetric                                           | Source             | Use \u002F Alert Threshold\n------------------------------------------------ | ------------------ | ---------------------\n`apiserver_request_duration_seconds`             | API Server (GMP)   | Control-plane latency (Alert: P99 > 5s)\n`scheduler_scheduling_attempt_duration_seconds`  | Scheduler (GMP)    | Current GKE scheduling latency (`scheduler_schedule_attempts_total{result=\"error\"} > 0`)\n`container_cpu_usage_seconds_total`              | cAdvisor (Node)    | Worker-node container CPU usage\n`container_memory_working_set_bytes`             | cAdvisor (Node)    | Worker-node container memory usage\n`kubernetes.io\u002Fnode\u002Fcpu\u002Fcore_usage_time`         | Cloud Monitoring   | Node CPU usage (`kubernetes.io\u002Fnode\u002Fmemory\u002Fused_bytes` for memory)\n`kube_pod_status_phase`                          | kube-state-metrics | Pod lifecycle phase (requires deploying GMP or self-managed kube-state-metrics)\n`kube_pod_container_status_restarts_total`       | kube-state-metrics | Pod crash loops (Alert: > 5 in 10m; requires kube-state-metrics)\n`kube_node_status_condition`                     | kube-state-metrics | Node readiness (`condition=\"Ready\", status!=\"True\"`; or `kubernetes.io\u002Fnode\u002Fstatus_condition`)\n`kubelet_volume_stats_used_bytes \u002F capacity`     | Kubelet            | PVC capacity utilization (Alert: > 85%)\n`DCGM_FI_DEV_GPU_UTIL`                           | DCGM (GMP)         | GPU utilization (Alert: > 95% sustained)\n\n## Live Resource Usage & Cloud Logging Queries\n\nUse `kubectl top` for live usage (no MCP or `gcloud` equivalent):\n\n```bash\nkubectl top pods --all-namespaces --sort-by=cpu\nkubectl top nodes\nkubectl top pods --containers -n {namespace}\n```\n\nUse `gcloud logging read` (or Cloud Logging Log Explorer) with LQL filters:\n\n```bash\n# Control-plane component logs (requires API_SERVER \u002F SCHEDULER \u002F CONTROLLER_MANAGER enabled)\ngcloud logging read 'resource.type=\"k8s_control_plane_component\" AND resource.labels.cluster_name=\"{cluster_name}\"' --project {project_id} --limit 50 --quiet\n\n# Container errors: resource.type=\"k8s_container\" AND resource.labels.container_name=\"{container_name}\" AND severity>=ERROR\n# OOMKilled events: resource.type=\"k8s_event\" AND jsonPayload.reason=\"OOMKilling\"\n# FailedScheduling: resource.type=\"k8s_event\" AND jsonPayload.reason=\"FailedScheduling\"\n# Cloud Audit Logs: resource.type=\"k8s_cluster\" AND logName:\"cloudaudit.googleapis.com\"\n# Dataplane V2 network policy \u002F packet drop logs: logName:\"policy_action\" AND jsonPayload.disposition=\"DENY\"\n```\n\n## Distributed Tracing, Profiling & Recent Additions\n\n-   **Cloud Trace & OpenTelemetry**: Deploy the OpenTelemetry Collector\n    (`googlecloud` exporter with Workload Identity) or enable **Managed\n    OpenTelemetry for GKE (Preview)** on GKE 1.34.1-gke.2178000+ (`gcloud beta\n    container clusters update {cluster_name} --region {region}\n    --managed-otel-scope=COLLECTION_AND_INSTRUMENTATION_COMPONENTS`).\n-   **Cloud Profiler**: Continuous low-overhead production CPU\u002Fmemory profiling.\n-   **PSI (Pressure Stall Information) metrics**: cAdvisor\n    `container_pressure_{cpu,memory,io}_{waiting,stalled}_seconds_total` via GMP\n    `ClusterNodeMonitoring` (GKE 1.35+).\n\n## Supporting Links\n\n-   [GKE System Metrics](https:\u002F\u002Fdocs.cloud.google.com\u002Fmonitoring\u002Fapi\u002Fmetrics_kubernetes)\n-   [GKE Control-Plane Metrics](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcontrol-plane-metrics)\n-   [GKE Control-Plane Logs](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fview-logs#control_plane_logs)\n-   [Google Cloud Managed Service for Prometheus](https:\u002F\u002Fcloud.google.com\u002Fstackdriver\u002Fdocs\u002Fmanaged-prometheus)\n-   [Troubleshoot Managed Service for Prometheus](https:\u002F\u002Fdocs.cloud.google.com\u002Fstackdriver\u002Fdocs\u002Fmanaged-prometheus\u002Ftroubleshooting.md.txt)\n-   [Rule Evaluation (Rules \u002F ClusterRules \u002F GlobalRules)](https:\u002F\u002Fdocs.cloud.google.com\u002Fstackdriver\u002Fdocs\u002Fmanaged-prometheus\u002Frules-managed.md.txt)\n-   [Cloud Logging Query Language (LQL)](https:\u002F\u002Fcloud.google.com\u002Flogging\u002Fdocs\u002Fview\u002Flogging-query-language)\n-   [Google Cloud Monitoring Alerts](https:\u002F\u002Fcloud.google.com\u002Fmonitoring\u002Falerts)\n",{"name":5,"metadata":46,"description":7},{"version":47,"category":48},"1.2.1","CloudObservabilityAndMonitoring","main","071c4f7eba880767a647ce778650db8dbf7e16fc","skills\u002Fcloud\u002Fgke-observability",2005,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills","https:\u002F\u002Fgithub.com\u002Fgoogle","https:\u002F\u002Fopensource.google\u002F","Google ❤️ Open Source","2026-11-14 01:51:35",{"source":8,"skillName":5,"commands":59,"prompt":69,"zipAvailable":70},[60,63,66],{"tool":61,"command":62},"npx","npx skills add google\u002Fskills --skill gke-observability",{"tool":64,"command":65},"bunx","bunx skills add google\u002Fskills --skill gke-observability",{"tool":67,"command":68},"pnpm","pnpm dlx skills add google\u002Fskills --skill gke-observability","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fgke-observability\u002FSKILL.md",true,[72,78,82,86],{"path":73,"resourceType":74,"resourceTypeDesc":75,"sizeBytes":76,"sourceHtmlUrl":77},"references\u002Fattribution.md","REFERENCE","参考文档",1891,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fgke-observability\u002Freferences\u002Fattribution.md",{"path":79,"resourceType":74,"resourceTypeDesc":75,"sizeBytes":80,"sourceHtmlUrl":81},"references\u002Fcluster-node-pod-health-and-gmp.md",13258,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fgke-observability\u002Freferences\u002Fcluster-node-pod-health-and-gmp.md",{"path":83,"resourceType":74,"resourceTypeDesc":75,"sizeBytes":84,"sourceHtmlUrl":85},"references\u002Fcontrol-plane-metrics-and-storage.md",14534,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fgke-observability\u002Freferences\u002Fcontrol-plane-metrics-and-storage.md",{"path":87,"resourceType":74,"resourceTypeDesc":75,"sizeBytes":88,"sourceHtmlUrl":89},"references\u002Flogging-audit-and-apiserver-traces.md",14126,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fgke-observability\u002Freferences\u002Flogging-audit-and-apiserver-traces.md",[91,129,159,186,213,236],{"skillNo":92,"name":93,"nameZh":6,"description":94,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":95,"categories":96,"keywords":105,"publishedAt":41},"01M6998S2EZKD3GEJ7F943RZTE","sign-in-with-google-web","Implement, configure, and secure Sign In With Google (SiwG) using Google Identity Services (GIS \u002F `https:\u002F\u002Faccounts.google.com\u002Fgsi\u002Fclient`) across web architectures. Use when creating Google sign-in buttons, implementing Google One Tap with FedCM, integrating GIS in React\u002FNext.js\u002FAngular\u002FHTML, verifying ID tokens on backend runtimes (Python\u002FNode.js\u002FGo\u002FJava), enforcing Google Workspace domain restrictions (`hd`), securing client-side ID tokens via WebCrypto nonces or encrypted IndexedDB, embedding in cross-origin iframes via the Intermediate Iframe API (`https:\u002F\u002Faccounts.google.com\u002Fgsi\u002Fintermediate`), configuring Content Security Policy (CSP), COOP, and Permissions Policy (`identity-credentials-get`) headers, or handling sign-out and revocation. Don't use for legacy Google Sign-In (`gapi.auth2`), Google Cloud IAM service accounts, or Android\u002FiOS native Credential Manager.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fidentity\u002Fsign-in-with-google-web\u002FSKILL.md",[97,102],{"categoryNo":98,"categoryName":99,"taxonomyType":100,"taxonomyTypeDesc":101},"01JBSEEDCAT000000000000009","前端开发","SCENARIO","使用场景",{"categoryNo":103,"categoryName":104,"taxonomyType":100,"taxonomyTypeDesc":101},"01JBSEEDCAT000000000000010","后端与接口",[9,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128],"sign in with google web","sign","with","web","android","go","security","google workspace","google cloud","implement","policy","python","implementing","react","iam","permissions","buttons","secure","manager","identity","architectures","domain","runtimes",{"skillNo":130,"name":131,"nameZh":6,"description":132,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":133,"categories":134,"keywords":144,"publishedAt":41},"01M6998PHPDCAD684VW5WRG83D","dpop-adoption","Implement and debug OAuth 2.0 DPoP (RFC 9449) refresh token sender-constraining for WebCrypto, Node.js ES6, and browser runtimes integrating with Google's OAuth platform. Use when configuring non-extractable asymmetric key pairs (P-256), generating DPoP Proof JWTs for authorization code exchange and token refresh, or handling 400 use_dpop_nonce challenge retry loops at oauth2.googleapis.com\u002Ftoken. Don't use for unconstrained OAuth 2.0 flows (where refresh tokens are not bound to a client key pair), or for Google Cloud IAM \u002F service account authentication.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fidentity\u002Fdpop-adoption\u002FSKILL.md",[135,136,139],{"categoryNo":103,"categoryName":104,"taxonomyType":100,"taxonomyTypeDesc":101},{"categoryNo":137,"categoryName":138,"taxonomyType":100,"taxonomyTypeDesc":101},"01JBSEEDCAT000000000000023","Skill与智能体开发",{"categoryNo":140,"categoryName":141,"taxonomyType":142,"taxonomyTypeDesc":143},"01JBSEEDCAT000000000000030","开发工程师","PROFESSION","适用职业",[145,146,9,114,147,148,149,150,151,152,120,153,154,155,27,156,157,125,128,158,22],"dpop adoption","adoption","es6","retry","authentication","flows","challenge","code","browser","loops","at","token","pair","platform",{"skillNo":160,"name":161,"nameZh":6,"description":162,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":163,"categories":164,"keywords":171,"publishedAt":41},"01M6998M72TC73EEAQH4XC7EXW","retrieving-developer-knowledge","Searches, retrieves, and synthesizes official Google developer documentation across Google Cloud, AI\u002FGemini, Android, Chrome, Web, Flutter, Go, Firebase, and other Google developer platforms. Integrates with the Developer Knowledge MCP server (search_documents, get_documents, answer_query) or the Developer Knowledge REST API fallback. Use when searching for gcloud CLI commands, API syntax, IAM permissions, official documentation, architectural comparisons, or product choice overviews. Don't use for local filesystem lookups or non-Google documentation.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fdevelopers\u002Fretrieving-developer-knowledge\u002FSKILL.md",[165,166,169,170],{"categoryNo":103,"categoryName":104,"taxonomyType":100,"taxonomyTypeDesc":101},{"categoryNo":167,"categoryName":168,"taxonomyType":100,"taxonomyTypeDesc":101},"01JBSEEDCAT000000000000021","资料检索与研究",{"categoryNo":137,"categoryName":138,"taxonomyType":100,"taxonomyTypeDesc":101},{"categoryNo":140,"categoryName":141,"taxonomyType":142,"taxonomyTypeDesc":143},[172,173,174,175,114,176,177,178,179,180,36,181,182,22,183,32,184,120,185,121],"retrieving developer knowledge","knowledge","developer","MCP","mcp server","official","developer documentation","get","search","choice","documents","query","across","documentation",{"skillNo":187,"name":188,"nameZh":6,"description":189,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":190,"categories":191,"keywords":192,"publishedAt":41},"01M6998J2CZEXS9BES9S3WWFKC","migrate-to-rust","Clones, analyzes, plans, and migrates legacy codebases (C++, Fortran, Java, Python, C, Shell) from a GitHub URL or local directory to a modern, high-performance, deterministic, 100% Safe Rust architecture, tracking live telemetry across every task bucket, test failure\u002Ffix cycle, and Profile-Guided Optimization (PGO) stage, and producing a first-person AI storytelling report and interactive HTML infographic. Use when migrating a GitHub repository or local codebase to Rust, porting legacy C++, Fortran, Python, or Java workloads to memory-safe Rust, designing Struct-of-Arrays (SoA) or SIMD Rust architectures, or optimizing Rust ports with LLVM PGO. Don't use for unrelated non-Rust refactoring or single-line syntax questions.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fdevelopers\u002Fmigrate-to-rust\u002FSKILL.md",[],[193,194,195,196,197,198,199,200,117,201,202,203,204,205,206,36,207,208,209,22,210,211,184,212],"migrate to rust","rust","migrate","github","memory","performance","directory","safe","ports","llvm","soa","cycle","porting","fix","arrays","infographic","high-performance","URL","architecture","unrelated",{"skillNo":214,"name":215,"nameZh":6,"description":216,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":217,"categories":218,"keywords":222,"publishedAt":41},"01M6998GCC3K29WZ82KXMB9E02","finding-google-skills","Google platform decision and setup guidance, loaded on demand from Google's skill catalog. Use when a developer is choosing or setting up part of their stack, such as where to run a service, a database, storage, messaging, authentication, analytics, ads, or AI model serving, and a Google product is a reasonable candidate - whether or not a vendor is named - or when a request names a Google product or API. Brings in the matching Google skill so the answer can weigh Google options, their trade-offs, and when they are not the right fit. Skip when the stack is already settled on another provider and no Google product is named, or the task involves no platform choice.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fdevelopers\u002Ffinding-google-skills\u002FSKILL.md",[219,220,221],{"categoryNo":103,"categoryName":104,"taxonomyType":100,"taxonomyTypeDesc":101},{"categoryNo":137,"categoryName":138,"taxonomyType":100,"taxonomyTypeDesc":101},{"categoryNo":140,"categoryName":141,"taxonomyType":142,"taxonomyTypeDesc":143},[9,223,28,224,225,158,226,227,228,229,230,149,231,38,232,181,233,234,22,235,32],"finding google skills","as","messaging","model","trade-offs","provider","answer","fit","catalog","decision","use","database","candidate",{"skillNo":237,"name":238,"nameZh":6,"description":239,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":240,"categories":241,"keywords":244,"publishedAt":41},"01M6998DWNHTH0BS6W1CY8ZXMP","workload-manager-basics","Use this skill to manage Google Cloud Workload Manager evaluations, rules, scanned resources, and validation results by using public client libraries and the REST API. Use when you need to inspect workload best-practice rules, create and run evaluations for Google Cloud general best practices, SAP, SQL Server, or custom organizational rules, review violations, export results to BigQuery, or automate Workload Manager through client libraries because no service-specific public CLI or MCP server is available. Don't use for general Google Compute Engine instance management, VPC configuration, or standard IAM auditing.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fworkload-manager-basics\u002FSKILL.md",[242,243],{"categoryNo":103,"categoryName":104,"taxonomyType":100,"taxonomyTypeDesc":101},{"categoryNo":137,"categoryName":138,"taxonomyType":100,"taxonomyTypeDesc":101},[245,246,124,247,9,175,248,114,176,249,250,251,252,22,253,32,120,254,255,30,256],"workload manager basics","workload","basics","cli","review","vpc","resources","inspect","you","create","libraries","run"]