[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M69933KTF7WXWFGKMGHPCZY0":3,"skill-same:01M69933KTF7WXWFGKMGHPCZY0":76},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":47,"commitSha":48,"skillDirectory":49,"forksCount":50,"repositoryHtmlUrl":51,"repositoryHomepageUrl":6,"ownerHtmlUrl":52,"ownerBlogUrl":53,"ownerDescription":54,"updatedAt":55,"install":56,"resources":69},"01M69933KTF7WXWFGKMGHPCZY0","gke-ai-troubleshooting-tpu-partition-overlap",null,"Diagnose and resolve GKE TPU slice creation failures (SliceCreationFailed) caused by partition overlap conflicts where requested TPU partitions or their nodes are already used by another slice or JobSet. Use when diagnosing GKE TPU slice creation failures with SliceCreationFailed partition or node conflicts. Don't use for general slice lifecycle states or slice provisioning failures without node overlap (use gke-ai-troubleshooting-tpu-dynamic-slices-monitoring), quota exhaustion, or unresponsive TPU nodes.","google\u002Fskills","google","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F1342004?v=4","01M698VYEFDM0VJBZDYYJ2T83S","Google","01M698WR0H76ENHE587T9EJT89",24122,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fgke-ai-troubleshooting-tpu-partition-overlap\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"gke ai troubleshooting tpu partition overlap","tpu","gke","troubleshooting","partition","when","containers","slice","for","with","states","creation","partitions","quota","conflicts","general","use","lifecycle","monitoring","diagnose","2026-11-10 10:12:30",false,"---\nname: gke-ai-troubleshooting-tpu-partition-overlap\ndescription: >-\n  Diagnose and resolve GKE TPU slice creation failures (SliceCreationFailed)\n  caused by partition overlap conflicts where requested TPU partitions or their\n  nodes are already used by another slice or JobSet. Use when diagnosing GKE TPU\n  slice creation failures with SliceCreationFailed partition or node conflicts.\n  Don't use for general slice lifecycle states or slice provisioning failures\n  without node overlap (use\n  gke-ai-troubleshooting-tpu-dynamic-slices-monitoring), quota exhaustion, or\n  unresponsive TPU nodes.\nmetadata:\n  version: \"1.0.1\"\n  category: Containers\n---\n\n# Troubleshoot TPU slice creation failures caused by partition overlap\n\nA dynamic slice fails to form with the status reason `SliceCreationFailed` when,\namong other documented causes, any of its nodes are already used by another\nslice. This skill confirms the failure, rules out the other documented causes,\nfinds the slice that uses the same nodes, and routes the fix through GKE.\n\n## Attribution\n\n**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you\nMUST read [Usage Attribution](references\u002Fattribution.md). For commands executed\ndirectly in the terminal (never in generated scripts, Terraform, or user code),\nprefix `gcloud` with\n`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION=\"agent-skills\u002F1.0 (skill:gke-ai-troubleshooting-tpu-partition-overlap)\"`\nand append `agent-skills\u002F1.0 (skill:gke-ai-troubleshooting-tpu-partition-overlap)` to `User-Agent` headers.\n\n## Prerequisites\n\n- **Tools**:\n  - For installing and initializing `gcloud` (which covers authentication and\n    project selection), follow\n    [Before you begin](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt).\n  - For installing `kubectl` and required plugins, follow\n    [Install kubectl](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcluster-access-for-kubectl.md.txt)\n    and\n    [Install required plugins](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcluster-access-for-kubectl.md.txt).\n- **Cloud Billing & Project Configuration**: Ensure the project has an active\n  Cloud Billing account. Follow\n  [Before you begin](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt)\n  to initialize `gcloud` and enable the Google Kubernetes Engine API\n  (`container.googleapis.com`), ensure the Cloud Monitoring API\n  (`monitoring.googleapis.com`) is enabled, and follow\n  [Store cluster information for kubectl](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcluster-access-for-kubectl.md.txt)\n  to configure `kubectl` cluster credentials.\n- **Required IAM Roles**:\n  - Kubernetes Engine Viewer (`roles\u002Fcontainer.viewer`)\n  - Monitoring Viewer (`roles\u002Fmonitoring.viewer`)\n- **Documentation**:\n  - [Use dynamic slicing with a custom scheduler](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt)\n    (Sections: \"Before you begin\", \"Target your node pool to belong to a block\n    or a sub-block\", \"Verify the status of the nodes and the partitions\",\n    \"Create a Slice custom resource\", \"Monitor the status of the Slice custom\n    resource\", \"Delete the slice\")\n  - [Configure cluster access for kubectl](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcluster-access-for-kubectl.md.txt)\n    (Sections: \"Install kubectl\", \"Install required plugins\", \"Store cluster\n    information for kubectl\")\n  - [Slice](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Freference\u002Fcrds\u002Fslice.md.txt)\n    (Sections: \"Slice status\", \"Slice specification\")\n  - [Schedule dynamic slices with Kueue and TAS](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fuse-gke-dynamic-slicing.md.txt)\n    (Sections: \"Target your node pool to belong to a block or a sub-block\",\n    \"Create a dynamic slice with Kueue and TAS\", \"Define the partition health\n    selection\", \"Optimize workload scheduling with slice reuse\", \"Enable Slice\n    reuse\", \"Enforce slice deletion per workload\", \"Monitor the status of the\n    slice\", \"Monitor the metrics of the slice\", \"Clean up\")\n  - [GKE system metrics](https:\u002F\u002Fdocs.cloud.google.com\u002Fmonitoring\u002Fapi\u002Fmetrics_kubernetes.md.txt)\n    (Sections: \"Kubernetes metrics\")\n  - [Monitored resource types](https:\u002F\u002Fdocs.cloud.google.com\u002Fmonitoring\u002Fapi\u002Fresources.md.txt)\n    (Sections: \"k8s_node_pool\")\n  - [About GKE dynamic slicing](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fconcepts\u002Fdynamic-slicing.md.txt)\n    (Sections: \"Dynamic slicing configurations\")\n  - [Get support](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fgetting-support.md.txt)\n    (Sections: \"Before you contact support\", \"Recommendations for efficient\n    support\")\n\n> **Read-only rule**: Run read-only diagnostic commands only. Never drain,\n> delete, or re-create nodes, or run any other command that changes the cluster.\n> Give the user any fix to apply themselves.\n\nWhen you recommend a fix, link the doc section that describes it.\n\n---\n\n## Diagnostic workflow\n\n### Step 0: Collect context `[Low Risk]`\n\nCollect the target parameters:\n\n- `{project_id}`: Google Cloud project ID\n- `{cluster_name}`: GKE cluster name\n- `{location}`: Cluster region or zone\n- `{slice_name}`: Name of the Slice custom resource that failed to form\n- `{node_pool_name}`: Name of the TPU node pool (or node pools) backing the\n  partitions in `{slice_name}`, if known. If the user doesn't know\n  `{node_pool_name}`, after Step 1 run the Cloud Monitoring query in Step 3\n  for the partition IDs in `{partition_ids}` and read the `node_pool_name`\n  label on the\n  [k8s_node_pool](https:\u002F\u002Fdocs.cloud.google.com\u002Fmonitoring\u002Fapi\u002Fresources.md.txt)\n  monitored resource.\n- `{issue_time}`: Incident timestamp in RFC3339 UTC, if known. If the user\n  doesn't know `{issue_time}`, use the `Last Transition Time`\n  (`lastTransitionTime`, \"A timestamp of the most recent status change\" in\n  [Slice status](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Freference\u002Fcrds\u002Fslice.md.txt))\n  of the `Ready` condition from Step 1. By default, query Cloud Monitoring over\n  a 60-minute window around `{issue_time}`.\n\n---\n\n### Step 1: Confirm the `SliceCreationFailed` status `[Low Risk]`\n\nTo describe `{slice_name}`, follow the instructions in the section\n[Monitor the status of the Slice custom resource](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt).\n\n- In `Status` > `Conditions`, check that the `Reason` of the `Ready` condition\n  is `SliceCreationFailed`, and record its `Last Transition Time` (to use as\n  `{issue_time}` if needed).\n- Record the IDs in `Spec` > `Partition Ids` as `{partition_ids}`, the\n  `Topology`, and any `Labels` and `Annotations` on `{slice_name}`.\n- If the `Ready` reason isn't `SliceCreationFailed`, this skill doesn't apply.\n  See\n  [Slice status](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Freference\u002Fcrds\u002Fslice.md.txt)\n  for what each reason means, and hand off to\n  `gke-ai-troubleshooting-tpu-dynamic-slices-monitoring`.\n\n---\n\n### Step 2: Rule out the other documented causes `[Low Risk]`\n\nPer\n[Create a Slice custom resource](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt),\nslice creation also fails with `SliceCreationFailed` when:\n\n- The selected nodes don't exist.\n- The topology doesn't match the number of partitions. A dynamic sub-slice lists\n  exactly one partition ID. A dynamic super-slice lists `(A × B × C) \u002F 64`\n  partition IDs.\n\nIf either of those causes applies, hand off to\n`gke-ai-troubleshooting-tpu-dynamic-slices-monitoring`.\n\nOtherwise, follow\n[Verify the status of the nodes and the partitions](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt)\non `{node_pool_name}`:\n\n- Verify that the nodes exist and carry each ID in `{partition_ids}`.\n- On the nodes that carry `{partition_ids}`, record **every** partition ID of\n  **every** shape (`4x4x4`, `2x4x4`, `2x2x4`, `2x2x2`, and `2x2x1`) from the\n  `cloud.google.com\u002Fgke-tpu-partition-.*-id` labels. Per\n  [Dynamic slicing configurations](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fconcepts\u002Fdynamic-slicing.md.txt),\n  dynamic sub-slicing partitions a `4x4x4` sub-block into smaller topologies on\n  the same physical node pool, so partitions of different shapes share the same\n  nodes.\n- Read each partition's `cloud.google.com\u002Fgke-tpu-partition-.*-state` label. Per\n  [Verify the status of the nodes and the partitions](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt)\n  and\n  [Define the partition health selection](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fuse-gke-dynamic-slicing.md.txt),\n  count `HEALTHY` partitions and `DEGRADED` `4x4x4` partitions (\"still usable\n  for dynamic slice formation\"; \"Smaller topologies don't have a degraded\n  state\") as available. Partitions in `UNHEALTHY` (\"can't be used to form a\n  slice\"), `UNSET`, or `INCOMPLETE` aren't available. If a requested partition\n  is in `UNHEALTHY`, `UNSET`, or `INCOMPLETE`, suggest an available partition\n  whose nodes no other slice uses (Step 4, item 2), hand off to\n  `gke-ai-troubleshooting-tpu-dynamic-slices-monitoring`, or escalate to Cloud\n  Customer Care (Step 4, item 4).\n\n---\n\n### Step 3: Find the slice that uses the same nodes `[Low Risk]`\n\nPer\n[Create a Slice custom resource](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt),\nslice creation fails with `SliceCreationFailed` when any nodes on the custom\nresource are used by another slice. Because partitions of different shapes on\nthe same sub-block share nodes (see\n[Dynamic slicing configurations](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fconcepts\u002Fdynamic-slicing.md.txt)),\nanother slice conflicts with `{slice_name}` if it lists **any** of the partition\nIDs (of any shape) recorded on `{slice_name}`'s nodes in Step 2.\n\n1. List the `Slice` custom resources in the cluster (as shown in\n   [Delete the slice](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt))\n   and describe each one following\n   [Monitor the status of the Slice custom resource](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt)\n   to read `Spec` > `Partition Ids`, `Metadata` > `Labels`, and `Status` >\n   `Conditions` (`Ready` reason):\n   - Treat any slice other than `{slice_name}` whose `Spec` > `Partition Ids`\n     includes **any** partition ID recorded on `{slice_name}`'s nodes in Step 2\n     (whether the exact ID in `{partition_ids}` or another shape's partition ID\n     on the same nodes) as conflicting. Record its name, its `Ready` reason\n     (such as `ACTIVE`), and whether Kueue manages it (Step 4, item 1).\n2. To check partition-to-slice associations in Cloud Monitoring (including\n   around `{issue_time}`), query the `kubernetes.io\u002Faccelerator\u002Fpartition\u002Fstate`\n   metric listed in\n   [Monitor the metrics of the slice](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fuse-gke-dynamic-slicing.md.txt).\n   Per\n   [Kubernetes metrics](https:\u002F\u002Fdocs.cloud.google.com\u002Fmonitoring\u002Fapi\u002Fmetrics_kubernetes.md.txt),\n   this metric \"is 1 if the partition state is detected, 0 otherwise\", and its\n   labels include `partition_id`, `slice_id` (\"ID of the associated slice\"),\n   and `state` (\"Health of the partition (e.g., HEALTHY, UNHEALTHY)\"). Run the\n   query for each `{partition_id}` recorded on `{slice_name}`'s nodes in Step 2:\n\n```promql\nkubernetes_io:accelerator_partition_state{\n  monitored_resource=\"k8s_node_pool\",\n  cluster_name=\"{cluster_name}\",\n  partition_id=\"{partition_id}\"\n}\n```\n\n   - Read only time series whose value is `1` (where `state=\"HEALTHY\"` means the\n     partition is healthy and functional, per\n     [Verify the status of the nodes and the partitions](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt)).\n   - Because\n     [Kubernetes metrics](https:\u002F\u002Fdocs.cloud.google.com\u002Fmonitoring\u002Fapi\u002Fmetrics_kubernetes.md.txt)\n     defines `slice_id` only as \"ID of the associated slice\", always confirm the\n     conflicting slice from the `Slice` custom resources in the cluster rather\n     than from `slice_id` alone: list the `Slice` custom resources as shown in\n     [Delete the slice](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt),\n     describe each one following\n     [Monitor the status of the Slice custom resource](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt),\n     and check which slice's `Spec` > `Partition Ids` includes the partition ID\n     (or any other partition ID of any shape on the same nodes from Step 2).\n3. **Next steps based on Step 3 results**:\n   - **Another slice uses the nodes now**: Proceed to Step 4.\n   - **No other slice uses the nodes now, but the metric (value `1`) showed an\n     associated `slice_id` around `{issue_time}`**: The conflict may already be\n     over. Proceed to Step 4, item 1 (to re-create or retry `{slice_name}`) and\n     then Step 4, item 4.\n   - **Neither the current `Slice` custom resources nor the metric around\n     `{issue_time}` shows another slice**: Node overlap isn't confirmed. Hand\n     off to `gke-ai-troubleshooting-tpu-dynamic-slices-monitoring` or escalate\n     to Cloud Customer Care (Step 4, item 4).\n\n---\n\n### Step 4: Resolution `[High Risk]`\n\n> **Guardrails**:\n> - **Never** delete or change the other slice, or stop its workload, to free\n>   the nodes. Deleting a `Slice` custom resource starts dismantling that slice\n>   (`DEACTIVATING`, see\n>   [Monitor the status of the Slice custom resource](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt)).\n>   The owner of that slice decides when to release it.\n> - **Never** manually modify or delete `Slice` custom resources created by\n>   Kueue. Per\n>   [Target your node pool to belong to a block or a sub-block](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt),\n>   \"Avoid manually modifying Slice custom resources created by Kueue.\"\n> - **Never** change GKE-managed instance groups or VMs directly through Compute\n>   Engine, and never delete node pools to free a partition. Manage node pools\n>   and slices through GKE.\n\nFirst, check whether Kueue manages `{slice_name}` or the conflicting slice:\n\n- Per\n  [Monitor the status of the slice](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fuse-gke-dynamic-slicing.md.txt),\n  when a slice is created or actively used by a workload, the slice controller\n  attaches the `kueue.gke.io\u002Ftpu-slice-used-by-namespace` and\n  `kueue.gke.io\u002Ftpu-slice-used-by-workload` labels.\n- Per\n  [Optimize workload scheduling with slice reuse](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fuse-gke-dynamic-slicing.md.txt)\n  and\n  [Enable Slice reuse](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fuse-gke-dynamic-slicing.md.txt),\n  when slice reuse is enabled, \"Kueue marks the slice as idle at the end of a\n  job instead of deleting it\", and \"Reusable slices are named dynamically with\n  an `s-\u003Ctopology>` prefix, decoupling them from specific workload names.\"\n\nThen apply the matching option:\n\n1. **Wait for the conflicting slice to release the nodes and retry**:\n   - **If Kueue manages the conflicting slice** (it has the `kueue.gke.io\u002F...`\n     labels or an `s-\u003Ctopology>` name prefix): Do not suggest deleting the\n     `Slice` custom resource directly. Point the workload owner to\n     [Clean up](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fuse-gke-dynamic-slicing.md.txt)\n     (deleting the `JobSet` triggers Kueue to delete the associated `Slice`\n     custom resources) and, if slice reuse keeps an idle `s-\u003Ctopology>` slice\n     allocated after a job ends, to\n     [Enforce slice deletion per workload](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fuse-gke-dynamic-slicing.md.txt)\n     (`kueue.gke.io\u002Ftpu-enforce-slice-deletion: \"true\"`).\n   - **If a custom scheduler or user created the conflicting slice directly**:\n     Its owner can delete it when finished, as described in\n     [Delete the slice](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt).\n   - **How `{slice_name}` retries**: Check whether `{slice_name}` already has\n     `slice.gke.io\u002Fretry-on-failure: \"true\"` in `Metadata` > `Annotations`. Per\n     [Create a Slice custom resource](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt),\n     when a `Slice` custom resource is created with this annotation, \"If the\n     slice isn't created because of the `SliceCreationFailed` status reason, the\n     controller will retry until the slice is successfully formed.\" Because the\n     documentation only shows configuring\n     `slice.gke.io\u002Fretry-on-failure: \"true\"` in the `Slice` manifest when\n     creating a `Slice` custom resource (and doesn't state that adding the\n     annotation to an existing slice already in `SliceCreationFailed` starts a\n     retry), if `{slice_name}` wasn't created by Kueue and lacks the annotation,\n     tell the user to delete `{slice_name}` following\n     [Delete the slice](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt)\n     and re-create it with `slice.gke.io\u002Fretry-on-failure: \"true\"` in its\n     manifest following\n     [Create a Slice custom resource](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt).\n2. **Use available partitions whose nodes no other slice uses**: If the user\n   manages `Slice` custom resources directly (not through Kueue) and can't wait,\n   direct them to\n   [Verify the status of the nodes and the partitions](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt)\n   and\n   [Create a Slice custom resource](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt)\n   so they can pick available partitions (`HEALTHY` partitions, or `DEGRADED`\n   `4x4x4` partitions) whose nodes aren't used by any other slice (across all\n   partition shapes on those nodes) and create the `Slice` custom resource with\n   them.\n3. **Schedule dynamic slices with Kueue and TAS instead of creating `Slice`\n   custom resources directly**: Per\n   [Target your node pool to belong to a block or a sub-block](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fuse-gke-dynamic-slicing.md.txt)\n   and\n   [Create a dynamic slice with Kueue and TAS](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fuse-gke-dynamic-slicing.md.txt),\n   instead of creating `Slice` custom resources directly, teams can schedule\n   GKE workloads with Kueue and Topology Aware Scheduling (TAS), where \"Kueue\n   automatically handles the creation and deletion of Slice custom resources\"\n   and admits a workload after its slice becomes active.\n4. **Verify or escalate**: Repeat Step 1 following\n   [Monitor the status of the Slice custom resource](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt):\n   - **`ACTIVE`**: The slice is fully formed, healthy, and ready to execute\n     workloads.\n   - **`ACTIVE_DEGRADED`**: The slice is formed, and workloads can run, but\n     performance might be impacted.\n   - **Still `SliceCreationFailed` after no other slice uses the nodes, or\n     `FAILED`**: Hand off to\n     `gke-ai-troubleshooting-tpu-dynamic-slices-monitoring`, or advise the user\n     to contact Cloud Customer Care following\n     [Before you contact support](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fgetting-support.md.txt)\n     and\n     [Recommendations for efficient support](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fgetting-support.md.txt).\n","# Troubleshoot TPU slice creation failures caused by partition overlap\n\nA dynamic slice fails to form with the status reason `SliceCreationFailed` when,\namong other documented causes, any of its nodes are already used by another\nslice. This skill confirms the failure, rules out the other documented causes,\nfinds the slice that uses the same nodes, and routes the fix through GKE.\n\n## Attribution\n\n**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you\nMUST read [Usage Attribution](references\u002Fattribution.md). For commands executed\ndirectly in the terminal (never in generated scripts, Terraform, or user code),\nprefix `gcloud` with\n`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION=\"agent-skills\u002F1.0 (skill:gke-ai-troubleshooting-tpu-partition-overlap)\"`\nand append `agent-skills\u002F1.0 (skill:gke-ai-troubleshooting-tpu-partition-overlap)` to `User-Agent` headers.\n\n## Prerequisites\n\n- **Tools**:\n  - For installing and initializing `gcloud` (which covers authentication and\n    project selection), follow\n    [Before you begin](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt).\n  - For installing `kubectl` and required plugins, follow\n    [Install kubectl](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcluster-access-for-kubectl.md.txt)\n    and\n    [Install required plugins](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcluster-access-for-kubectl.md.txt).\n- **Cloud Billing & Project Configuration**: Ensure the project has an active\n  Cloud Billing account. Follow\n  [Before you begin](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt)\n  to initialize `gcloud` and enable the Google Kubernetes Engine API\n  (`container.googleapis.com`), ensure the Cloud Monitoring API\n  (`monitoring.googleapis.com`) is enabled, and follow\n  [Store cluster information for kubectl](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcluster-access-for-kubectl.md.txt)\n  to configure `kubectl` cluster credentials.\n- **Required IAM Roles**:\n  - Kubernetes Engine Viewer (`roles\u002Fcontainer.viewer`)\n  - Monitoring Viewer (`roles\u002Fmonitoring.viewer`)\n- **Documentation**:\n  - [Use dynamic slicing with a custom scheduler](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt)\n    (Sections: \"Before you begin\", \"Target your node pool to belong to a block\n    or a sub-block\", \"Verify the status of the nodes and the partitions\",\n    \"Create a Slice custom resource\", \"Monitor the status of the Slice custom\n    resource\", \"Delete the slice\")\n  - [Configure cluster access for kubectl](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcluster-access-for-kubectl.md.txt)\n    (Sections: \"Install kubectl\", \"Install required plugins\", \"Store cluster\n    information for kubectl\")\n  - [Slice](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Freference\u002Fcrds\u002Fslice.md.txt)\n    (Sections: \"Slice status\", \"Slice specification\")\n  - [Schedule dynamic slices with Kueue and TAS](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fuse-gke-dynamic-slicing.md.txt)\n    (Sections: \"Target your node pool to belong to a block or a sub-block\",\n    \"Create a dynamic slice with Kueue and TAS\", \"Define the partition health\n    selection\", \"Optimize workload scheduling with slice reuse\", \"Enable Slice\n    reuse\", \"Enforce slice deletion per workload\", \"Monitor the status of the\n    slice\", \"Monitor the metrics of the slice\", \"Clean up\")\n  - [GKE system metrics](https:\u002F\u002Fdocs.cloud.google.com\u002Fmonitoring\u002Fapi\u002Fmetrics_kubernetes.md.txt)\n    (Sections: \"Kubernetes metrics\")\n  - [Monitored resource types](https:\u002F\u002Fdocs.cloud.google.com\u002Fmonitoring\u002Fapi\u002Fresources.md.txt)\n    (Sections: \"k8s_node_pool\")\n  - [About GKE dynamic slicing](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fconcepts\u002Fdynamic-slicing.md.txt)\n    (Sections: \"Dynamic slicing configurations\")\n  - [Get support](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fgetting-support.md.txt)\n    (Sections: \"Before you contact support\", \"Recommendations for efficient\n    support\")\n\n> **Read-only rule**: Run read-only diagnostic commands only. Never drain,\n> delete, or re-create nodes, or run any other command that changes the cluster.\n> Give the user any fix to apply themselves.\n\nWhen you recommend a fix, link the doc section that describes it.\n\n---\n\n## Diagnostic workflow\n\n### Step 0: Collect context `[Low Risk]`\n\nCollect the target parameters:\n\n- `{project_id}`: Google Cloud project ID\n- `{cluster_name}`: GKE cluster name\n- `{location}`: Cluster region or zone\n- `{slice_name}`: Name of the Slice custom resource that failed to form\n- `{node_pool_name}`: Name of the TPU node pool (or node pools) backing the\n  partitions in `{slice_name}`, if known. If the user doesn't know\n  `{node_pool_name}`, after Step 1 run the Cloud Monitoring query in Step 3\n  for the partition IDs in `{partition_ids}` and read the `node_pool_name`\n  label on the\n  [k8s_node_pool](https:\u002F\u002Fdocs.cloud.google.com\u002Fmonitoring\u002Fapi\u002Fresources.md.txt)\n  monitored resource.\n- `{issue_time}`: Incident timestamp in RFC3339 UTC, if known. If the user\n  doesn't know `{issue_time}`, use the `Last Transition Time`\n  (`lastTransitionTime`, \"A timestamp of the most recent status change\" in\n  [Slice status](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Freference\u002Fcrds\u002Fslice.md.txt))\n  of the `Ready` condition from Step 1. By default, query Cloud Monitoring over\n  a 60-minute window around `{issue_time}`.\n\n---\n\n### Step 1: Confirm the `SliceCreationFailed` status `[Low Risk]`\n\nTo describe `{slice_name}`, follow the instructions in the section\n[Monitor the status of the Slice custom resource](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt).\n\n- In `Status` > `Conditions`, check that the `Reason` of the `Ready` condition\n  is `SliceCreationFailed`, and record its `Last Transition Time` (to use as\n  `{issue_time}` if needed).\n- Record the IDs in `Spec` > `Partition Ids` as `{partition_ids}`, the\n  `Topology`, and any `Labels` and `Annotations` on `{slice_name}`.\n- If the `Ready` reason isn't `SliceCreationFailed`, this skill doesn't apply.\n  See\n  [Slice status](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Freference\u002Fcrds\u002Fslice.md.txt)\n  for what each reason means, and hand off to\n  `gke-ai-troubleshooting-tpu-dynamic-slices-monitoring`.\n\n---\n\n### Step 2: Rule out the other documented causes `[Low Risk]`\n\nPer\n[Create a Slice custom resource](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt),\nslice creation also fails with `SliceCreationFailed` when:\n\n- The selected nodes don't exist.\n- The topology doesn't match the number of partitions. A dynamic sub-slice lists\n  exactly one partition ID. A dynamic super-slice lists `(A × B × C) \u002F 64`\n  partition IDs.\n\nIf either of those causes applies, hand off to\n`gke-ai-troubleshooting-tpu-dynamic-slices-monitoring`.\n\nOtherwise, follow\n[Verify the status of the nodes and the partitions](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt)\non `{node_pool_name}`:\n\n- Verify that the nodes exist and carry each ID in `{partition_ids}`.\n- On the nodes that carry `{partition_ids}`, record **every** partition ID of\n  **every** shape (`4x4x4`, `2x4x4`, `2x2x4`, `2x2x2`, and `2x2x1`) from the\n  `cloud.google.com\u002Fgke-tpu-partition-.*-id` labels. Per\n  [Dynamic slicing configurations](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fconcepts\u002Fdynamic-slicing.md.txt),\n  dynamic sub-slicing partitions a `4x4x4` sub-block into smaller topologies on\n  the same physical node pool, so partitions of different shapes share the same\n  nodes.\n- Read each partition's `cloud.google.com\u002Fgke-tpu-partition-.*-state` label. Per\n  [Verify the status of the nodes and the partitions](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt)\n  and\n  [Define the partition health selection](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fuse-gke-dynamic-slicing.md.txt),\n  count `HEALTHY` partitions and `DEGRADED` `4x4x4` partitions (\"still usable\n  for dynamic slice formation\"; \"Smaller topologies don't have a degraded\n  state\") as available. Partitions in `UNHEALTHY` (\"can't be used to form a\n  slice\"), `UNSET`, or `INCOMPLETE` aren't available. If a requested partition\n  is in `UNHEALTHY`, `UNSET`, or `INCOMPLETE`, suggest an available partition\n  whose nodes no other slice uses (Step 4, item 2), hand off to\n  `gke-ai-troubleshooting-tpu-dynamic-slices-monitoring`, or escalate to Cloud\n  Customer Care (Step 4, item 4).\n\n---\n\n### Step 3: Find the slice that uses the same nodes `[Low Risk]`\n\nPer\n[Create a Slice custom resource](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt),\nslice creation fails with `SliceCreationFailed` when any nodes on the custom\nresource are used by another slice. Because partitions of different shapes on\nthe same sub-block share nodes (see\n[Dynamic slicing configurations](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fconcepts\u002Fdynamic-slicing.md.txt)),\nanother slice conflicts with `{slice_name}` if it lists **any** of the partition\nIDs (of any shape) recorded on `{slice_name}`'s nodes in Step 2.\n\n1. List the `Slice` custom resources in the cluster (as shown in\n   [Delete the slice](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt))\n   and describe each one following\n   [Monitor the status of the Slice custom resource](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt)\n   to read `Spec` > `Partition Ids`, `Metadata` > `Labels`, and `Status` >\n   `Conditions` (`Ready` reason):\n   - Treat any slice other than `{slice_name}` whose `Spec` > `Partition Ids`\n     includes **any** partition ID recorded on `{slice_name}`'s nodes in Step 2\n     (whether the exact ID in `{partition_ids}` or another shape's partition ID\n     on the same nodes) as conflicting. Record its name, its `Ready` reason\n     (such as `ACTIVE`), and whether Kueue manages it (Step 4, item 1).\n2. To check partition-to-slice associations in Cloud Monitoring (including\n   around `{issue_time}`), query the `kubernetes.io\u002Faccelerator\u002Fpartition\u002Fstate`\n   metric listed in\n   [Monitor the metrics of the slice](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fuse-gke-dynamic-slicing.md.txt).\n   Per\n   [Kubernetes metrics](https:\u002F\u002Fdocs.cloud.google.com\u002Fmonitoring\u002Fapi\u002Fmetrics_kubernetes.md.txt),\n   this metric \"is 1 if the partition state is detected, 0 otherwise\", and its\n   labels include `partition_id`, `slice_id` (\"ID of the associated slice\"),\n   and `state` (\"Health of the partition (e.g., HEALTHY, UNHEALTHY)\"). Run the\n   query for each `{partition_id}` recorded on `{slice_name}`'s nodes in Step 2:\n\n```promql\nkubernetes_io:accelerator_partition_state{\n  monitored_resource=\"k8s_node_pool\",\n  cluster_name=\"{cluster_name}\",\n  partition_id=\"{partition_id}\"\n}\n```\n\n   - Read only time series whose value is `1` (where `state=\"HEALTHY\"` means the\n     partition is healthy and functional, per\n     [Verify the status of the nodes and the partitions](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt)).\n   - Because\n     [Kubernetes metrics](https:\u002F\u002Fdocs.cloud.google.com\u002Fmonitoring\u002Fapi\u002Fmetrics_kubernetes.md.txt)\n     defines `slice_id` only as \"ID of the associated slice\", always confirm the\n     conflicting slice from the `Slice` custom resources in the cluster rather\n     than from `slice_id` alone: list the `Slice` custom resources as shown in\n     [Delete the slice](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt),\n     describe each one following\n     [Monitor the status of the Slice custom resource](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt),\n     and check which slice's `Spec` > `Partition Ids` includes the partition ID\n     (or any other partition ID of any shape on the same nodes from Step 2).\n3. **Next steps based on Step 3 results**:\n   - **Another slice uses the nodes now**: Proceed to Step 4.\n   - **No other slice uses the nodes now, but the metric (value `1`) showed an\n     associated `slice_id` around `{issue_time}`**: The conflict may already be\n     over. Proceed to Step 4, item 1 (to re-create or retry `{slice_name}`) and\n     then Step 4, item 4.\n   - **Neither the current `Slice` custom resources nor the metric around\n     `{issue_time}` shows another slice**: Node overlap isn't confirmed. Hand\n     off to `gke-ai-troubleshooting-tpu-dynamic-slices-monitoring` or escalate\n     to Cloud Customer Care (Step 4, item 4).\n\n---\n\n### Step 4: Resolution `[High Risk]`\n\n> **Guardrails**:\n> - **Never** delete or change the other slice, or stop its workload, to free\n>   the nodes. Deleting a `Slice` custom resource starts dismantling that slice\n>   (`DEACTIVATING`, see\n>   [Monitor the status of the Slice custom resource](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt)).\n>   The owner of that slice decides when to release it.\n> - **Never** manually modify or delete `Slice` custom resources created by\n>   Kueue. Per\n>   [Target your node pool to belong to a block or a sub-block](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt),\n>   \"Avoid manually modifying Slice custom resources created by Kueue.\"\n> - **Never** change GKE-managed instance groups or VMs directly through Compute\n>   Engine, and never delete node pools to free a partition. Manage node pools\n>   and slices through GKE.\n\nFirst, check whether Kueue manages `{slice_name}` or the conflicting slice:\n\n- Per\n  [Monitor the status of the slice](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fuse-gke-dynamic-slicing.md.txt),\n  when a slice is created or actively used by a workload, the slice controller\n  attaches the `kueue.gke.io\u002Ftpu-slice-used-by-namespace` and\n  `kueue.gke.io\u002Ftpu-slice-used-by-workload` labels.\n- Per\n  [Optimize workload scheduling with slice reuse](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fuse-gke-dynamic-slicing.md.txt)\n  and\n  [Enable Slice reuse](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fuse-gke-dynamic-slicing.md.txt),\n  when slice reuse is enabled, \"Kueue marks the slice as idle at the end of a\n  job instead of deleting it\", and \"Reusable slices are named dynamically with\n  an `s-\u003Ctopology>` prefix, decoupling them from specific workload names.\"\n\nThen apply the matching option:\n\n1. **Wait for the conflicting slice to release the nodes and retry**:\n   - **If Kueue manages the conflicting slice** (it has the `kueue.gke.io\u002F...`\n     labels or an `s-\u003Ctopology>` name prefix): Do not suggest deleting the\n     `Slice` custom resource directly. Point the workload owner to\n     [Clean up](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fuse-gke-dynamic-slicing.md.txt)\n     (deleting the `JobSet` triggers Kueue to delete the associated `Slice`\n     custom resources) and, if slice reuse keeps an idle `s-\u003Ctopology>` slice\n     allocated after a job ends, to\n     [Enforce slice deletion per workload](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fuse-gke-dynamic-slicing.md.txt)\n     (`kueue.gke.io\u002Ftpu-enforce-slice-deletion: \"true\"`).\n   - **If a custom scheduler or user created the conflicting slice directly**:\n     Its owner can delete it when finished, as described in\n     [Delete the slice](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt).\n   - **How `{slice_name}` retries**: Check whether `{slice_name}` already has\n     `slice.gke.io\u002Fretry-on-failure: \"true\"` in `Metadata` > `Annotations`. Per\n     [Create a Slice custom resource](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt),\n     when a `Slice` custom resource is created with this annotation, \"If the\n     slice isn't created because of the `SliceCreationFailed` status reason, the\n     controller will retry until the slice is successfully formed.\" Because the\n     documentation only shows configuring\n     `slice.gke.io\u002Fretry-on-failure: \"true\"` in the `Slice` manifest when\n     creating a `Slice` custom resource (and doesn't state that adding the\n     annotation to an existing slice already in `SliceCreationFailed` starts a\n     retry), if `{slice_name}` wasn't created by Kueue and lacks the annotation,\n     tell the user to delete `{slice_name}` following\n     [Delete the slice](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt)\n     and re-create it with `slice.gke.io\u002Fretry-on-failure: \"true\"` in its\n     manifest following\n     [Create a Slice custom resource](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt).\n2. **Use available partitions whose nodes no other slice uses**: If the user\n   manages `Slice` custom resources directly (not through Kueue) and can't wait,\n   direct them to\n   [Verify the status of the nodes and the partitions](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt)\n   and\n   [Create a Slice custom resource](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt)\n   so they can pick available partitions (`HEALTHY` partitions, or `DEGRADED`\n   `4x4x4` partitions) whose nodes aren't used by any other slice (across all\n   partition shapes on those nodes) and create the `Slice` custom resource with\n   them.\n3. **Schedule dynamic slices with Kueue and TAS instead of creating `Slice`\n   custom resources directly**: Per\n   [Target your node pool to belong to a block or a sub-block](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fuse-gke-dynamic-slicing.md.txt)\n   and\n   [Create a dynamic slice with Kueue and TAS](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fuse-gke-dynamic-slicing.md.txt),\n   instead of creating `Slice` custom resources directly, teams can schedule\n   GKE workloads with Kueue and Topology Aware Scheduling (TAS), where \"Kueue\n   automatically handles the creation and deletion of Slice custom resources\"\n   and admits a workload after its slice becomes active.\n4. **Verify or escalate**: Repeat Step 1 following\n   [Monitor the status of the Slice custom resource](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fcreate-dynamic-slices.md.txt):\n   - **`ACTIVE`**: The slice is fully formed, healthy, and ready to execute\n     workloads.\n   - **`ACTIVE_DEGRADED`**: The slice is formed, and workloads can run, but\n     performance might be impacted.\n   - **Still `SliceCreationFailed` after no other slice uses the nodes, or\n     `FAILED`**: Hand off to\n     `gke-ai-troubleshooting-tpu-dynamic-slices-monitoring`, or advise the user\n     to contact Cloud Customer Care following\n     [Before you contact support](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fgetting-support.md.txt)\n     and\n     [Recommendations for efficient support](https:\u002F\u002Fdocs.cloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fgetting-support.md.txt).\n",{"name":5,"metadata":44,"description":7},{"version":45,"category":46},"1.0.1","Containers","main","bb9b3f464aae37af360d079c1d863524955b0363","skills\u002Fcloud\u002Fgke-ai-troubleshooting-tpu-partition-overlap",2005,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills","https:\u002F\u002Fgithub.com\u002Fgoogle","https:\u002F\u002Fopensource.google\u002F","Google ❤️ Open Source","2026-11-14 01:51:35",{"source":8,"skillName":5,"commands":57,"prompt":67,"zipAvailable":68},[58,61,64],{"tool":59,"command":60},"npx","npx skills add google\u002Fskills --skill gke-ai-troubleshooting-tpu-partition-overlap",{"tool":62,"command":63},"bunx","bunx skills add google\u002Fskills --skill gke-ai-troubleshooting-tpu-partition-overlap",{"tool":65,"command":66},"pnpm","pnpm dlx skills add google\u002Fskills --skill gke-ai-troubleshooting-tpu-partition-overlap","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fgke-ai-troubleshooting-tpu-partition-overlap\u002FSKILL.md",true,[70],{"path":71,"resourceType":72,"resourceTypeDesc":73,"sizeBytes":74,"sourceHtmlUrl":75},"references\u002Fattribution.md","REFERENCE","参考文档",1980,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fgke-ai-troubleshooting-tpu-partition-overlap\u002Freferences\u002Fattribution.md",[77,114,145,174,201,225],{"skillNo":78,"name":79,"nameZh":6,"description":80,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":81,"categories":82,"keywords":91,"publishedAt":39},"01M6998S2EZKD3GEJ7F943RZTE","sign-in-with-google-web","Implement, configure, and secure Sign In With Google (SiwG) using Google Identity Services (GIS \u002F `https:\u002F\u002Faccounts.google.com\u002Fgsi\u002Fclient`) across web architectures. Use when creating Google sign-in buttons, implementing Google One Tap with FedCM, integrating GIS in React\u002FNext.js\u002FAngular\u002FHTML, verifying ID tokens on backend runtimes (Python\u002FNode.js\u002FGo\u002FJava), enforcing Google Workspace domain restrictions (`hd`), securing client-side ID tokens via WebCrypto nonces or encrypted IndexedDB, embedding in cross-origin iframes via the Intermediate Iframe API (`https:\u002F\u002Faccounts.google.com\u002Fgsi\u002Fintermediate`), configuring Content Security Policy (CSP), COOP, and Permissions Policy (`identity-credentials-get`) headers, or handling sign-out and revocation. Don't use for legacy Google Sign-In (`gapi.auth2`), Google Cloud IAM service accounts, or Android\u002FiOS native Credential Manager.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fidentity\u002Fsign-in-with-google-web\u002FSKILL.md",[83,88],{"categoryNo":84,"categoryName":85,"taxonomyType":86,"taxonomyTypeDesc":87},"01JBSEEDCAT000000000000009","前端开发","SCENARIO","使用场景",{"categoryNo":89,"categoryName":90,"taxonomyType":86,"taxonomyTypeDesc":87},"01JBSEEDCAT000000000000010","后端与接口",[9,92,93,28,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113],"sign in with google web","sign","web","android","go","security","google workspace","google cloud","implement","policy","python","implementing","react","iam","permissions","buttons","secure","manager","identity","architectures","domain","runtimes",{"skillNo":115,"name":116,"nameZh":6,"description":117,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":118,"categories":119,"keywords":129,"publishedAt":39},"01M6998PHPDCAD684VW5WRG83D","dpop-adoption","Implement and debug OAuth 2.0 DPoP (RFC 9449) refresh token sender-constraining for WebCrypto, Node.js ES6, and browser runtimes integrating with Google's OAuth platform. Use when configuring non-extractable asymmetric key pairs (P-256), generating DPoP Proof JWTs for authorization code exchange and token refresh, or handling 400 use_dpop_nonce challenge retry loops at oauth2.googleapis.com\u002Ftoken. Don't use for unconstrained OAuth 2.0 flows (where refresh tokens are not bound to a client key pair), or for Google Cloud IAM \u002F service account authentication.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fidentity\u002Fdpop-adoption\u002FSKILL.md",[120,121,124],{"categoryNo":89,"categoryName":90,"taxonomyType":86,"taxonomyTypeDesc":87},{"categoryNo":122,"categoryName":123,"taxonomyType":86,"taxonomyTypeDesc":87},"01JBSEEDCAT000000000000023","Skill与智能体开发",{"categoryNo":125,"categoryName":126,"taxonomyType":127,"taxonomyTypeDesc":128},"01JBSEEDCAT000000000000030","开发工程师","PROFESSION","适用职业",[130,131,9,99,132,133,134,135,136,137,105,138,139,140,141,142,143,110,113,144,24],"dpop adoption","adoption","es6","retry","authentication","flows","challenge","code","browser","loops","at","node","token","pair","platform",{"skillNo":146,"name":147,"nameZh":6,"description":148,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":149,"categories":150,"keywords":157,"publishedAt":39},"01M6998M72TC73EEAQH4XC7EXW","retrieving-developer-knowledge","Searches, retrieves, and synthesizes official Google developer documentation across Google Cloud, AI\u002FGemini, Android, Chrome, Web, Flutter, Go, Firebase, and other Google developer platforms. Integrates with the Developer Knowledge MCP server (search_documents, get_documents, answer_query) or the Developer Knowledge REST API fallback. Use when searching for gcloud CLI commands, API syntax, IAM permissions, official documentation, architectural comparisons, or product choice overviews. Don't use for local filesystem lookups or non-Google documentation.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fdevelopers\u002Fretrieving-developer-knowledge\u002FSKILL.md",[151,152,155,156],{"categoryNo":89,"categoryName":90,"taxonomyType":86,"taxonomyTypeDesc":87},{"categoryNo":153,"categoryName":154,"taxonomyType":86,"taxonomyTypeDesc":87},"01JBSEEDCAT000000000000021","资料检索与研究",{"categoryNo":122,"categoryName":123,"taxonomyType":86,"taxonomyTypeDesc":87},{"categoryNo":125,"categoryName":126,"taxonomyType":127,"taxonomyTypeDesc":128},[158,159,160,161,99,162,163,164,165,166,167,168,169,24,170,171,172,105,173,106],"retrieving developer knowledge","knowledge","developer","MCP","mcp server","official","developer documentation","get","search","local","choice","documents","query","api","across","documentation",{"skillNo":175,"name":176,"nameZh":6,"description":177,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":178,"categories":179,"keywords":180,"publishedAt":39},"01M6998J2CZEXS9BES9S3WWFKC","migrate-to-rust","Clones, analyzes, plans, and migrates legacy codebases (C++, Fortran, Java, Python, C, Shell) from a GitHub URL or local directory to a modern, high-performance, deterministic, 100% Safe Rust architecture, tracking live telemetry across every task bucket, test failure\u002Ffix cycle, and Profile-Guided Optimization (PGO) stage, and producing a first-person AI storytelling report and interactive HTML infographic. Use when migrating a GitHub repository or local codebase to Rust, porting legacy C++, Fortran, Python, or Java workloads to memory-safe Rust, designing Struct-of-Arrays (SoA) or SIMD Rust architectures, or optimizing Rust ports with LLVM PGO. Don't use for unrelated non-Rust refactoring or single-line syntax questions.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fdevelopers\u002Fmigrate-to-rust\u002FSKILL.md",[],[181,182,183,184,185,186,187,188,102,189,190,191,192,193,194,167,195,196,197,24,198,199,172,200],"migrate to rust","rust","migrate","github","memory","performance","directory","safe","ports","llvm","soa","cycle","porting","fix","arrays","infographic","high-performance","URL","architecture","unrelated",{"skillNo":202,"name":203,"nameZh":6,"description":204,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":205,"categories":206,"keywords":210,"publishedAt":39},"01M6998GCC3K29WZ82KXMB9E02","finding-google-skills","Google platform decision and setup guidance, loaded on demand from Google's skill catalog. Use when a developer is choosing or setting up part of their stack, such as where to run a service, a database, storage, messaging, authentication, analytics, ads, or AI model serving, and a Google product is a reasonable candidate - whether or not a vendor is named - or when a request names a Google product or API. Brings in the matching Google skill so the answer can weigh Google options, their trade-offs, and when they are not the right fit. Skip when the stack is already settled on another provider and no Google product is named, or the task involves no platform choice.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fdevelopers\u002Ffinding-google-skills\u002FSKILL.md",[207,208,209],{"categoryNo":89,"categoryName":90,"taxonomyType":86,"taxonomyTypeDesc":87},{"categoryNo":122,"categoryName":123,"taxonomyType":86,"taxonomyTypeDesc":87},{"categoryNo":125,"categoryName":126,"taxonomyType":127,"taxonomyTypeDesc":128},[9,211,212,213,214,144,215,216,217,218,219,134,220,221,222,168,35,223,24,224,171],"finding google skills","storage","as","messaging","model","trade-offs","provider","answer","fit","catalog","up","decision","database","candidate",{"skillNo":226,"name":227,"nameZh":6,"description":228,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":229,"categories":230,"keywords":233,"publishedAt":39},"01M6998DWNHTH0BS6W1CY8ZXMP","workload-manager-basics","Use this skill to manage Google Cloud Workload Manager evaluations, rules, scanned resources, and validation results by using public client libraries and the REST API. Use when you need to inspect workload best-practice rules, create and run evaluations for Google Cloud general best practices, SAP, SQL Server, or custom organizational rules, review violations, export results to BigQuery, or automate Workload Manager through client libraries because no service-specific public CLI or MCP server is available. Don't use for general Google Compute Engine instance management, VPC configuration, or standard IAM auditing.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fworkload-manager-basics\u002FSKILL.md",[231,232],{"categoryNo":89,"categoryName":90,"taxonomyType":86,"taxonomyTypeDesc":87},{"categoryNo":122,"categoryName":123,"taxonomyType":86,"taxonomyTypeDesc":87},[234,235,109,236,9,161,237,99,162,238,239,240,241,24,242,171,105,243,244,245,246],"workload manager basics","workload","basics","cli","review","vpc","resources","inspect","you","create","libraries","server","run"]