[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M69AWYRJY05J3N3K9PZ0X38H":3,"skill-same:01M69AWYRJY05J3N3K9PZ0X38H":73},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":23,"publishedAt":45,"linkOnly":46,"markdownContent":47,"markdownBody":48,"frontMatter":49,"refName":50,"commitSha":51,"skillDirectory":52,"forksCount":53,"repositoryHtmlUrl":54,"repositoryHomepageUrl":6,"ownerHtmlUrl":55,"ownerBlogUrl":56,"ownerDescription":57,"updatedAt":58,"install":59,"resources":72},"01M69AWYRJY05J3N3K9PZ0X38H","skill-koji-triage",null,"[Skill] Examine Koji builds, fetch task info and logs from the Koji Web UI, identify failures, and provide root cause analysis. Use when triaging Koji build failures, investigating failed tasks, downloading build logs, or searching for broken packages. Triggers: koji failure, koji build failed, koji task, koji log, koji triage, build failure analysis.","microsoft\u002Fazurelinux","microsoft","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F6154722?v=4","01M698VYDXZ6JW5WH3CRV650R8","Microsoft","01M69AVY9Y3D59ZBRVNKRJ851E",5364,"MIT","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fazurelinux\u002Fblob\u002F4.0\u002F.agents\u002Fskills\u002Fskill-koji-triage\u002FSKILL.md",[18],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44],"skill koji triage","triage","fetch","when","root cause","for","web ui","failure analysis","triggers","info","task","log","use","broken","analysis","logs","ui","packages","tasks","root cause analysis","web","2026-11-10 10:12:30",false,"---\nname: skill-koji-triage\ndescription: \"[Skill] Examine Koji builds, fetch task info and logs from the Koji Web UI, identify failures, and provide root cause analysis. Use when triaging Koji build failures, investigating failed tasks, downloading build logs, or searching for broken packages. Triggers: koji failure, koji build failed, koji task, koji log, koji triage, build failure analysis.\"\n---\n\n# Koji Build Failure Triage\n\n## Koji Web UI\n\n- **Base URL**: Get from the user, or from env variable `KOJI_BASE_URL`, or pre-configured in the MCP server.\n  - If the env variable is not set, and the MCP is not pre-configured, prompt the user to input the base URL of the Koji Web UI (e.g., `https:\u002F\u002Fkoji.example.com`).\n- **MCP tools (preferred)**: Use the `koji` MCP server tools to fetch pages and logs. These write output to temp files under `base\u002Fbuild\u002Fwork\u002Fscratch\u002Fkoji\u002F` to avoid bloating context. Use `read_file` and `grep_search` on the resulting files to inspect content.\n\n  **Available tools:** `koji_status`, `set_koji_url`, `koji_fetch`, `koji_allow_insecure`, `koji_cleanup`. All tools return a dict with the result plus server state (base URL, output dir, etc.).\n\n  1. Call `koji_status` to check if a base URL is already configured (via env or a prior call).\n  2. Either call `set_koji_url` to set a default, or pass `override_base_url` directly to `koji_fetch` per-call. Setting a default is convenient when all requests target the same instance, and `override_base_url` is useful for ad-hoc queries against other instances.\n  3. Call `koji_fetch` with the path (e.g., `\u002Fkoji\u002Ftaskinfo?taskID=3307`). It returns the file path where content was saved.\n     - If the fetch fails with an SSL certificate error (e.g., self-signed cert), ask the user if they want to skip SSL verification, then call `koji_allow_insecure` (optionally with `override_base_url`) to proceed. Do **NOT** proceed without explicit user approval for the **specific** URL.\n  4. Use `read_file` or `grep_search` on the saved file to extract the information you need.\n  5. Call `koji_cleanup` when done to remove fetched temp files and reclaim disk space (NOTE: This will clean up ALL scratch files, so only call it when you're completely done with the current files).\n- **MCP is required.** If MCP tools are not working, ask the user to fix their MCP configuration rather than falling back to CLI tools.\n- **Network issues**: Koji is typically only accessible via VPN or corporate network. If you encounter connection errors or timeouts, assume it is a network permissions issue — ask the user to confirm they are connected to the appropriate network before retrying. Do NOT keep retrying on your own.\n\n## Discovery: Finding Builds by Package Name\n\nWhen the user provides a **package name** instead of a task ID, use **both** discovery paths below to find all recent failures. Neither path alone is sufficient.\n\n> **Why two paths?** The package info page only lists builds where Koji created a build record. If a task fails early (e.g., `buildSRPMFromSCM` fails), no build record exists and the failure is invisible on the package page. The failed tasks list catches these, but doesn't filter by package — you must grep for the name. Always run both.\n\n### Path A: Package info page (finds builds with records)\n\n1. **Search for the package:**\n\n```\n\u002Fkoji\u002Fsearch?match=glob&type=package&terms=\u003CPACKAGE_NAME>\n```\n\n- If the exact name doesn't match, try a glob: `terms=\u003CNAME>*`\n- Extract the `packageID` from the results (look for `packageinfo?packageID=` links)\n- If the search returns only one match, Koji redirects directly to the package info page (skip step 2).\n\n2. **Get recent builds from the package info page:**\n\n```\n\u002Fkoji\u002Fpackageinfo?packageID=\u003CPACKAGE_ID>\n```\n\n- Look for `buildinfo?buildID=` links with state `failed`\n- Note the build NVRs (Name-Version-Release) for context\n\n3. **Get the build's parent task ID:**\n\n```\n\u002Fkoji\u002Fbuildinfo?buildID=\u003CBUILD_ID>\n```\n\n- Extract the `taskinfo?taskID=` link — this is the parent `build` task\n- The parent task's Descendants section lists the actual failed child tasks (`buildArch`, `buildSRPMFromSCM`)\n\n### Path B: Failed tasks list (catches failures with no build record)\n\nSearch the global failed tasks list for the package name:\n\n```\n\u002Fkoji\u002Ftasks?owner=&state=failed&view=flat&method=build&order=-id\n```\n\n- This lists all recent failed `build` tasks, newest first\n- **Grep the results for the package name** to find relevant task IDs\n- This catches early failures (e.g., `buildSRPMFromSCM` errors, plugin\u002Fcallback errors) that never create a build record and would be invisible on the package info page\n\n### Combine results\n\nCollect the parent task IDs from both paths, deduplicate, then for each parent task:\n1. Fetch `taskinfo?taskID=\u003CPARENT_ID>` to see the Descendants tree\n2. Identify which child task(s) failed (`buildSRPMFromSCM` or `buildArch`)\n3. Proceed to the Investigation Workflow below for each failed child\n\n### Quick-reference: Discovery URLs\n\n| Goal | URL Path |\n|------|----------|\n| Search packages by name | `\u002Fkoji\u002Fsearch?match=glob&type=package&terms=\u003CNAME>` |\n| Package info (recent builds) | `\u002Fkoji\u002Fpackageinfo?packageID=\u003CID>` |\n| Build details → parent task | `\u002Fkoji\u002Fbuildinfo?buildID=\u003CID>` |\n| All recent failed build tasks | `\u002Fkoji\u002Ftasks?owner=&state=failed&view=flat&method=build&order=-id` |\n| Search builds by NVR | `\u002Fkoji\u002Fsearch?match=glob&type=build&terms=\u003CNAME>*` |\n\n### Example: \"Why is the kernel broken?\"\n\n```\n# Path A: Package info page\n# 1. Search for the kernel package\nkoji_fetch(path=\"\u002Fkoji\u002Fsearch?match=glob&type=package&terms=kernel\")\n# → grep for 'packageinfo?packageID=' → e.g., packageID=6\n# → (if single match, Koji redirects directly to package info page)\n\n# 2. Get package info to see recent builds\nkoji_fetch(path=\"\u002Fkoji\u002Fpackageinfo?packageID=6\")\n# → grep for 'failed' and 'buildinfo?buildID=' → e.g., buildID=1482\n\n# 3. Get the parent task ID from the build info page\nkoji_fetch(path=\"\u002Fkoji\u002Fbuildinfo?buildID=1482\")\n# → grep for 'taskinfo?taskID=' → e.g., parent task 32608\n\n# Path B: Failed tasks list (run in parallel with Path A)\n# 4. Search all recent failed build tasks for \"kernel\"\nkoji_fetch(path=\"\u002Fkoji\u002Ftasks?owner=&state=failed&view=flat&method=build&order=-id\")\n# → grep for 'kernel' → may find additional tasks (e.g., task 87474)\n#    that failed before creating a build record\n\n# Combine: parent tasks = {32608, 87474}\n# 5. For each parent, fetch task info and find the failed child\nkoji_fetch(path=\"\u002Fkoji\u002Ftaskinfo?taskID=32608\")\n# → Descendants: buildArch x86_64 (task 39696) = failed\nkoji_fetch(path=\"\u002Fkoji\u002Ftaskinfo?taskID=87474\")\n# → Descendants: buildSRPMFromSCM (task 87475) = failed\n\n# 6. Investigate each failed child using the workflow below\n```\n\n## Investigation Workflow\n\n> **Branch mismatch warning:** The branch checked out locally may not correspond to the code that Koji built. If the local component definition, spec, or overlays don't seem to match what the build logs show, check which branch\u002Fcommit Koji used (visible in the task info page or `checkout.log`) and compare with the current local branch (`git branch --show-current` \u002F `git log --oneline -1`). Use non-destructive git commands to inspect the Koji branch without modifying the working tree — e.g., `git show \u003Ckoji-ref>:\u003Cpath>` to read files, `git diff HEAD..\u003Ckoji-ref> -- \u003Cpath>` to compare. Avoid `git checkout` to switch branches, as the user may have uncommitted work.\n\n### 1. Get Task Info\n\nFetch the task page and extract key details (state, method, children).\n\nIf you don't know the status of the koji tool, check its status.\n\nDecide if the base URL should be changed (is this a one-off, or long-term investigation?).\n\nCall `koji_fetch` with path `\u002Fkoji\u002Ftaskinfo?taskID=\u003CTASK_ID>`, either using the default base URL or passing an `override_base_url`. Use either built-in tools or cmdline tools on the saved file to extract state, method, and child task IDs.\n\nParse the HTML to find:\n- **State**: `failed`, `closed` (success), `canceled`, `free`, `open`\n- **Method**: `build` (parent), `buildSRPMFromSCM`, `buildArch`\n- **Child task IDs**: Look for `taskinfo?taskID=` links in the Descendants section\n\nA `build` task always has children: one `buildSRPMFromSCM` (SRPM creation) and one or more `buildArch` tasks (per architecture). The parent fails if any child fails.\n\n### 2. Find the Failed Child\n\nExtract all `taskinfo?taskID=\u003CID>` links from the parent task page. Then fetch each child task page and check its state. Typically:\n- `buildSRPMFromSCM` succeeds\n- One of the `buildArch` tasks fails (look for `state.*failed` in the HTML)\n\n**Important**: Also check the **Result** field in the task info HTML. For some failures (especially `buildSRPMFromSCM`), the error message is embedded directly in the page under \"Result\" and is NOT in any downloadable log file.\n\n### 3. Download Logs\n\nDownload logs from the failed child task.\n\nCall `koji_fetch` with the path `\u002Fkoji\u002Fgetfile?taskID=\u003CCHILD_TASK_ID>&volume=DEFAULT&name=\u003CFILENAME>&offset=0`. The content is saved to a temp file — use `grep_search` or `read_file` to inspect it. For large logs, fetch with `&offset=-4000` first to get the tail.\n\nAvailable log files (listed on the child task page):\n| File | Contains |\n|------|----------|\n| `root.log` | Mock chroot setup, dependency resolution, RPM build commands |\n| `build.log` | RPM build output (`%prep`, `%build`, `%install`, `%check`) |\n| `mock_output.log` | Mock orchestration output, systemd-nspawn errors |\n| `checkout.log` | Git clone\u002Ffetch output (buildSRPMFromSCM tasks only) |\n| `dnf5.log` | Package manager operations |\n| `state.log` | Mock state transitions |\n| `hw_info.log` | Builder hardware info |\n| `mock_config.log` | Mock configuration |\n\nUse `offset=-4000` instead of `offset=0` to get just the tail (last 4KB) for a quick peek.\n\n### 4. Identify the Failure\n\nThere are four main failure categories. Check them in this order:\n\n#### A. Plugin\u002Fcallback failure (buildSRPMFromSCM only) — check the **Result field in the task info HTML**\n- `CallbackError: Error running postSCMCheckout callback` — The `azldev` builder plugin failed during source preparation\n- `azldev failed with return code 1` — The Azure Linux source prep tool could not process the package\n- These errors appear in the task page Result field, NOT in any log file\n- Cause: Package spec\u002Fsources are malformed, or the azldev tool doesn't support the package layout\n\n#### B. Mock infrastructure failure (mock exit status 30) — check `mock_output.log` FIRST, then `root.log`\n- `Failed to register machine: The name org.freedesktop.machine1 was not provided` — systemd-nspawn\u002Fmachined not available on builder (infrastructure issue, retry on different host)\n- `could not init mock buildroot` — Mock chroot setup failed before builds started\n- These are builder infrastructure issues, not package issues. Retrying usually fixes them.\n\n#### C. Dependency resolution failure (mock exit status 30) — check `root.log`\n- `No match for argument:` — A `BuildRequires` package is not available in the build tag\n- `Failed to resolve the transaction` — dnf5 cannot satisfy dependencies\n- Cause: Missing packages in the build tag, build order issues, or circular dependencies\n\n#### D. Build\u002Fcompilation\u002Ftest failure (mock exit status 1) — check `build.log`\n- `error:` — Compiler or build tool errors\n- `FAIL ` \u002F `ERROR ` — Test suite failures in `%check` phase\n- `Bad exit status from` — RPM scriptlet failure (shows which phase: `%build`, `%check`, `%install`)\n- Look for the **Testsuite summary** block to see PASS\u002FFAIL\u002FERROR\u002FSKIP counts\n\n**Common root causes**:\n| Pattern | Cause | Log \u002F Location |\n|---------|-------|----------------|\n| `CallbackError...azldev` | Source prep plugin failure | Task info HTML Result field |\n| `azldev failed with return code 1` | Package spec\u002Fsource issue | Task info HTML Result field |\n| `Failed to register machine` | Builder missing systemd-machined | `mock_output.log` |\n| `could not init mock buildroot` | Mock infrastructure failure | `mock_output.log` |\n| `No match for argument: \u003Cpkg>` | Missing BuildRequires in build tag | `root.log` |\n| `Failed to resolve the transaction` | Dependency resolution failure | `root.log` |\n| `FAIL tests\u002F...` with test summary | Test failure in `%check` | `build.log` |\n| `Bad exit status from` (%check) | Test suite failed | `build.log` |\n| `Bad exit status from` (%build) | Compilation failed | `build.log` |\n| `error: unpacking of archive failed` | Corrupt source\u002FSRPM | `build.log` |\n| `no inotify_add_watch` | Container inotify limitation | `build.log` |\n| `mock exited with status 30` | Dependency OR infrastructure | `root.log` or `mock_output.log` |\n| `mock exited with status 1` | Build or test failure | `build.log` |\n\n## Upstream Reference\n\nUpstream Fedora's Koji (https:\u002F\u002Fkoji.fedoraproject.org\u002Fkoji\u002F) and the Fedora dist-git are incredibly useful references for understanding package issues. If a package is failing in Azure Linux, check if it fails in Fedora too. Check for upstream fixes to the issue, or if it's fixed in newer Fedora releases. **NEVER** assume that a package is \"just broke\", if the same package builds successfully in Fedora, then it's critical to understand WHAT the difference is between the Fedora and Azure Linux build environments that is causing the failure. This can often lead to insights about missing dependencies, unsupported configurations, or other issues in Azure Linux that need to be addressed.\n\nFor older upstream logs, you may have to reference (https:\u002F\u002Fkojipkgs.fedoraproject.org), which archives old build logs that are no longer on the main koji server. This server is usually protected by a bot challenge, you can re-purpose the `koji_fetch` tool with `override_base_url` to fetch from this server as well, it will often pass the bot challenge successfully.\n\n## Example: Full Investigation\n\n```\n# 1. Set the Koji base URL (once per session), or pass override_base_url to each call\nset_koji_url(base_url=\"https:\u002F\u002Fkoji.example.com\")\n\n# 2. Fetch parent task info → saved to temp file\nkoji_fetch(path=\"\u002Fkoji\u002Ftaskinfo?taskID=3307\")\n# → {\"output\": \"Wrote 12345 bytes (200 lines) to ...\u002Fkoji\u002Fkoji_abc123\", \"default_base_url\": \"...\", ...}\n\n# 3. Use grep_search on the saved file to find child task IDs and state\n#    grep for 'taskinfo?taskID=' to find children\n#    grep for 'failed' to find which child failed\n\n# 4. Fetch the failed child's task info (check Result field for plugin errors)\nkoji_fetch(path=\"\u002Fkoji\u002Ftaskinfo?taskID=6059\")\n\n# 5. Fetch logs in order of priority:\n# 5a. mock_output.log (infrastructure errors)\nkoji_fetch(path=\"\u002Fkoji\u002Fgetfile?taskID=6059&volume=DEFAULT&name=mock_output.log&offset=-4000\")\n# → grep the file from the \"output\" field for: ERROR, Failed to register, could not init\n\n# 5b. root.log (dependency resolution errors)\nkoji_fetch(path=\"\u002Fkoji\u002Fgetfile?taskID=6059&volume=DEFAULT&name=root.log\")\n# → grep for: No match for, Failed to resolve\n\n# 5c. build.log (build\u002Ftest errors)\nkoji_fetch(path=\"\u002Fkoji\u002Fgetfile?taskID=6059&volume=DEFAULT&name=build.log\")\n# → grep for: FAIL, ERROR, Bad exit status, error:\n\n# 5d. For exceptionally large files, consider using 'fold' on the saved file.\n```\n\n## If Koji tools are not working\n\nThe MCP server is the only supported method for fetching Koji information: it handles URL construction, SSL issues, output file creation and cleanup, etc.\n\nIf MCP tools are not working, guide the user to help them fix their MCP configuration. A (non exhaustive) list of some things to check:\n- If they are in VSCode or Copilot CLI, ensure they are running in a reasonable workspace (ie not a parent folder that doesn't have the right mcp.json file, or adhoc files that don't have a root dir)\n- If they REALLY don't want to be in a workspace, guide them through adding a global mcp configuration\n  - Copilot CLI: `~\u002F.copilot\u002Fmcp-config.json` (https:\u002F\u002Fdocs.github.com\u002Fen\u002Fcopilot\u002Fhow-tos\u002Fcopilot-cli\u002Fuse-copilot-cli#add-an-mcp-server)\n  - VSCode: user profile: (https:\u002F\u002Fcode.visualstudio.com\u002Fdocs\u002Fcopilot\u002Fcustomization\u002Fmcp-servers#_add-an-mcp-server)\n- Check that the MCP server can start - it might be missing the `mcp` python package dependency\n- Something else...\n","# Koji Build Failure Triage\n\n## Koji Web UI\n\n- **Base URL**: Get from the user, or from env variable `KOJI_BASE_URL`, or pre-configured in the MCP server.\n  - If the env variable is not set, and the MCP is not pre-configured, prompt the user to input the base URL of the Koji Web UI (e.g., `https:\u002F\u002Fkoji.example.com`).\n- **MCP tools (preferred)**: Use the `koji` MCP server tools to fetch pages and logs. These write output to temp files under `base\u002Fbuild\u002Fwork\u002Fscratch\u002Fkoji\u002F` to avoid bloating context. Use `read_file` and `grep_search` on the resulting files to inspect content.\n\n  **Available tools:** `koji_status`, `set_koji_url`, `koji_fetch`, `koji_allow_insecure`, `koji_cleanup`. All tools return a dict with the result plus server state (base URL, output dir, etc.).\n\n  1. Call `koji_status` to check if a base URL is already configured (via env or a prior call).\n  2. Either call `set_koji_url` to set a default, or pass `override_base_url` directly to `koji_fetch` per-call. Setting a default is convenient when all requests target the same instance, and `override_base_url` is useful for ad-hoc queries against other instances.\n  3. Call `koji_fetch` with the path (e.g., `\u002Fkoji\u002Ftaskinfo?taskID=3307`). It returns the file path where content was saved.\n     - If the fetch fails with an SSL certificate error (e.g., self-signed cert), ask the user if they want to skip SSL verification, then call `koji_allow_insecure` (optionally with `override_base_url`) to proceed. Do **NOT** proceed without explicit user approval for the **specific** URL.\n  4. Use `read_file` or `grep_search` on the saved file to extract the information you need.\n  5. Call `koji_cleanup` when done to remove fetched temp files and reclaim disk space (NOTE: This will clean up ALL scratch files, so only call it when you're completely done with the current files).\n- **MCP is required.** If MCP tools are not working, ask the user to fix their MCP configuration rather than falling back to CLI tools.\n- **Network issues**: Koji is typically only accessible via VPN or corporate network. If you encounter connection errors or timeouts, assume it is a network permissions issue — ask the user to confirm they are connected to the appropriate network before retrying. Do NOT keep retrying on your own.\n\n## Discovery: Finding Builds by Package Name\n\nWhen the user provides a **package name** instead of a task ID, use **both** discovery paths below to find all recent failures. Neither path alone is sufficient.\n\n> **Why two paths?** The package info page only lists builds where Koji created a build record. If a task fails early (e.g., `buildSRPMFromSCM` fails), no build record exists and the failure is invisible on the package page. The failed tasks list catches these, but doesn't filter by package — you must grep for the name. Always run both.\n\n### Path A: Package info page (finds builds with records)\n\n1. **Search for the package:**\n\n```\n\u002Fkoji\u002Fsearch?match=glob&type=package&terms=\u003CPACKAGE_NAME>\n```\n\n- If the exact name doesn't match, try a glob: `terms=\u003CNAME>*`\n- Extract the `packageID` from the results (look for `packageinfo?packageID=` links)\n- If the search returns only one match, Koji redirects directly to the package info page (skip step 2).\n\n2. **Get recent builds from the package info page:**\n\n```\n\u002Fkoji\u002Fpackageinfo?packageID=\u003CPACKAGE_ID>\n```\n\n- Look for `buildinfo?buildID=` links with state `failed`\n- Note the build NVRs (Name-Version-Release) for context\n\n3. **Get the build's parent task ID:**\n\n```\n\u002Fkoji\u002Fbuildinfo?buildID=\u003CBUILD_ID>\n```\n\n- Extract the `taskinfo?taskID=` link — this is the parent `build` task\n- The parent task's Descendants section lists the actual failed child tasks (`buildArch`, `buildSRPMFromSCM`)\n\n### Path B: Failed tasks list (catches failures with no build record)\n\nSearch the global failed tasks list for the package name:\n\n```\n\u002Fkoji\u002Ftasks?owner=&state=failed&view=flat&method=build&order=-id\n```\n\n- This lists all recent failed `build` tasks, newest first\n- **Grep the results for the package name** to find relevant task IDs\n- This catches early failures (e.g., `buildSRPMFromSCM` errors, plugin\u002Fcallback errors) that never create a build record and would be invisible on the package info page\n\n### Combine results\n\nCollect the parent task IDs from both paths, deduplicate, then for each parent task:\n1. Fetch `taskinfo?taskID=\u003CPARENT_ID>` to see the Descendants tree\n2. Identify which child task(s) failed (`buildSRPMFromSCM` or `buildArch`)\n3. Proceed to the Investigation Workflow below for each failed child\n\n### Quick-reference: Discovery URLs\n\n| Goal | URL Path |\n|------|----------|\n| Search packages by name | `\u002Fkoji\u002Fsearch?match=glob&type=package&terms=\u003CNAME>` |\n| Package info (recent builds) | `\u002Fkoji\u002Fpackageinfo?packageID=\u003CID>` |\n| Build details → parent task | `\u002Fkoji\u002Fbuildinfo?buildID=\u003CID>` |\n| All recent failed build tasks | `\u002Fkoji\u002Ftasks?owner=&state=failed&view=flat&method=build&order=-id` |\n| Search builds by NVR | `\u002Fkoji\u002Fsearch?match=glob&type=build&terms=\u003CNAME>*` |\n\n### Example: \"Why is the kernel broken?\"\n\n```\n# Path A: Package info page\n# 1. Search for the kernel package\nkoji_fetch(path=\"\u002Fkoji\u002Fsearch?match=glob&type=package&terms=kernel\")\n# → grep for 'packageinfo?packageID=' → e.g., packageID=6\n# → (if single match, Koji redirects directly to package info page)\n\n# 2. Get package info to see recent builds\nkoji_fetch(path=\"\u002Fkoji\u002Fpackageinfo?packageID=6\")\n# → grep for 'failed' and 'buildinfo?buildID=' → e.g., buildID=1482\n\n# 3. Get the parent task ID from the build info page\nkoji_fetch(path=\"\u002Fkoji\u002Fbuildinfo?buildID=1482\")\n# → grep for 'taskinfo?taskID=' → e.g., parent task 32608\n\n# Path B: Failed tasks list (run in parallel with Path A)\n# 4. Search all recent failed build tasks for \"kernel\"\nkoji_fetch(path=\"\u002Fkoji\u002Ftasks?owner=&state=failed&view=flat&method=build&order=-id\")\n# → grep for 'kernel' → may find additional tasks (e.g., task 87474)\n#    that failed before creating a build record\n\n# Combine: parent tasks = {32608, 87474}\n# 5. For each parent, fetch task info and find the failed child\nkoji_fetch(path=\"\u002Fkoji\u002Ftaskinfo?taskID=32608\")\n# → Descendants: buildArch x86_64 (task 39696) = failed\nkoji_fetch(path=\"\u002Fkoji\u002Ftaskinfo?taskID=87474\")\n# → Descendants: buildSRPMFromSCM (task 87475) = failed\n\n# 6. Investigate each failed child using the workflow below\n```\n\n## Investigation Workflow\n\n> **Branch mismatch warning:** The branch checked out locally may not correspond to the code that Koji built. If the local component definition, spec, or overlays don't seem to match what the build logs show, check which branch\u002Fcommit Koji used (visible in the task info page or `checkout.log`) and compare with the current local branch (`git branch --show-current` \u002F `git log --oneline -1`). Use non-destructive git commands to inspect the Koji branch without modifying the working tree — e.g., `git show \u003Ckoji-ref>:\u003Cpath>` to read files, `git diff HEAD..\u003Ckoji-ref> -- \u003Cpath>` to compare. Avoid `git checkout` to switch branches, as the user may have uncommitted work.\n\n### 1. Get Task Info\n\nFetch the task page and extract key details (state, method, children).\n\nIf you don't know the status of the koji tool, check its status.\n\nDecide if the base URL should be changed (is this a one-off, or long-term investigation?).\n\nCall `koji_fetch` with path `\u002Fkoji\u002Ftaskinfo?taskID=\u003CTASK_ID>`, either using the default base URL or passing an `override_base_url`. Use either built-in tools or cmdline tools on the saved file to extract state, method, and child task IDs.\n\nParse the HTML to find:\n- **State**: `failed`, `closed` (success), `canceled`, `free`, `open`\n- **Method**: `build` (parent), `buildSRPMFromSCM`, `buildArch`\n- **Child task IDs**: Look for `taskinfo?taskID=` links in the Descendants section\n\nA `build` task always has children: one `buildSRPMFromSCM` (SRPM creation) and one or more `buildArch` tasks (per architecture). The parent fails if any child fails.\n\n### 2. Find the Failed Child\n\nExtract all `taskinfo?taskID=\u003CID>` links from the parent task page. Then fetch each child task page and check its state. Typically:\n- `buildSRPMFromSCM` succeeds\n- One of the `buildArch` tasks fails (look for `state.*failed` in the HTML)\n\n**Important**: Also check the **Result** field in the task info HTML. For some failures (especially `buildSRPMFromSCM`), the error message is embedded directly in the page under \"Result\" and is NOT in any downloadable log file.\n\n### 3. Download Logs\n\nDownload logs from the failed child task.\n\nCall `koji_fetch` with the path `\u002Fkoji\u002Fgetfile?taskID=\u003CCHILD_TASK_ID>&volume=DEFAULT&name=\u003CFILENAME>&offset=0`. The content is saved to a temp file — use `grep_search` or `read_file` to inspect it. For large logs, fetch with `&offset=-4000` first to get the tail.\n\nAvailable log files (listed on the child task page):\n| File | Contains |\n|------|----------|\n| `root.log` | Mock chroot setup, dependency resolution, RPM build commands |\n| `build.log` | RPM build output (`%prep`, `%build`, `%install`, `%check`) |\n| `mock_output.log` | Mock orchestration output, systemd-nspawn errors |\n| `checkout.log` | Git clone\u002Ffetch output (buildSRPMFromSCM tasks only) |\n| `dnf5.log` | Package manager operations |\n| `state.log` | Mock state transitions |\n| `hw_info.log` | Builder hardware info |\n| `mock_config.log` | Mock configuration |\n\nUse `offset=-4000` instead of `offset=0` to get just the tail (last 4KB) for a quick peek.\n\n### 4. Identify the Failure\n\nThere are four main failure categories. Check them in this order:\n\n#### A. Plugin\u002Fcallback failure (buildSRPMFromSCM only) — check the **Result field in the task info HTML**\n- `CallbackError: Error running postSCMCheckout callback` — The `azldev` builder plugin failed during source preparation\n- `azldev failed with return code 1` — The Azure Linux source prep tool could not process the package\n- These errors appear in the task page Result field, NOT in any log file\n- Cause: Package spec\u002Fsources are malformed, or the azldev tool doesn't support the package layout\n\n#### B. Mock infrastructure failure (mock exit status 30) — check `mock_output.log` FIRST, then `root.log`\n- `Failed to register machine: The name org.freedesktop.machine1 was not provided` — systemd-nspawn\u002Fmachined not available on builder (infrastructure issue, retry on different host)\n- `could not init mock buildroot` — Mock chroot setup failed before builds started\n- These are builder infrastructure issues, not package issues. Retrying usually fixes them.\n\n#### C. Dependency resolution failure (mock exit status 30) — check `root.log`\n- `No match for argument:` — A `BuildRequires` package is not available in the build tag\n- `Failed to resolve the transaction` — dnf5 cannot satisfy dependencies\n- Cause: Missing packages in the build tag, build order issues, or circular dependencies\n\n#### D. Build\u002Fcompilation\u002Ftest failure (mock exit status 1) — check `build.log`\n- `error:` — Compiler or build tool errors\n- `FAIL ` \u002F `ERROR ` — Test suite failures in `%check` phase\n- `Bad exit status from` — RPM scriptlet failure (shows which phase: `%build`, `%check`, `%install`)\n- Look for the **Testsuite summary** block to see PASS\u002FFAIL\u002FERROR\u002FSKIP counts\n\n**Common root causes**:\n| Pattern | Cause | Log \u002F Location |\n|---------|-------|----------------|\n| `CallbackError...azldev` | Source prep plugin failure | Task info HTML Result field |\n| `azldev failed with return code 1` | Package spec\u002Fsource issue | Task info HTML Result field |\n| `Failed to register machine` | Builder missing systemd-machined | `mock_output.log` |\n| `could not init mock buildroot` | Mock infrastructure failure | `mock_output.log` |\n| `No match for argument: \u003Cpkg>` | Missing BuildRequires in build tag | `root.log` |\n| `Failed to resolve the transaction` | Dependency resolution failure | `root.log` |\n| `FAIL tests\u002F...` with test summary | Test failure in `%check` | `build.log` |\n| `Bad exit status from` (%check) | Test suite failed | `build.log` |\n| `Bad exit status from` (%build) | Compilation failed | `build.log` |\n| `error: unpacking of archive failed` | Corrupt source\u002FSRPM | `build.log` |\n| `no inotify_add_watch` | Container inotify limitation | `build.log` |\n| `mock exited with status 30` | Dependency OR infrastructure | `root.log` or `mock_output.log` |\n| `mock exited with status 1` | Build or test failure | `build.log` |\n\n## Upstream Reference\n\nUpstream Fedora's Koji (https:\u002F\u002Fkoji.fedoraproject.org\u002Fkoji\u002F) and the Fedora dist-git are incredibly useful references for understanding package issues. If a package is failing in Azure Linux, check if it fails in Fedora too. Check for upstream fixes to the issue, or if it's fixed in newer Fedora releases. **NEVER** assume that a package is \"just broke\", if the same package builds successfully in Fedora, then it's critical to understand WHAT the difference is between the Fedora and Azure Linux build environments that is causing the failure. This can often lead to insights about missing dependencies, unsupported configurations, or other issues in Azure Linux that need to be addressed.\n\nFor older upstream logs, you may have to reference (https:\u002F\u002Fkojipkgs.fedoraproject.org), which archives old build logs that are no longer on the main koji server. This server is usually protected by a bot challenge, you can re-purpose the `koji_fetch` tool with `override_base_url` to fetch from this server as well, it will often pass the bot challenge successfully.\n\n## Example: Full Investigation\n\n```\n# 1. Set the Koji base URL (once per session), or pass override_base_url to each call\nset_koji_url(base_url=\"https:\u002F\u002Fkoji.example.com\")\n\n# 2. Fetch parent task info → saved to temp file\nkoji_fetch(path=\"\u002Fkoji\u002Ftaskinfo?taskID=3307\")\n# → {\"output\": \"Wrote 12345 bytes (200 lines) to ...\u002Fkoji\u002Fkoji_abc123\", \"default_base_url\": \"...\", ...}\n\n# 3. Use grep_search on the saved file to find child task IDs and state\n#    grep for 'taskinfo?taskID=' to find children\n#    grep for 'failed' to find which child failed\n\n# 4. Fetch the failed child's task info (check Result field for plugin errors)\nkoji_fetch(path=\"\u002Fkoji\u002Ftaskinfo?taskID=6059\")\n\n# 5. Fetch logs in order of priority:\n# 5a. mock_output.log (infrastructure errors)\nkoji_fetch(path=\"\u002Fkoji\u002Fgetfile?taskID=6059&volume=DEFAULT&name=mock_output.log&offset=-4000\")\n# → grep the file from the \"output\" field for: ERROR, Failed to register, could not init\n\n# 5b. root.log (dependency resolution errors)\nkoji_fetch(path=\"\u002Fkoji\u002Fgetfile?taskID=6059&volume=DEFAULT&name=root.log\")\n# → grep for: No match for, Failed to resolve\n\n# 5c. build.log (build\u002Ftest errors)\nkoji_fetch(path=\"\u002Fkoji\u002Fgetfile?taskID=6059&volume=DEFAULT&name=build.log\")\n# → grep for: FAIL, ERROR, Bad exit status, error:\n\n# 5d. For exceptionally large files, consider using 'fold' on the saved file.\n```\n\n## If Koji tools are not working\n\nThe MCP server is the only supported method for fetching Koji information: it handles URL construction, SSL issues, output file creation and cleanup, etc.\n\nIf MCP tools are not working, guide the user to help them fix their MCP configuration. A (non exhaustive) list of some things to check:\n- If they are in VSCode or Copilot CLI, ensure they are running in a reasonable workspace (ie not a parent folder that doesn't have the right mcp.json file, or adhoc files that don't have a root dir)\n- If they REALLY don't want to be in a workspace, guide them through adding a global mcp configuration\n  - Copilot CLI: `~\u002F.copilot\u002Fmcp-config.json` (https:\u002F\u002Fdocs.github.com\u002Fen\u002Fcopilot\u002Fhow-tos\u002Fcopilot-cli\u002Fuse-copilot-cli#add-an-mcp-server)\n  - VSCode: user profile: (https:\u002F\u002Fcode.visualstudio.com\u002Fdocs\u002Fcopilot\u002Fcustomization\u002Fmcp-servers#_add-an-mcp-server)\n- Check that the MCP server can start - it might be missing the `mcp` python package dependency\n- Something else...\n",{"name":5,"description":7},"4.0","431ad36a7a827e8ec414e96ae75517ec1ded115d",".agents\u002Fskills\u002Fskill-koji-triage",714,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fazurelinux","https:\u002F\u002Fgithub.com\u002Fmicrosoft","https:\u002F\u002Fopensource.microsoft.com","Open source projects and samples from Microsoft","2026-11-15 02:36:52",{"source":8,"skillName":5,"commands":60,"prompt":70,"zipAvailable":71},[61,64,67],{"tool":62,"command":63},"npx","npx skills add microsoft\u002Fazurelinux --skill skill-koji-triage",{"tool":65,"command":66},"bunx","bunx skills add microsoft\u002Fazurelinux --skill skill-koji-triage",{"tool":68,"command":69},"pnpm","pnpm dlx skills add microsoft\u002Fazurelinux --skill skill-koji-triage","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fazurelinux\u002Fblob\u002F4.0\u002F.agents\u002Fskills\u002Fskill-koji-triage\u002FSKILL.md",true,[],[74,98,118,140,164,181],{"skillNo":75,"name":76,"nameZh":6,"description":77,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":78,"categories":79,"keywords":81,"publishedAt":45},"01M69AX4PP9MJPS4TE3XDW93NJ","skill-metrics","[Skill] metrics, azure-monitor, aks - Query Azure Monitor metrics for Koji AKS node CPU, memory, disk usage, and pod readiness.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fazurelinux\u002Fblob\u002F4.0\u002F.agents\u002Fskills\u002Fskill-metrics\u002FSKILL.md",[80],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},[82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,29],"skill metrics","metrics","memory","azure","monitor","azure-monitor","aks","disk","node","query","azure monitor","false","readiness","CPU","disk usage","usage",{"skillNo":99,"name":100,"nameZh":6,"description":101,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":102,"categories":103,"keywords":105,"publishedAt":45},"01M69AX2WT9YWZTVMMQ6QWHG3Y","skill-mass-triage","[Skill] Batch-triage build failures from a JSON results file — diagnose with parallel sub-agents, bucketize by root cause, and produce a consolidated summary. Use when asked to triage a Koji results file. Triggers: batch triage, mass triage, triage results file, bucketize failures.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fazurelinux\u002Fblob\u002F4.0\u002F.agents\u002Fskills\u002Fskill-mass-triage\u002FSKILL.md",[104],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},[25,106,107,108,109,110,111,112,113,36,114,115,32,116,117,28,27],"skill mass triage","diagnose","produce","batch","file","json","parallel","results","build","summary","with","root",{"skillNo":119,"name":120,"nameZh":6,"description":121,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":122,"categories":123,"keywords":128,"publishedAt":45},"01M69AX11X7RJS8A0GBJV2X0WD","skill-kql-queries","[Skill] kql, queries, log analytics, container logs - KQL query templates for Koji container logs, pod errors, restarts, Kubernetes events, build job activity, and node resource usage via Log Analytics.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fazurelinux\u002Fblob\u002F4.0\u002F.agents\u002Fskills\u002Fskill-kql-queries\u002FSKILL.md",[124,127],{"categoryNo":125,"categoryName":126,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000011","运维与部署",{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},[129,130,131,132,133,39,134,90,135,35,114,91,93,136,137,138,139,97,29],"skill kql queries","kql","queries","kubernetes","analytics","templates","container","events","via","activity","errors",{"skillNo":141,"name":142,"nameZh":6,"description":143,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":144,"categories":145,"keywords":147,"publishedAt":45},"01M69AWWNGM4S4DDN6MSXMJ1XY","skill-fedora-mirror-injections","[Skill] Diagnose Azure Linux Stage 1 (bootstrap-mirror) BuildRequires resolution failures and backfill missing RPMs into the prod Fedora updates mirror via the azl-infra injections file. Use when triaging stage-1 \u002F azl4-bootstrap-mirror build failures, finding dependency gaps in the Fedora mirror, deciding whether a failure is a real mirror gap vs a transient timeout or a spec bug, adding entries to azl4-stage1-injections.yaml, or opening a PR with mirror injections. Triggers: bootstrap-mirror, stage 1 build failure, fedora mirror gap, missing BuildRequires, nothing provides crate, no match for argument, injections file, azl4-stage1-injections, mirror injection.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fazurelinux\u002Fblob\u002F4.0\u002F.agents\u002Fskills\u002Fskill-fedora-mirror-injections\u002FSKILL.md",[146],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},[148,149,150,151,152,107,85,153,154,155,36,156,157,158,159,32,160,161,162,29,163,27],"skill fedora mirror injections","fedora","mirror","spec","linux","pr","bootstrap","injection","dependency","bug","infra","entries","missing","argument","updates","backfill",{"skillNo":165,"name":166,"nameZh":6,"description":167,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":168,"categories":169,"keywords":171,"publishedAt":45},"01M69AWTVNFMTX4KQVR6EK62C0","skill-deployment-context","[Skill] deployment context, aks, azure, koji - Resolve Koji AKS deployment context -- resource group, cluster name, subscription, Log Analytics workspace, and monitoring resource names from deployment_summary.yaml when available, or discover them dynamically from Azure.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fazurelinux\u002Fblob\u002F4.0\u002F.agents\u002Fskills\u002Fskill-deployment-context\u002FSKILL.md",[170],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},[172,173,174,133,85,175,176,88,177,35,115,178,93,179,180,27],"skill deployment context","deployment","context","yaml","monitoring","workspace","discover","name","cluster",{"skillNo":182,"name":183,"nameZh":6,"description":184,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":185,"categories":186,"keywords":188,"publishedAt":45},"01M69AWS0NN17EN4JDDKYBNY6X","skill-aks-health","[Skill] aks, aks health, cluster health, node pool, activity logs - Inspect Koji AKS cluster health, node pool status, autoscaling, activity logs, deployment failures, and Azure control-plane operations.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fazurelinux\u002Fblob\u002F4.0\u002F.agents\u002Fskills\u002Fskill-aks-health\u002FSKILL.md",[187],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},[189,88,190,191,85,173,39,192,90,193,194,93,195,180,138,196,197,198,199],"skill aks health","health","status","control-plane","operations","control","cluster health","plane","inspect","pool","autoscaling"]