[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7B4QD02N0YBVT0838Z5K5VC":3,"skill-same:01M7B4QD02N0YBVT0838Z5K5VC":98},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":27,"commitSha":28,"skillDirectory":29,"forksCount":30,"repositoryHtmlUrl":31,"repositoryHomepageUrl":6,"ownerHtmlUrl":32,"ownerBlogUrl":33,"ownerDescription":34,"updatedAt":35,"install":36,"resources":49},"01M7B4QD02N0YBVT0838Z5K5VC","agent-platform-autorater-customization",null,"Customizes an autorater (evaluation metric) for an AI agent so its grades match the user's own judgment, starting from the agent's real trajectories. Use when a user wants a metric tailored to their agent, wants to turn failing trajectories into rubrics, says an existing autorater grades their agent wrong, or wants an autorater to agree with human labels (autorater hillclimbing, judge calibration). Don't use for quick evals with built-in metrics or for improving the agent itself (use agent-platform-eval-flywheel).","google\u002Fskills","google","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F1342004?v=4","01M698VYEFDM0VJBZDYYJ2T83S","Google","01M698WR0H76ENHE587T9EJT89",24235,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fagent-platform-autorater-customization\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: agent-platform-autorater-customization\nmetadata:\n  version: \"1.0.0\"\n  category: AiAndMachineLearning\ndescription: >-\n  Customizes an autorater (evaluation metric) for an AI agent so its grades\n  match the user's own judgment, starting from the agent's real trajectories.\n  Use when a user wants a metric tailored to their agent, wants to turn failing\n  trajectories into rubrics, says an existing autorater grades their agent\n  wrong, or wants an autorater to agree with human labels (autorater\n  hillclimbing, judge calibration). Don't use for quick evals with built-in\n  metrics or for improving the agent itself (use agent-platform-eval-flywheel).\n---\n\n# Agent Platform Autorater Customization\n\nThis skill helps a user build graders for their agent that judge the way they\ndo. You do the reading, writing and testing; the user makes the judgment calls.\nThe result is a set of metrics registered in the Agent Platform evaluation\nservice, one or more for each criterion the user cares about.\n\nCarry out the steps below in order. Don't summarize them to the user.\n\n## Usage Attribution\n\n**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you\nMUST read and follow the instructions in\n[Usage Attribution](references\u002Fattribution.md).\n\n## Setup\n\nCheck each item before the first evaluation call, and ask the user to fix what\nis missing. Ask before installing anything, because it changes the user's\nenvironment.\n\n1.  **Python SDK.** Install or upgrade the Agent Platform SDK with the\n    evaluation extra:\n\n    ```bash\n    pip install --upgrade 'google-cloud-aiplatform[evaluation]>=2.8.0,\u003C3'\n    ```\n\n    Keep the quotes; unquoted, the shell reads `>=` as a redirect. The\n    `gcloud` CLI is also needed for authentication; if it is missing, point the\n    user to https:\u002F\u002Fcloud.google.com\u002Fsdk\u002Fdocs\u002Finstall instead of installing it.\n2.  **Project and billing.** Use the project the user names, or ask for one\n    (`gcloud config set project {project_id}`). The project needs an active\n    billing account: evaluation calls are billed as model usage. See\n    https:\u002F\u002Fcloud.google.com\u002Fpricing for current prices; don't quote numbers.\n3.  **API.** Enable the Agent Platform API:\n    `gcloud services enable aiplatform.googleapis.com`.\n4.  **Authentication.** Run `gcloud auth login` for the CLI and\n    `gcloud auth application-default login` for the SDK and scripts, or set\n    `GOOGLE_APPLICATION_CREDENTIALS` to a service account key file.\n5.  **IAM.** The user or service account needs `roles\u002Faiplatform.user` to run\n    evaluations and register metrics. Don't grant broader roles such as\n    `roles\u002Faiplatform.admin`; ask the user to request the narrower role if it\n    is missing.\n6.  **Region.** Use `us-central1` unless the user needs another region (see\n    https:\u002F\u002Fcloud.google.com\u002Fabout\u002Flocations).\n\n## Working with the user\n\n-   At each **Approval** point, show the proposal and ask: \"Would you like to\n    change anything, or go ahead with this version?\" Wait for a clear yes;\n    silence is not approval.\n-   For setup choices (project, region, storage), suggest a default.\n-   Everything the user reviews (cases, grades, metric definitions) goes in a\n    review page, not in chat; see [Review pages](#review-pages). Chat updates\n    say what you did, the numbers, and what you need.\n-   When you ask about a trajectory in chat, quote the relevant lines.\n\n## Inputs\n\nThe user brings a folder of trajectories: the ones they think went wrong, plus\nsome good ones. Optional: a note or rating per trajectory (thumbs, human score,\nan old autorater's verdict), an existing rubric, or reference answers.\n\n-   Use all of them unless grading them would be slow or costly in the\n    evaluation service. Then agree a sample with the user that covers the\n    failures and a few good cases, and say what was left out.\n-   AgentData JSONL is best; see\n    [Converting trajectories](references\u002Fconverting_trajectories.md). If you\n    can't map a format to AgentData with confidence, don't guess: say you need\n    to convert it to AgentData, and ask the user for a schema, a converter, or\n    one example mapped by hand. Then check every converted case.\n-   Building a dataset is not this skill's job. If the user has no folder yet,\n    help them pick and convert trajectories, then continue.\n\n## Terms\n\n-   **Criterion:** one thing the user cares about in the agent's behavior: the\n    answer is correct, the right tool is called, the system instructions are\n    followed throughout. A criterion holds for the whole dataset; every\n    trajectory, or every turn, is graded on it, and it can be N\u002FA for some.\n-   **Rubric:** one yes\u002Fno check.\n-   **Metric:** what grades a criterion in the evaluation service: code, one\n    rubric, or several weighted rubrics. A criterion gets as many metrics as it\n    needs, for example a code check for the tool call plus a judge for the\n    answer.\n\nThe predefined multi-turn metrics also turn intents into rubrics with an\nimportance, but they score the unweighted share of rubrics that pass. Custom\nmetrics add rubrics written for the user's agent, weights, must-pass rubrics,\nand a pass threshold.\n\n## Steps\n\n### 1. Confirm the inputs\n\nAsk where the folder is and whether any trajectories have a note or rating.\n\n### 2. Propose criteria (Approval)\n\nRead the trajectories and the user's notes, and propose the criteria the data\nsupports, as few as cover the problems the user cares about. They can cover any\npart of the trajectory: the final answer, tool calls and arguments, following\nthe system instructions, handling of errors or clarifications. In a multi-agent\nsystem, a criterion can be about the whole system (routing, the final answer) or\nabout one sub-agent; say which. For each, give:\n\n-   a one-line statement and why it matters for this agent;\n-   how many cases show a problem, with one quoted example;\n-   the metrics you plan to build for it (see step 4), with one line on why.\n\nNever propose a criterion you can't point to in the data or the user's notes.\nFlag any criterion you doubt, with the reason (for example, it can't be graded\nwhere tool results were trimmed). Also note problems in any existing grading: a\njudge that never saw data the agent had, evidence trimmed from the trajectory,\ngolden answers that were never run.\n\nThis step makes no evaluation-service calls. If the user wants help finding\npatterns in ten or more failing cases, you can grade them with\n`MULTI_TURN_TASK_SUCCESS` and run `client.evals.generate_loss_clusters` after\nstep 3. Loss clusters only work in the `global` region and for\n`multi_turn_task_success` and `multi_turn_tool_use_quality`, not for custom\nmetrics.\n\nSave the approved criteria as `criteria.md`.\n\n### 3. Agree data rules and spend (Approval)\n\nAsk whether the trajectories may be sent to the evaluation service, which\nproject to bill, and where outputs may be stored. Show the number of calls\n(cases × metrics × samples × rounds, with one call per step for step-based\nmetrics) and point to https:\u002F\u002Fcloud.google.com\u002Fpricing for the cost. Ask again\nbefore any run beyond that.\n\n### 4. Build the metrics\n\nUse the simplest form that grades the criterion: simpler forms cost less, give\nthe same answer every time, and are easier to trust.\n\nCriterion can be graded by                                                  | Form                    | SDK\n--------------------------------------------------------------------------- | ----------------------- | ---\nAn exact rule on the output or trajectory (a tool name, a format, a number) | Code                    | `types.CodeExecutionMetric` (runs in the service; can be registered)\nOne judgment that is the same for every case                                | LLM, one rubric         | `types.LLMMetric` with a yes\u002Fno check and a reference\nSeveral checks, some more important than others                             | LLM, weighted rubrics   | `types.StepBasedLLMMetric`: one judge step; code computes the weighted score\nChecks that differ per case, with no good reference                         | Generated checklist     | `types.StepBasedLLMMetric`: one step writes checks from the conversation, one grades them\nLong or multi-turn traces, per-turn verdicts, or several judging stages     | Step-based, multi-stage | `types.StepBasedLLMMetric`\nOne sub-agent of a multi-agent system                                       | Step-based, one agent   | `types.StepBasedLLMMetric` with a code preprocessor that keeps that agent's events\n\nStart each metric from its template in\n[Metric templates](references\u002Fmetric_templates.md); each template says what it\ndoesn't catch. For weighted rubrics, propose the rubrics, a weight for each, the\nmust-pass rubrics and a pass threshold, and keep the weights in the parser code\nso the judge grades each rubric on its own. Write checks as described in\n[Writing LLM checks](references\u002Fwriting_checks.md). For multi-turn\nconversations, decide per criterion whether to grade each turn or the whole\nconversation; see [Multi-turn conversations](references\u002Fmulti_turn.md).\n\n### 5. Grade and show the results\n\nGrade the cases through the evaluation service with `scripts\u002Frun_metrics.py`,\nthen build a review page with `scripts\u002Fbuild_review_page.py`. Pass each metric's\npass threshold as `--threshold`, so PASS and FAIL on the page match the metric.\n\nOptional: if the user has an existing autorater, or wants to compare with a\nbuilt-in metric such as `MULTI_TURN_TASK_SUCCESS`, grade with it too and show it\nas a baseline. It adds calls, and a built-in metric grades overall task success\nrather than the user's criteria.\n\n### 6. Optional: align the metrics with the user's ratings\n\nAsk whether the user wants to mark the grades and have you improve the metrics,\nat about a minute per case. Skip to step 7 if not.\n\n1.  Set aside about 20% of the cases as held back. Build the review page without\n    them.\n2.  The user marks each verdict Agree or Disagree on the review page, with a\n    reason when they disagree, and gives you `labels.json`. If the user already\n    has human ratings, use them and ask only about the cases where a metric\n    disagrees.\n3.  Compute agreement per metric and per rubric with `scripts\u002Fagreement.py`.\n    Measure judge noise before trusting a change; see\n    [Noise and agreement](references\u002Fnoise_and_agreement.md).\n4.  Revise the metrics yourself (rubric wording, weights, must-pass rubrics,\n    threshold, reference, code), save each round as a new version, grade again,\n    and show the user what changed and the new agreement. When a label\n    contradicts the trajectory, quote the lines and ask the user.\n5.  Stop when agreement stops improving by more than the noise, or when the user\n    is satisfied. Grade the held-back cases once. Don't revise on them; if a\n    held-back case breaks, report overfitting and suggest more labeled cases.\n\n### 7. Review and register (Approval)\n\nRegistering the metrics is the goal of this skill. Add to the review page, for\neach metric: its criterion, the final definition (rubrics, weights, must-pass\nrubrics, threshold, judge model, sample count), its grades, agreement on labeled\nand held-back cases if step 6 ran, and open questions. After approval, register\neach metric with `create_evaluation_metric`; it creates a shared resource.\n\nHand over an output folder with:\n\n-   `criteria.md`: the approved criteria, each listing the metrics that grade\n    it;\n-   `metrics\u002F`: the final definition of every metric, as the file the evaluation\n    service ran (the metrics file passed to `scripts\u002Frun_metrics.py`), with its\n    registered name;\n-   `metrics.md`: one entry per metric with its criterion, form, rubrics,\n    weights, must-pass rubrics, threshold, judge model and sample count, and a\n    code snippet that runs it by registered name;\n-   `labels.json` and earlier versions, if step 6 ran;\n-   `report.md`: the numbers and open questions.\n\nTo run the metrics on a full dataset, point to `agent-platform-eval-flywheel`.\n\n## Review pages\n\nBuild one self-contained HTML page per round with\n`scripts\u002Fbuild_review_page.py`, and give the user its path (or serve it where\nthe user's browser can reach it). For each case it shows the user's note or\nrating, each metric's PASS\u002FFAIL, score, explanation and rubric verdicts, the\nbaseline if one ran, and the trajectory with tool results collapsed. The user\nmarks Agree or Disagree on each verdict with a reason; marks are saved in the\nbrowser, and \"Download labels.json\" exports them in the format\n`scripts\u002Fagreement.py` reads. Never ask the user to type labels in chat.\n\n## Rules\n\n-   All grading goes through the evaluation service. Labels come only from the\n    user: don't grade trajectories or draft labels with your own model calls or\n    subagents, because tuning a metric to another model's labels only teaches\n    one model to agree with another.\n-   Grade only the cases in the folder while building; the full dataset is for\n    after registration.\n-   Every number you report comes from a result file. Report a range when runs\n    differ.\n-   Keep proprietary trajectories and outputs, including review pages, where the\n    user said. Never paste them into shared docs, bugs or chats.\n-   Read the installed SDK source before using a name; versions differ.\n\n## Cleanup\n\nRegistered metrics stay in the project until deleted. When the user wants to\nremove a metric they no longer use, or one registered by mistake, confirm the\nname with them, then run\n`client.evals.delete_evaluation_metric(metric_resource_name={name})`. List them\nwith `client.evals.list_evaluation_metrics()`. Local outputs (results, review\npages, labels) stay where the user chose; offer to delete them at the end.\n\n## Scripts\n\nAll have `--help`.\n\n-   `scripts\u002Frender_agent_data.py`: trajectories as readable text.\n-   `scripts\u002Frun_metrics.py`: runs SDK metrics with `client.evals.evaluate()`.\n-   `scripts\u002Fbuild_review_page.py`: the review page and labels export.\n-   `scripts\u002Fagreement.py`: agreement, Cohen's kappa and disagreements, per\n    metric, per rubric or per turn.\n\n## References\n\n-   [Converting trajectories](references\u002Fconverting_trajectories.md)\n-   [Metric templates](references\u002Fmetric_templates.md)\n-   [Writing LLM checks](references\u002Fwriting_checks.md)\n-   [Multi-turn conversations](references\u002Fmulti_turn.md)\n-   [Noise and agreement](references\u002Fnoise_and_agreement.md)\n-   [SDK gotchas](references\u002Fsdk_gotchas.md)\n","# Agent Platform Autorater Customization\n\nThis skill helps a user build graders for their agent that judge the way they\ndo. You do the reading, writing and testing; the user makes the judgment calls.\nThe result is a set of metrics registered in the Agent Platform evaluation\nservice, one or more for each criterion the user cares about.\n\nCarry out the steps below in order. Don't summarize them to the user.\n\n## Usage Attribution\n\n**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you\nMUST read and follow the instructions in\n[Usage Attribution](references\u002Fattribution.md).\n\n## Setup\n\nCheck each item before the first evaluation call, and ask the user to fix what\nis missing. Ask before installing anything, because it changes the user's\nenvironment.\n\n1.  **Python SDK.** Install or upgrade the Agent Platform SDK with the\n    evaluation extra:\n\n    ```bash\n    pip install --upgrade 'google-cloud-aiplatform[evaluation]>=2.8.0,\u003C3'\n    ```\n\n    Keep the quotes; unquoted, the shell reads `>=` as a redirect. The\n    `gcloud` CLI is also needed for authentication; if it is missing, point the\n    user to https:\u002F\u002Fcloud.google.com\u002Fsdk\u002Fdocs\u002Finstall instead of installing it.\n2.  **Project and billing.** Use the project the user names, or ask for one\n    (`gcloud config set project {project_id}`). The project needs an active\n    billing account: evaluation calls are billed as model usage. See\n    https:\u002F\u002Fcloud.google.com\u002Fpricing for current prices; don't quote numbers.\n3.  **API.** Enable the Agent Platform API:\n    `gcloud services enable aiplatform.googleapis.com`.\n4.  **Authentication.** Run `gcloud auth login` for the CLI and\n    `gcloud auth application-default login` for the SDK and scripts, or set\n    `GOOGLE_APPLICATION_CREDENTIALS` to a service account key file.\n5.  **IAM.** The user or service account needs `roles\u002Faiplatform.user` to run\n    evaluations and register metrics. Don't grant broader roles such as\n    `roles\u002Faiplatform.admin`; ask the user to request the narrower role if it\n    is missing.\n6.  **Region.** Use `us-central1` unless the user needs another region (see\n    https:\u002F\u002Fcloud.google.com\u002Fabout\u002Flocations).\n\n## Working with the user\n\n-   At each **Approval** point, show the proposal and ask: \"Would you like to\n    change anything, or go ahead with this version?\" Wait for a clear yes;\n    silence is not approval.\n-   For setup choices (project, region, storage), suggest a default.\n-   Everything the user reviews (cases, grades, metric definitions) goes in a\n    review page, not in chat; see [Review pages](#review-pages). Chat updates\n    say what you did, the numbers, and what you need.\n-   When you ask about a trajectory in chat, quote the relevant lines.\n\n## Inputs\n\nThe user brings a folder of trajectories: the ones they think went wrong, plus\nsome good ones. Optional: a note or rating per trajectory (thumbs, human score,\nan old autorater's verdict), an existing rubric, or reference answers.\n\n-   Use all of them unless grading them would be slow or costly in the\n    evaluation service. Then agree a sample with the user that covers the\n    failures and a few good cases, and say what was left out.\n-   AgentData JSONL is best; see\n    [Converting trajectories](references\u002Fconverting_trajectories.md). If you\n    can't map a format to AgentData with confidence, don't guess: say you need\n    to convert it to AgentData, and ask the user for a schema, a converter, or\n    one example mapped by hand. Then check every converted case.\n-   Building a dataset is not this skill's job. If the user has no folder yet,\n    help them pick and convert trajectories, then continue.\n\n## Terms\n\n-   **Criterion:** one thing the user cares about in the agent's behavior: the\n    answer is correct, the right tool is called, the system instructions are\n    followed throughout. A criterion holds for the whole dataset; every\n    trajectory, or every turn, is graded on it, and it can be N\u002FA for some.\n-   **Rubric:** one yes\u002Fno check.\n-   **Metric:** what grades a criterion in the evaluation service: code, one\n    rubric, or several weighted rubrics. A criterion gets as many metrics as it\n    needs, for example a code check for the tool call plus a judge for the\n    answer.\n\nThe predefined multi-turn metrics also turn intents into rubrics with an\nimportance, but they score the unweighted share of rubrics that pass. Custom\nmetrics add rubrics written for the user's agent, weights, must-pass rubrics,\nand a pass threshold.\n\n## Steps\n\n### 1. Confirm the inputs\n\nAsk where the folder is and whether any trajectories have a note or rating.\n\n### 2. Propose criteria (Approval)\n\nRead the trajectories and the user's notes, and propose the criteria the data\nsupports, as few as cover the problems the user cares about. They can cover any\npart of the trajectory: the final answer, tool calls and arguments, following\nthe system instructions, handling of errors or clarifications. In a multi-agent\nsystem, a criterion can be about the whole system (routing, the final answer) or\nabout one sub-agent; say which. For each, give:\n\n-   a one-line statement and why it matters for this agent;\n-   how many cases show a problem, with one quoted example;\n-   the metrics you plan to build for it (see step 4), with one line on why.\n\nNever propose a criterion you can't point to in the data or the user's notes.\nFlag any criterion you doubt, with the reason (for example, it can't be graded\nwhere tool results were trimmed). Also note problems in any existing grading: a\njudge that never saw data the agent had, evidence trimmed from the trajectory,\ngolden answers that were never run.\n\nThis step makes no evaluation-service calls. If the user wants help finding\npatterns in ten or more failing cases, you can grade them with\n`MULTI_TURN_TASK_SUCCESS` and run `client.evals.generate_loss_clusters` after\nstep 3. Loss clusters only work in the `global` region and for\n`multi_turn_task_success` and `multi_turn_tool_use_quality`, not for custom\nmetrics.\n\nSave the approved criteria as `criteria.md`.\n\n### 3. Agree data rules and spend (Approval)\n\nAsk whether the trajectories may be sent to the evaluation service, which\nproject to bill, and where outputs may be stored. Show the number of calls\n(cases × metrics × samples × rounds, with one call per step for step-based\nmetrics) and point to https:\u002F\u002Fcloud.google.com\u002Fpricing for the cost. Ask again\nbefore any run beyond that.\n\n### 4. Build the metrics\n\nUse the simplest form that grades the criterion: simpler forms cost less, give\nthe same answer every time, and are easier to trust.\n\nCriterion can be graded by                                                  | Form                    | SDK\n--------------------------------------------------------------------------- | ----------------------- | ---\nAn exact rule on the output or trajectory (a tool name, a format, a number) | Code                    | `types.CodeExecutionMetric` (runs in the service; can be registered)\nOne judgment that is the same for every case                                | LLM, one rubric         | `types.LLMMetric` with a yes\u002Fno check and a reference\nSeveral checks, some more important than others                             | LLM, weighted rubrics   | `types.StepBasedLLMMetric`: one judge step; code computes the weighted score\nChecks that differ per case, with no good reference                         | Generated checklist     | `types.StepBasedLLMMetric`: one step writes checks from the conversation, one grades them\nLong or multi-turn traces, per-turn verdicts, or several judging stages     | Step-based, multi-stage | `types.StepBasedLLMMetric`\nOne sub-agent of a multi-agent system                                       | Step-based, one agent   | `types.StepBasedLLMMetric` with a code preprocessor that keeps that agent's events\n\nStart each metric from its template in\n[Metric templates](references\u002Fmetric_templates.md); each template says what it\ndoesn't catch. For weighted rubrics, propose the rubrics, a weight for each, the\nmust-pass rubrics and a pass threshold, and keep the weights in the parser code\nso the judge grades each rubric on its own. Write checks as described in\n[Writing LLM checks](references\u002Fwriting_checks.md). For multi-turn\nconversations, decide per criterion whether to grade each turn or the whole\nconversation; see [Multi-turn conversations](references\u002Fmulti_turn.md).\n\n### 5. Grade and show the results\n\nGrade the cases through the evaluation service with `scripts\u002Frun_metrics.py`,\nthen build a review page with `scripts\u002Fbuild_review_page.py`. Pass each metric's\npass threshold as `--threshold`, so PASS and FAIL on the page match the metric.\n\nOptional: if the user has an existing autorater, or wants to compare with a\nbuilt-in metric such as `MULTI_TURN_TASK_SUCCESS`, grade with it too and show it\nas a baseline. It adds calls, and a built-in metric grades overall task success\nrather than the user's criteria.\n\n### 6. Optional: align the metrics with the user's ratings\n\nAsk whether the user wants to mark the grades and have you improve the metrics,\nat about a minute per case. Skip to step 7 if not.\n\n1.  Set aside about 20% of the cases as held back. Build the review page without\n    them.\n2.  The user marks each verdict Agree or Disagree on the review page, with a\n    reason when they disagree, and gives you `labels.json`. If the user already\n    has human ratings, use them and ask only about the cases where a metric\n    disagrees.\n3.  Compute agreement per metric and per rubric with `scripts\u002Fagreement.py`.\n    Measure judge noise before trusting a change; see\n    [Noise and agreement](references\u002Fnoise_and_agreement.md).\n4.  Revise the metrics yourself (rubric wording, weights, must-pass rubrics,\n    threshold, reference, code), save each round as a new version, grade again,\n    and show the user what changed and the new agreement. When a label\n    contradicts the trajectory, quote the lines and ask the user.\n5.  Stop when agreement stops improving by more than the noise, or when the user\n    is satisfied. Grade the held-back cases once. Don't revise on them; if a\n    held-back case breaks, report overfitting and suggest more labeled cases.\n\n### 7. Review and register (Approval)\n\nRegistering the metrics is the goal of this skill. Add to the review page, for\neach metric: its criterion, the final definition (rubrics, weights, must-pass\nrubrics, threshold, judge model, sample count), its grades, agreement on labeled\nand held-back cases if step 6 ran, and open questions. After approval, register\neach metric with `create_evaluation_metric`; it creates a shared resource.\n\nHand over an output folder with:\n\n-   `criteria.md`: the approved criteria, each listing the metrics that grade\n    it;\n-   `metrics\u002F`: the final definition of every metric, as the file the evaluation\n    service ran (the metrics file passed to `scripts\u002Frun_metrics.py`), with its\n    registered name;\n-   `metrics.md`: one entry per metric with its criterion, form, rubrics,\n    weights, must-pass rubrics, threshold, judge model and sample count, and a\n    code snippet that runs it by registered name;\n-   `labels.json` and earlier versions, if step 6 ran;\n-   `report.md`: the numbers and open questions.\n\nTo run the metrics on a full dataset, point to `agent-platform-eval-flywheel`.\n\n## Review pages\n\nBuild one self-contained HTML page per round with\n`scripts\u002Fbuild_review_page.py`, and give the user its path (or serve it where\nthe user's browser can reach it). For each case it shows the user's note or\nrating, each metric's PASS\u002FFAIL, score, explanation and rubric verdicts, the\nbaseline if one ran, and the trajectory with tool results collapsed. The user\nmarks Agree or Disagree on each verdict with a reason; marks are saved in the\nbrowser, and \"Download labels.json\" exports them in the format\n`scripts\u002Fagreement.py` reads. Never ask the user to type labels in chat.\n\n## Rules\n\n-   All grading goes through the evaluation service. Labels come only from the\n    user: don't grade trajectories or draft labels with your own model calls or\n    subagents, because tuning a metric to another model's labels only teaches\n    one model to agree with another.\n-   Grade only the cases in the folder while building; the full dataset is for\n    after registration.\n-   Every number you report comes from a result file. Report a range when runs\n    differ.\n-   Keep proprietary trajectories and outputs, including review pages, where the\n    user said. Never paste them into shared docs, bugs or chats.\n-   Read the installed SDK source before using a name; versions differ.\n\n## Cleanup\n\nRegistered metrics stay in the project until deleted. When the user wants to\nremove a metric they no longer use, or one registered by mistake, confirm the\nname with them, then run\n`client.evals.delete_evaluation_metric(metric_resource_name={name})`. List them\nwith `client.evals.list_evaluation_metrics()`. Local outputs (results, review\npages, labels) stay where the user chose; offer to delete them at the end.\n\n## Scripts\n\nAll have `--help`.\n\n-   `scripts\u002Frender_agent_data.py`: trajectories as readable text.\n-   `scripts\u002Frun_metrics.py`: runs SDK metrics with `client.evals.evaluate()`.\n-   `scripts\u002Fbuild_review_page.py`: the review page and labels export.\n-   `scripts\u002Fagreement.py`: agreement, Cohen's kappa and disagreements, per\n    metric, per rubric or per turn.\n\n## References\n\n-   [Converting trajectories](references\u002Fconverting_trajectories.md)\n-   [Metric templates](references\u002Fmetric_templates.md)\n-   [Writing LLM checks](references\u002Fwriting_checks.md)\n-   [Multi-turn conversations](references\u002Fmulti_turn.md)\n-   [Noise and agreement](references\u002Fnoise_and_agreement.md)\n-   [SDK gotchas](references\u002Fsdk_gotchas.md)\n",{"name":5,"metadata":24,"description":7},{"version":25,"category":26},"1.0.0","AiAndMachineLearning","main","67c4dbc8ff6313ef117d938e53bb21315fc281ae","skills\u002Fcloud\u002Fagent-platform-autorater-customization",2014,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills","https:\u002F\u002Fgithub.com\u002Fgoogle","https:\u002F\u002Fopensource.google\u002F","Google ❤️ Open Source","2026-11-14 23:31:00",{"source":8,"skillName":5,"commands":37,"prompt":47,"zipAvailable":48},[38,41,44],{"tool":39,"command":40},"npx","npx skills add google\u002Fskills --skill agent-platform-autorater-customization",{"tool":42,"command":43},"bunx","bunx skills add google\u002Fskills --skill agent-platform-autorater-customization",{"tool":45,"command":46},"pnpm","pnpm dlx skills add google\u002Fskills --skill agent-platform-autorater-customization","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fagent-platform-autorater-customization\u002FSKILL.md",true,[50,56,60,64,68,72,76,80,86,90,94],{"path":51,"resourceType":52,"resourceTypeDesc":53,"sizeBytes":54,"sourceHtmlUrl":55},"references\u002Fattribution.md","REFERENCE","参考文档",1958,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fagent-platform-autorater-customization\u002Freferences\u002Fattribution.md",{"path":57,"resourceType":52,"resourceTypeDesc":53,"sizeBytes":58,"sourceHtmlUrl":59},"references\u002Fconverting_trajectories.md",1932,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fagent-platform-autorater-customization\u002Freferences\u002Fconverting_trajectories.md",{"path":61,"resourceType":52,"resourceTypeDesc":53,"sizeBytes":62,"sourceHtmlUrl":63},"references\u002Fmetric_templates.md",24531,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fagent-platform-autorater-customization\u002Freferences\u002Fmetric_templates.md",{"path":65,"resourceType":52,"resourceTypeDesc":53,"sizeBytes":66,"sourceHtmlUrl":67},"references\u002Fmulti_turn.md",2095,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fagent-platform-autorater-customization\u002Freferences\u002Fmulti_turn.md",{"path":69,"resourceType":52,"resourceTypeDesc":53,"sizeBytes":70,"sourceHtmlUrl":71},"references\u002Fnoise_and_agreement.md",1272,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fagent-platform-autorater-customization\u002Freferences\u002Fnoise_and_agreement.md",{"path":73,"resourceType":52,"resourceTypeDesc":53,"sizeBytes":74,"sourceHtmlUrl":75},"references\u002Fsdk_gotchas.md",2760,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fagent-platform-autorater-customization\u002Freferences\u002Fsdk_gotchas.md",{"path":77,"resourceType":52,"resourceTypeDesc":53,"sizeBytes":78,"sourceHtmlUrl":79},"references\u002Fwriting_checks.md",1437,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fagent-platform-autorater-customization\u002Freferences\u002Fwriting_checks.md",{"path":81,"resourceType":82,"resourceTypeDesc":83,"sizeBytes":84,"sourceHtmlUrl":85},"scripts\u002Fagreement.py","SCRIPT","脚本",7064,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fagent-platform-autorater-customization\u002Fscripts\u002Fagreement.py",{"path":87,"resourceType":82,"resourceTypeDesc":83,"sizeBytes":88,"sourceHtmlUrl":89},"scripts\u002Fbuild_review_page.py",10493,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fagent-platform-autorater-customization\u002Fscripts\u002Fbuild_review_page.py",{"path":91,"resourceType":82,"resourceTypeDesc":83,"sizeBytes":92,"sourceHtmlUrl":93},"scripts\u002Frender_agent_data.py",3374,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fagent-platform-autorater-customization\u002Fscripts\u002Frender_agent_data.py",{"path":95,"resourceType":82,"resourceTypeDesc":83,"sizeBytes":96,"sourceHtmlUrl":97},"scripts\u002Frun_metrics.py",6328,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fagent-platform-autorater-customization\u002Fscripts\u002Frun_metrics.py",[99,106,113,120,127,134],{"skillNo":100,"name":101,"nameZh":6,"description":102,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":103,"categories":104,"keywords":105,"publishedAt":19},"01M7DQ9K7JF4T7NJZ2K1MQZ44Q","cloud-operator-share-context","Shares Google Cloud operating context the user or their team already have — environments, projects, services and regions, product preferences, best practices, architecture guidance, runbooks — found in agent instruction files, local memory, skills and workflows, and the repository's GCP docs, with the operator agent's memory after the user reviews the list. Use when the user runs this skill, accepts the cloud-operator or cloud-operator-setup skill's offer to share their existing Google Cloud context, or asks to share, import or save that context or a GCP fact for the operator agent. Don't use for ordinary GCP requests, which go to the cloud-operator skill.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fcloud-operator-share-context\u002FSKILL.md",[],[],{"skillNo":107,"name":108,"nameZh":6,"description":109,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":110,"categories":111,"keywords":112,"publishedAt":19},"01M7DQ9ERFGGQCFAWQW0KRKYBY","cloud-operator-setup","Sets up the operator agent in a Google Cloud project via the autocloud MCP server: picks the operator project, creates or checks the user's operator agent, saves the project, and offers to share the user's existing Google Cloud operating context with the operator agent's memory. Use when the user runs this skill, asks to set up the operator or create their operator agent, or chooses setup when the cloud-operator skill asks. Don't use for ordinary GCP requests or installing gcloud.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fcloud-operator-setup\u002FSKILL.md",[],[],{"skillNo":114,"name":115,"nameZh":6,"description":116,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":117,"categories":118,"keywords":119,"publishedAt":19},"01M7B4RHQCM0DK2XQ7S7ZXVN8D","developer-device-platform-device-run","Runs automated, non-interactive jobs on remote Android devices with Developer Device Platform (DDP) Device Run using `gcloud beta device-run sessions`: native Android executables or shell scripts (for example CLI tools, custom runners, test harnesses or benchmarks) and Android instrumentation tests, on one or many devices, then waits for completion and collects per-device outputs and logs. Use when running a binary or test suite on remote devices without an interactive connection. Don't use it for interactive device reservations, adb tunnels or screen streaming (use developer-device-platform-basics), or for local devices.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fdeveloper-device-platform-device-run\u002FSKILL.md",[],[],{"skillNo":121,"name":122,"nameZh":6,"description":123,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":124,"categories":125,"keywords":126,"publishedAt":19},"01M7B4R9YXMRW7RZP9Q9FMZJ2P","compute-vm-creation","Guides the planning, configuration, and provisioning of standalone Google Compute Engine virtual machines (VMs) and bare metal (-metal) instances using gcloud CLI commands or direct MCP tool execution. Use ONLY when the user wants to create, configure, or provision a new standalone Compute Engine VM or bare metal instance from conversational intents (hardware sizing, region selection, OS image, networking, security defaults). DO NOT read or use for Google Kubernetes Engine (GKE) Autopilot or Standard clusters, Kubernetes node pools or Pods, container orchestration, managing existing VMs, Managed Instance Groups (MIGs), or Instance Templates.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fcompute-vm-creation\u002FSKILL.md",[],[],{"skillNo":128,"name":129,"nameZh":6,"description":130,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":131,"categories":132,"keywords":133,"publishedAt":19},"01M78JP22K73KK96JH0JNDT5C4","iap-agent-platform-troubleshooting","Diagnoses Identity-Aware Proxy (IAP) denials of outbound requests from Gemini Enterprise Agent Platform agents that run on Agent Runtime and send traffic through Agent Gateway. Use when an agent request fails with \"Egress request is not authorized\", when Cloud Audit Logs show an IAP egressViaIAP decision with granted false, when IAP audit entries are missing while debugging agent egress, or when Access policies, IAP allow policies, policy bindings, or Agent Registry entries might block agent egress. Don't use for Gemini model IAM permission errors (such as aiplatform.endpoints.predict), Agent Platform API errors, or Agent Runtime deployment or 500 errors (use agent-platform-troubleshooting instead). Don't use for setting up, creating, or connecting an Agent Gateway, or for IAP web apps, IAP TCP forwarding, or SSH.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fiap-agent-platform-troubleshooting\u002FSKILL.md",[],[],{"skillNo":135,"name":136,"nameZh":6,"description":137,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":138,"categories":139,"keywords":140,"publishedAt":19},"01M78JJJ5CSDTKX3HRH34K37Z8","bigquery-migration-validator","Validates that tables migrated to BigQuery match their source (Teradata, Snowflake, Oracle, Redshift, Databricks, PostgreSQL, SQL Server, MySQL, MariaDB, Hive, Impala, Vertica) with the BigQuery Migration Service (BQMS) validation tool: schema and data type comparison, column aggregate comparison, and row-level data comparison. Use when generating or editing a validation config, running a validation, choosing sampling or partitioning, fixing connection or driver errors, or investigating validation results in BigQuery and Cloud Storage to decide which mismatches to fix. Don't use for SQL translation (use bigquery-migration-sql-translator), extracting source database catalog metadata or query logs, generating SQL scripts (such as COPY INTO) or exporting or unloading source tables to Parquet on Cloud Storage, loading staged files into BigQuery (such as bq load), or general BigQuery query failures and tuning (use bigquery-troubleshooting or bigquery-optimization).","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fbigquery-migration-validator\u002FSKILL.md",[],[]]