[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M698Y88W0NXEG6APK2Y8TMWV":3,"skill-same:01M698Y88W0NXEG6APK2Y8TMWV":134},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":23,"publishedAt":47,"linkOnly":48,"markdownContent":49,"markdownBody":50,"frontMatter":51,"refName":55,"commitSha":56,"skillDirectory":57,"forksCount":58,"repositoryHtmlUrl":59,"repositoryHomepageUrl":6,"ownerHtmlUrl":60,"ownerBlogUrl":61,"ownerDescription":62,"updatedAt":63,"install":64,"resources":77},"01M698Y88W0NXEG6APK2Y8TMWV","agent-platform-eval-flywheel",null,"Measures and improves the quality of AI models and agents on Google Cloud using the Eval Quality Flywheel methodology. Use when generating synthetic user scenarios, evaluating an agent or model, building an eval dataset, picking or writing evaluation metrics, analyzing failures, comparing results before and after a fix, or when guidance is needed on Agent Platform eval methodology — including dataset schema, LLM-as-judge scoring, and common failure causes. For fine-tuning, use agent-platform-tuning. For general production deployment, use agent-platform-deploy.","google\u002Fskills","google","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F1342004?v=4","01M698VYEFDM0VJBZDYYJ2T83S","Google","01M698WR0H76ENHE587T9EJT89",24122,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fagent-platform-eval-flywheel\u002FSKILL.md",[18],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,9],"agent platform eval flywheel","agent platform","flywheel","eval","platform","scoring","LLM-as-Judge","when","agent-platform","judge","ai models","synthetic","dataset","as","writing","fix","common","model","tuning","analyzing","Evaluation","metrics","google cloud","2026-11-10 10:12:30",false,"---\nname: agent-platform-eval-flywheel\nmetadata:\n version: \"1.0.4\"\n category: AiAndMachineLearning\ndescription: >-\n Measures and improves the quality of AI models and agents on Google Cloud\n using the Eval Quality Flywheel methodology. Use when generating synthetic\n user scenarios, evaluating an agent or model, building an eval dataset,\n picking or writing evaluation metrics, analyzing failures, comparing results\n before and after a fix, or when guidance is needed on Agent Platform eval\n methodology — including dataset schema, LLM-as-judge scoring, and common\n failure causes. For fine-tuning, use agent-platform-tuning. For general\n production deployment, use agent-platform-deploy.\n---\n\n# Agent Platform Eval Flywheel Skill\n\nHelp users evaluate and iteratively improve GenAI models and agents using the\nAgent Platform GenAI Evaluation SDK (`google.genai` \u002F `agentplatform`).\n\n## Attribution\n\n**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you MUST\nread [Usage Attribution](references\u002Fattribution.md). For commands executed\ndirectly in the terminal (never in generated scripts, Terraform, or user code),\nprefix `gcloud` with `CLOUDSDK_METRICS_REQUEST_ATTRIBUTION` and append the\nattribution suffix to `User-Agent` headers:\n\n```text\nCLOUDSDK_METRICS_REQUEST_ATTRIBUTION=\"agent-skills\u002F1.0 (skill:agent-platform-eval-flywheel)\"\nUser-Agent: \u003Cexisting_user_agent> agent-skills\u002F1.0 (skill:agent-platform-eval-flywheel)\n```\n\n## When to use this skill\n\n- Evaluating GenAI agents or models with the Agent Platform GenAI Evaluation\n SDK (`client.evals.evaluate()`).\n- Creating evaluation datasets from session traces, pandas DataFrames, or\n synthetic generation.\n- Selecting, configuring, or writing custom evaluation metrics.\n- Analyzing rubric verdicts, loss patterns, and clustering failures.\n- Suggesting concrete code\u002Fprompt improvements based on eval results.\n- Evaluating a model served on an Agent Platform **endpoint** (BYOM) or a\n **Model-as-a-Service (MaaS)** model by ID — including deploying the model\n first if needed. For this case, follow\n [references\u002Fdeployment.md](references\u002Fdeployment.md) and use the\n `endpoint_evaluation.py` \u002F `maas_evaluation.py` scripts.\n\n## Safety & Confirmation Tiers (CRITICAL)\n\nBefore executing any commands or scripts on behalf of the user, you MUST adhere\nto the following safety tiers based on the action requested:\n\n1. **Tier R**: Read-only (`inspect_results.py`, `compare_results.py`,\n `validate_dataset.py`, `parse_adk_traces.py`, `render_html_report.py`)\n * **Rule**: No confirmation needed. You may execute these helper scripts\n immediately to inspect data, validate schemas, parse traces, or compare\n evaluation results.\n2. **Tier M: Evaluation & Generation Runs (`client.evals.run_inference`,\n `client.evals.evaluate`, `client.evals.generate_conversation_scenarios`,\n `client.evals.generate_loss_clusters`)**\n * **Rule**: For predictable user approval over dataset reads, result\n writes (`dest`), and remote compute\u002FAPI usage, every call to\n `client.evals.evaluate()`, `client.evals.run_inference()`,\n `client.evals.generate_conversation_scenarios()`, or\n `client.evals.generate_loss_clusters()` — whether using remote metrics\n (LLM\u002Frubric metrics or sandboxed `types.CodeExecutionMetric`) or local\n in-process metrics (`types.Metric` with a Python callable) — requires\n presenting a dry-run confirmation preview showing the proposed\n evaluation call and parameters and waiting for explicit 'Yes'\u002F'No' user\n approval (do not quote remote compute charges when only local in-process\n `types.Metric` callables are used without `dest` or model inference).\n * **Confirmation for EVERY evaluation run**: Every evaluation,\n re-evaluation, metric update, parameter change, or synthetic scenario\n generation requires its own dry-run preview and interactive\n confirmation. Never execute a second evaluation, comparison pass, or\n modified evaluation without presenting a new confirmation preview and\n obtaining user approval.\n * **Same-turn restriction**: Do not run the evaluation in the same turn as\n presenting the confirmation prompt. End your turn after asking and wait\n for the user's reply; only execute after explicit 'Yes' \u002F approval.\n Printing a preview and then calling the tool before the user can answer\n does not count as obtaining confirmation.\n * **No Pre-Execution of Evaluation Calls**: NEVER execute\n `client.evals.evaluate()`, `client.evals.run_inference()`,\n `client.evals.generate_conversation_scenarios()`, or run any script\n invoking these SDK methods before user confirmation. In the initial\n turn, you may write the evaluation script to disk, but you MUST present\n the dry-run confirmation preview and obtain explicit user confirmation\n before running `client.evals.evaluate()`.\n * **Immediate Execution Upon Approval**: Once the user explicitly approves\n (e.g., 'Yes', 'Approved', 'Go ahead', 'Proceed'), proceed directly to\n executing the previewed evaluation script via `run_command` and report\n the results. Do not conclude the turn without executing the approved\n action.\n\n## Setup\n\nThe scripts need `vertexai` (from `google-cloud-aiplatform[evaluation]`),\n`google-genai`, `pandas`, and `requests`. Do **not** create a virtual\nenvironment — it starts empty and hides packages the environment already\nprovides, forcing a redundant install. Probe, and install only what is missing:\n\n```bash\npython3 -c \"import vertexai, google.genai, pandas, requests\" \\\n || pip install 'google-cloud-aiplatform[evaluation]>=1.163.0' 'google-genai>=1.0.0'\n```\n\nThe version specifiers must stay quoted: unquoted, bash reads `>=1.154.0` as a\nredirect and silently writes an empty file instead of constraining the install.\n\nNeed `GOOGLE_CLOUD_PROJECT` and `GOOGLE_CLOUD_LOCATION`.\n\n- **Preserve User Project and Location**: Always prioritize the user's\n explicitly provided project and location (e.g. `project='\u003CPROJECT_NUMBER>'`,\n `location='us-central1'`). Never change or override the user's requested\n location to 'global' unless the user explicitly requested 'global'.\n- **Missing Parameters**: If the user's request omits the project or location,\n you MUST pause in your response and ask the user for the missing\n location\u002Fproject before preparing or running the evaluation.\n\n### Correct SDK entrypoints\n\n```python\nimport agentplatform\nclient = agentplatform.Client(project=PROJECT, location=LOCATION)\n\nclient.evals.run_inference(model=..., src=...)\nclient.evals.evaluate(dataset=..., metrics=...)\nclient.evals.generate_conversation_scenarios(...)\n```\n\nTwo imports that look plausible and are not:\n\n- `from agentplatform.types import evals` -- `ModuleNotFoundError`. `types` is\n a module, not a package; use `from agentplatform import types`.\n- `from vertexai.evaluation import PointwiseMetric, EvalTask` -- the\n superseded SDK. Its classes take different arguments (`PointwiseMetric` has\n no `system_instruction`), so code written against it fails with `TypeError`\n rather than an import error. Use `agentplatform` throughout.\n\n## The Quality Flywheel\n\nFive stages, run in order on the first pass, then loop 2 → 5 until quality\ntargets are met.\n\n### Shortcuts that waste time\n\n| Shortcut | Why it fails |\n| ------------------------------------ | ------------------------------------ |\n| \"I'll tune the metric threshold down | Hides real failures. Fix the agent, |\n: so it passes.\" : not the bar. :\n| \"This case is flaky, I'll skip it.\" | Flakiness reveals non-determinism in |\n: : the agent. Fix with `temperature=0` :\n: : or stricter instructions. :\n| \"I just need to fix the eval | If expected outputs keep moving, the |\n: dataset, not the agent.\" : agent has a behavior problem. :\n| \"I can tell from the trace it works | Self-grading doesn't generalize. |\n: — skip Stage 3.\" : Always run `evaluate()` and read :\n: : scores. :\n| \"One iteration is enough.\" | Expect 5–10+ iterations. Stopping |\n: : early leaves regressions on other :\n: : metrics undetected. :\n\n### 1. Prepare Data\n\nProduce an `EvaluationDataset`. There are three input shapes, pick the one that\nmatches the data the user already has:\n\n- **`EvalCase` list (single-turn or multi-turn):**\n\n ```python\n from agentplatform import types\n from google.genai import types as genai_types\n\n # prompt\u002Freference\u002Fresponse values are Content, not str. UserContent and\n # ModelContent wrap a plain string and set the right role.\n dataset = types.EvaluationDataset(eval_cases=[\n types.EvalCase(\n prompt=genai_types.UserContent(\"What is 2+2?\"),\n responses=[types.ResponseCandidate(\n response=genai_types.ModelContent(\"4\"))],\n reference=types.ResponseCandidate(\n response=genai_types.ModelContent(\"4\")),\n ),\n # For multi-turn agent traces, set agent_data instead of prompt\u002Fresponses.\n ])\n ```\n\n Multi-turn agent traces wrap each conversation in `AgentData` →\n `ConversationTurn` → `AgentEvent`. See\n [references\u002Fdataset_schema.md](references\u002Fdataset_schema.md) for the full\n type hierarchy.\n\n- **Pandas DataFrame (tabular sources — CSV, BigQuery, Sheets):**\n\n ```python\n import pandas as pd\n from agentplatform import types\n\n df = pd.DataFrame({\n \"prompt\": [\"What is 2+2?\", \"Capital of France?\"],\n \"response\": [\"4\", \"Paris\"],\n \"reference\": [\"4\", \"Paris\"],\n })\n dataset = types.EvaluationDataset(eval_dataset_df=df)\n ```\n\n Column names must match the fields the chosen metrics expect (see\n [references\u002Fdataset_schema.md](references\u002Fdataset_schema.md) for the\n per-metric requirements table).\n\n- **Cold start (no data at all):** synthesize scenarios server-side with\n `client.evals.generate_conversation_scenarios(agent=..., config=...)` -- the\n parameter is `agent` or `agent_info`, not `agents`, and `config` is\n required. The config class is `types.evals.UserScenarioGenerationConfig`,\n not `types.UserScenarioGenerationConfig`. Set its `user_scenario_count`\n (1-100): it defaults to None, the client accepts that, and the server\n rejects the call with `400 INVALID_ARGUMENT`. `count` is a separate field\n and does not substitute for it. Stage 2 plays the scenarios out.\n * **CRITICAL - Multi-turn user-simulation scenarios must use this API**:\n When the user asks to synthesize multi-turn **user scenarios** to\n simulate customers interacting with an agent (as opposed to writing a\n local single-turn test dataset in the sandbox), never author the\n conversation scenarios yourself in prose or a handwritten file. Calling\n `client.evals.generate_conversation_scenarios` is a server-side Tier M\n API call (using the user's project, location, model, and environment\n data) and requires a confirmation card; local single-turn dataset files\n written in the sandbox without calling Agent Platform APIs are Tier R.\n * **CRITICAL - Underspecified Requests**: When asked to synthesize\n scenarios, if the request omits required parameters (such as `location`,\n `environment_data`, `simulation_instruction`, or `model_name`), do NOT\n assume defaults or guess values. You MUST pause in your first turn and\n explicitly ask the user for the missing information (e.g., \"Please\n provide the missing simulation instructions, environment data, model\n name, and location\"). Only proceed with the dry-run preview after the\n user provides them. `location` has no safe default: neither\n `us-central1` nor a location you assumed on your own for another call is\n a substitute for a location the user has not given.\n * **Friction & Parameter Changes**: When asked to generate synthetic user\n scenarios, if the user modifies requested parameters (such as scenario\n count, model, or instructions) or pushes back, you MUST present a\n revised dry-run confirmation card with the updated parameters and wait\n for explicit user approval before executing generation code via\n `run_command`.\n\n- **Managed Agents (Gemini Agents API):** evaluate agents created with the\n [Managed Agents API](https:\u002F\u002Fdocs.cloud.google.com\u002Fgemini-enterprise-agent-platform\u002Fbuild\u002Fmanaged-agents).\n Use `generate_conversation_scenarios` to create test scenarios from the\n agent's configuration, `run_inference` to execute the agent, and `evaluate`\n to score the traces. These functions now accept managed agents and\n interaction ids as input. You can also evaluate existing interactions\n recorded via the Interactions API using `InteractionsDataSource`. See\n [references\u002Fsdk_patterns.md](references\u002Fsdk_patterns.md) Pattern 8 for the\n full code pattern.\n\nFor ADK session dumps, use `scripts\u002Fparse_adk_traces.py` instead of writing the\nconversion by hand.\n\n### 2. Run Inference\n\nPopulate responses\u002Ftraces on the dataset. **Skip this stage** if traces are\nalready complete (e.g., production logs or replay).\n\n```python\n# Agent eval — pass a callable wrapping the user's ADK Agent\u002FApp.\nclient.evals.run_inference(model=agent_callable, src=dataset)\n\n# Model eval — pass a model ID directly.\nclient.evals.run_inference(model=\"gemini-2.5-flash\", src=dataset)\n\n# Synthesized scenarios — let the simulator drive.\nclient.evals.run_inference(\n model=agent_callable,\n src=dataset,\n user_simulator_config=UserSimulatorConfig(max_turn=10),\n)\n\n# DataFrame also works as src= — no EvalCase wrapping needed.\nclient.evals.run_inference(model=\"gemini-2.5-flash\", src=df)\n\n# Managed Agent — pass an agent resource name.\nAGENT_RESOURCE = f\"projects\u002F{PROJECT_ID}\u002Flocations\u002Fglobal\u002Fagents\u002F{AGENT_ID}\"\nclient.evals.run_inference(\n agent=AGENT_RESOURCE,\n src=scenarios,\n config={\"user_simulator_config\": {\"max_turn\": 3}},\n)\n```\n\n### 3. Grade (always run)\n\n```python\nresult = client.evals.evaluate(dataset=dataset, metrics=[...])\nresult.show() # Interactive HTML report with scores, rubrics, and traces.\n```\n\n**Pick metrics by what you want to measure.** Full catalog in\n[references\u002Fmetric_registry.md](references\u002Fmetric_registry.md).\n\n**If the user names a metric, use it directly.** Every identifier in the tables\nbelow (`general_quality`, `text_quality`, `instruction_following`,\n`hallucination`, `grounding`, `safety`, `multi_turn_*`, `final_response_*`,\n`tool_use_quality`) is a `types.RubricMetric.\u003CUPPERCASE_NAME>` accessor — pass\nit straight into `metrics=[types.RubricMetric.GENERAL_QUALITY, ...]`. Do not\nscaffold a custom `LLMMetric` for a name that appears here, and do not reach for\n`vertexai.evaluation.EvalTask` \u002F `PointwiseMetric` \u002F\n`MetricPromptTemplateExamples` — that SDK is superseded (see Setup).\n\n**`judge_model` format and handling user-named models:**\n\n* **Full resource name required for `judge_model`**: Both `LLMMetric` and\n `RubricMetric` send `judge_model` as `autorater_config.autorater_model` to\n the backend (even though `agentplatform` logs a client-side \"ignored\"\n warning on `RubricMetric`). Short names like `\"gemini-2.5-flash\"` or\n `\"gemini-2.5-pro\"` fail with `400 INVALID_ARGUMENT (\"Invalid autorater model\n resource name\")`. Always pass the full publisher model resource name:\n `f\"projects\u002F{PROJECT_ID}\u002Flocations\u002F{LOCATION}\u002Fpublishers\u002Fgoogle\u002Fmodels\u002F{MODEL_NAME}\"`\n (prefer `gemini-2.5-flash` as autorater; `gemini-2.5-pro` can intermittently\n fail server-side with `\"Rubric results could not be reliably computed\"`).\n* **Datasets without vs. with responses**:\n * If the dataset contains **prompts only** (no `response` column), call\n `run_inference(model=\"\u003Cthe named model>\", src=dataset)` to generate the\n candidate model's responses first, then grade them with `evaluate()`.\n * If the user **already supplies `response`s** in the dataset, **do NOT\n call `run_inference`** — `run_inference` overwrites the existing\n `response` column with the model's generated output and discards the\n user's responses! Instead, grade the user's supplied `response`s\n directly with `evaluate(dataset=dataset, ...)`, passing the named model\n as the full resource path in `judge_model` (or if a predefined\n `RubricMetric` cannot reliably compute rubric results with that\n autorater model, fall back to the default autorater and explain why).\n\n**Agent metrics (multi-turn, adaptive rubrics)** — start here for agent eval.\n\nGoal | Metric\n--------------------------------------------- | -------------------------------\nDid the agent achieve the user's goal? | `multi_turn_task_success`\nWas the reasoning path logical and efficient? | `multi_turn_trajectory_quality`\nTool\u002Ffunction calling quality across turns | `multi_turn_tool_use_quality`\nOverall conversational quality | `multi_turn_general_quality`\nFinal response quality (no reference needed) | `final_response_quality`\nFinal response vs. a golden reference | `final_response_match`\nSingle-turn tool use | `tool_use_quality`\n\n**General quality metrics (single-turn, adaptive rubrics)** — for model eval.\n\nGoal | Metric\n----------------------------------------------------- | -----------------------\nOverall response quality (recommended starting point) | `general_quality`\nLinguistic quality (fluency, coherence, grammar) | `text_quality`\nAdherence to specific constraints \u002F instructions | `instruction_following`\n\n**Static rubric metrics (fixed criteria)** — apply alongside the above.\n\nGoal | Metric\n------------------------------------------------- | ---------------\nCatch hallucinated claims (RAG, factual answers) | `hallucination`\nFactuality \u002F consistency against provided context | `grounding`\nSafety policy compliance | `safety`\n\n**Domain-specific check no built-in covers:** write a custom metric.\n\n- **Predefined:** `types.RubricMetric.\u003CNAME>` — server-side AutoRater, no\n judge model needed.\n- **Custom LLM-as-a-judge:** `types.LLMMetric` with `prompt_template` or\n `types.MetricPromptBuilder` for structured rubrics. Always set\n `judge_model`; it defaults to `None` and every case then fails with `400\n INVALID_ARGUMENT: Error parsing JSON`.\n * **Judge Model Selection**: If the user specifies a judge model (e.g.\n `gemini-2.5-pro`), use it. If the user omits the judge model or states\n they do not have information \u002F preference for one, default to\n `gemini-2.5-flash` as the judge model in the dry-run preview card and\n ask for confirmation to run the evaluation. Do NOT halt or refuse to\n evaluate when the user does not specify a judge model.\n- **Custom code:** `types.CodeExecutionMetric` with a `custom_function` string\n containing `def evaluate(instance: dict)` for remote sandboxed execution; or\n `types.Metric` with `custom_function=\u003Ccallable>` for local execution. The\n `instance` dict's `prompt` \u002F `response` \u002F `reference` are `Content` dicts,\n not strings; read\n [references\u002Fmetric_registry.md](references\u002Fmetric_registry.md) (\"What the\n function receives\") before writing the body, rather than discovering the\n shape by running the evaluation.\n\n**Always persist the result** so Stage 4 and 5 can read it. Save both JSON\n(machine-readable, diffable) and HTML (human-readable, linkable):\n\n```python\nimport datetime\nfrom pathlib import Path\n\nfrom agentplatform._genai import _evals_visualization\n\nout_dir = Path(\"artifacts\u002Fgrade_results\")\nout_dir.mkdir(parents=True, exist_ok=True)\nts = datetime.datetime.now().strftime(\"%Y%m%d_%H%M%S\")\n\n# fallback=str, or a DataFrame-backed dataset raises PydanticSerializationError.\nresult_json = result.model_dump_json(fallback=str)\n(out_dir \u002F f\"results_{ts}.json\").write_text(result_json)\n\nhtml = _evals_visualization.get_evaluation_html(result_json)\n(out_dir \u002F f\"results_{ts}.html\").write_text(str(html))\n```\n\nOr after the fact: `scripts\u002Frender_html_report.py --type evaluation` or\n`scripts\u002Finspect_results.py --save-html`.\n\n### 4. Analyze Failures\n\nRead `summary_metrics` and `eval_case_results` — never fabricate scores. Use\n`scripts\u002Finspect_results.py --failing-only` to filter to failures.\n\nFor each failed metric, see\n[references\u002Ffailure_patterns.md](references\u002Ffailure_patterns.md) for deeper\ndiagnoses. The compact mapping:\n\n| Failing metric | What to change |\n| ----------------------------------- | -------------------------------------- |\n| `multi_turn_task_success` low | The agent isn't completing the goal — |\n: : fix orchestration, missing tool calls, :\n: : premature termination, wrong tool :\n: : selection. :\n| `multi_turn_trajectory_quality` low | The agent reaches the goal |\n: : inefficiently — refine planning :\n: : prompts, remove redundant tool calls. :\n| `multi_turn_tool_use_quality` low | Fix tool descriptions, parameter |\n: : docstrings, or agent instructions for :\n: : tool selection. :\n| `final_response_quality` low | Read auto-generated rubric verdicts; |\n: : refine instructions to address the :\n: : worst-scoring criterion. :\n| `final_response_match` low | The agent's final answer doesn't match |\n: : the golden reference — adjust response :\n: : format or update the reference. :\n| `hallucination` low | Tighten instructions to stay grounded |\n: : in tool output; verify the tool :\n: : actually returned the claimed data. :\n| `grounding` low | The response contradicts the provided |\n: : context — add explicit \"cite only from :\n: : context\" instructions. :\n| `safety` low | Add safety guardrails; review the |\n: : violating content category in the :\n: : rubric verdict. :\n| `general_quality` \u002F `text_quality` | Adjust system instruction wording; the |\n: low : model's default phrasing is too :\n: : generic for the task. :\n| `instruction_following` low | The agent is ignoring constraints — |\n: : restate them in the system instruction :\n: : or use stricter wording. :\n| Agent calls wrong tools | Fix tool descriptions, agent |\n: : instructions, or `tool_config`. :\n| Agent calls extra tools | Add explicit stop instructions, or |\n: : switch to :\n: : `multi_turn_tool_use_quality` to :\n: : surface the extra calls in the rubric. :\n\n**For 10+ failures on the same metric**, use the **Error Analysis service** to\ncluster failures into themes (L1\u002FL2 taxonomy categories) instead of reading\nevery trace:\n\n```python\n# Only supports multi_turn_task_success and multi_turn_tool_use_quality.\n# Service runs in the global region.\nanalysis_client = agentplatform.Client(project=\"PROJECT_ID\", location=\"global\")\nresponse = analysis_client.evals.generate_loss_clusters(\n eval_result=result,\n metric=\"multi_turn_task_success\",\n config={\"max_top_cluster_count\": 5},\n)\nfor r in response.results:\n for cluster in r.clusters:\n print(\n f\"[{cluster.taxonomy_entry.l1_category}\u002F\"\n f\"{cluster.taxonomy_entry.l2_category}] \"\n f\"{cluster.item_count} cases — {cluster.taxonomy_entry.description}\"\n )\n```\n\nSave `response.model_dump_json()` and render with `scripts\u002Frender_html_report.py\n--type loss-analysis`.\n\n### 5. Optimize & Iterate\n\nApply a fix targeting the failing metric. Re-run Stage 3. Compare with\n`scripts\u002Fcompare_results.py --baseline \u003Cprev> --candidate \u003Cnew>` to confirm the\ntarget improved AND no other metric regressed.\n\nTrack progress across iterations:\n\nIteration | Metric A | Metric B | Change made\n--------- | -------- | -------- | ----------------------\nBaseline | 0.62 | 0.55 | —\nv2 | 0.78 | 0.68 | Added grounding prompt\nv3 | 0.81 | 0.72 | Fixed tool selection\n\nExpect 5–10+ iterations per failing case. Only after a case passes should you\nexpand coverage with more eval cases.\n\n## Proving your work\n\nNever claim eval results you didn't read from an actual `result` object.\n\n- After running eval, print the `summary_metrics` table\n (`scripts\u002Finspect_results.py`).\n- After a fix, show before\u002Fafter via `scripts\u002Fcompare_results.py`.\n- Before declaring success, confirm ALL cases pass — not just the one you were\n working on.\n\nIf you can't produce the evidence (SDK call failed, result truncated, metric\nunsupported), say so explicitly. Don't paper over gaps.\n\n## Rules of Engagement\n\n1. **Always Plan First:** Before writing a script, output a `\u003Cplan>` block\n detailing the steps you are about to take.\n2. **Step-by-Step Execution:** Prepare the data and evaluation script, present\n the dry-run confirmation card with full parameters, wait for user approval,\n execute only after explicit confirmation, then inspect and analyze results.\n Do NOT run evaluation calls before user confirmation.\n3. **Standard Python:** Use standard Python imports (`import agentplatform`,\n `from google.genai import types`). Don't use internal import paths.\n4. **Verify Before Guessing:** When unsure about SDK types or metrics, check\n the SDK source code rather than guessing or hallucinating.\n5. **Never End a Turn Silently:** Every turn must end with a non-empty,\n informative text reply to the user summarizing the actions taken or\n presenting the next steps. Returning nothing reads as a failure no matter\n what the tools did.\n\n## SDK Quick Reference\n\n```python\nimport agentplatform\nfrom agentplatform import types\nfrom google.genai import types as genai_types\nimport pandas as pd\n\n# Initialize client\nclient = agentplatform.Client(project=\"PROJECT_ID\", location=\"LOCATION\")\n\n# --- SINGLE-TURN EVAL (pandas DataFrame) -- RECOMMENDED ---\n# The converter wraps plain strings for you.\ndf = pd.DataFrame({\n \"prompt\": [\"Q1\", \"Q2\"],\n \"response\": [\"A1\", \"A2\"],\n})\ndataset = types.EvaluationDataset(eval_dataset_df=df)\n\n# --- SINGLE-TURN EVAL (direct EvalCase) ---\n# Verbose and easy to get wrong; see references\u002Fdataset_schema.md for the\n# exact types before using this form.\ndataset = types.EvaluationDataset(eval_cases=[\n types.EvalCase(\n prompt=genai_types.UserContent(\"Query here\"),\n responses=[types.ResponseCandidate(\n response=genai_types.ModelContent(\"Model response here\"))],\n reference=types.ResponseCandidate(\n response=genai_types.ModelContent(\"Ground truth here\")),\n ),\n])\n\n# --- MULTI-TURN AGENT EVAL ---\nagent_data = types.evals.AgentData(\n agents={\"my_agent\": types.evals.AgentConfig(\n agent_id=\"my_agent\", instruction=\"You are helpful.\")},\n turns=[types.evals.ConversationTurn(turn_index=0, events=[\n types.evals.AgentEvent(author=\"user\",\n content=genai_types.Content(role=\"user\",\n parts=[genai_types.Part(text=\"Hello\")])),\n types.evals.AgentEvent(author=\"my_agent\",\n content=genai_types.Content(role=\"model\",\n parts=[genai_types.Part(text=\"Hi! How can I help?\")])),\n ])],\n)\ndataset = types.EvaluationDataset(\n eval_cases=[types.EvalCase(agent_data=agent_data)])\n\n# --- METRICS ---\npredefined = types.RubricMetric.MULTI_TURN_TRAJECTORY_QUALITY\ncustom_llm = types.LLMMetric(name=\"tone\",\n prompt_template=\"Is this polite? Response: {response}\")\ncustom_code = types.CodeExecutionMetric(name=\"check\",\n custom_function='def evaluate(instance): return {\"score\": 1.0}')\n\n# --- EVALUATE ---\nresult = client.evals.evaluate(dataset=dataset, metrics=[predefined])\n\n# --- RESULTS ---\nfor s in result.summary_metrics:\n print(f\"{s.metric_name}: mean={s.mean_score}, pass_rate={s.pass_rate}\")\nfor case in result.eval_case_results:\n for cand in case.response_candidate_results:\n for name, r in cand.metric_results.items():\n print(f\" {name}: score={r.score}, explanation={r.explanation}\")\n```\n\nSee [references\u002Fsdk_patterns.md](references\u002Fsdk_patterns.md) for advanced\npatterns: synthetic data generation, pairwise comparison, `MetricPromptBuilder`,\nmulti-agent evaluation.\n\n## Bundled scripts\n\nScript | When to use\n------------------------ | -----------\n`validate_dataset.py` | Before Stage 3 — catch malformed `EvaluationDataset` JSON.\n`parse_adk_traces.py` | Stage 1 — convert ADK session dumps to the canonical dataset shape.\n`inspect_results.py` | Stages 3\u002F4 — render summary + per-case scores. `--save-html` for a browsable report.\n`compare_results.py` | Stage 5 — diff baseline vs. candidate, detect regressions.\n`render_html_report.py` | Render HTML from a saved result JSON or loss-clusters JSON.\n`endpoint_evaluation.py` | Stages 2\u002F3 against a deployed Agent Platform endpoint (BYOM). See [references\u002Fdeployment.md](references\u002Fdeployment.md).\n`maas_evaluation.py` | Stages 2\u002F3 against a Model-as-a-Service model by ID. See [references\u002Fdeployment.md](references\u002Fdeployment.md).\n","# Agent Platform Eval Flywheel Skill\n\nHelp users evaluate and iteratively improve GenAI models and agents using the\nAgent Platform GenAI Evaluation SDK (`google.genai` \u002F `agentplatform`).\n\n## Attribution\n\n**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you MUST\nread [Usage Attribution](references\u002Fattribution.md). For commands executed\ndirectly in the terminal (never in generated scripts, Terraform, or user code),\nprefix `gcloud` with `CLOUDSDK_METRICS_REQUEST_ATTRIBUTION` and append the\nattribution suffix to `User-Agent` headers:\n\n```text\nCLOUDSDK_METRICS_REQUEST_ATTRIBUTION=\"agent-skills\u002F1.0 (skill:agent-platform-eval-flywheel)\"\nUser-Agent: \u003Cexisting_user_agent> agent-skills\u002F1.0 (skill:agent-platform-eval-flywheel)\n```\n\n## When to use this skill\n\n- Evaluating GenAI agents or models with the Agent Platform GenAI Evaluation\n SDK (`client.evals.evaluate()`).\n- Creating evaluation datasets from session traces, pandas DataFrames, or\n synthetic generation.\n- Selecting, configuring, or writing custom evaluation metrics.\n- Analyzing rubric verdicts, loss patterns, and clustering failures.\n- Suggesting concrete code\u002Fprompt improvements based on eval results.\n- Evaluating a model served on an Agent Platform **endpoint** (BYOM) or a\n **Model-as-a-Service (MaaS)** model by ID — including deploying the model\n first if needed. For this case, follow\n [references\u002Fdeployment.md](references\u002Fdeployment.md) and use the\n `endpoint_evaluation.py` \u002F `maas_evaluation.py` scripts.\n\n## Safety & Confirmation Tiers (CRITICAL)\n\nBefore executing any commands or scripts on behalf of the user, you MUST adhere\nto the following safety tiers based on the action requested:\n\n1. **Tier R**: Read-only (`inspect_results.py`, `compare_results.py`,\n `validate_dataset.py`, `parse_adk_traces.py`, `render_html_report.py`)\n * **Rule**: No confirmation needed. You may execute these helper scripts\n immediately to inspect data, validate schemas, parse traces, or compare\n evaluation results.\n2. **Tier M: Evaluation & Generation Runs (`client.evals.run_inference`,\n `client.evals.evaluate`, `client.evals.generate_conversation_scenarios`,\n `client.evals.generate_loss_clusters`)**\n * **Rule**: For predictable user approval over dataset reads, result\n writes (`dest`), and remote compute\u002FAPI usage, every call to\n `client.evals.evaluate()`, `client.evals.run_inference()`,\n `client.evals.generate_conversation_scenarios()`, or\n `client.evals.generate_loss_clusters()` — whether using remote metrics\n (LLM\u002Frubric metrics or sandboxed `types.CodeExecutionMetric`) or local\n in-process metrics (`types.Metric` with a Python callable) — requires\n presenting a dry-run confirmation preview showing the proposed\n evaluation call and parameters and waiting for explicit 'Yes'\u002F'No' user\n approval (do not quote remote compute charges when only local in-process\n `types.Metric` callables are used without `dest` or model inference).\n * **Confirmation for EVERY evaluation run**: Every evaluation,\n re-evaluation, metric update, parameter change, or synthetic scenario\n generation requires its own dry-run preview and interactive\n confirmation. Never execute a second evaluation, comparison pass, or\n modified evaluation without presenting a new confirmation preview and\n obtaining user approval.\n * **Same-turn restriction**: Do not run the evaluation in the same turn as\n presenting the confirmation prompt. End your turn after asking and wait\n for the user's reply; only execute after explicit 'Yes' \u002F approval.\n Printing a preview and then calling the tool before the user can answer\n does not count as obtaining confirmation.\n * **No Pre-Execution of Evaluation Calls**: NEVER execute\n `client.evals.evaluate()`, `client.evals.run_inference()`,\n `client.evals.generate_conversation_scenarios()`, or run any script\n invoking these SDK methods before user confirmation. In the initial\n turn, you may write the evaluation script to disk, but you MUST present\n the dry-run confirmation preview and obtain explicit user confirmation\n before running `client.evals.evaluate()`.\n * **Immediate Execution Upon Approval**: Once the user explicitly approves\n (e.g., 'Yes', 'Approved', 'Go ahead', 'Proceed'), proceed directly to\n executing the previewed evaluation script via `run_command` and report\n the results. Do not conclude the turn without executing the approved\n action.\n\n## Setup\n\nThe scripts need `vertexai` (from `google-cloud-aiplatform[evaluation]`),\n`google-genai`, `pandas`, and `requests`. Do **not** create a virtual\nenvironment — it starts empty and hides packages the environment already\nprovides, forcing a redundant install. Probe, and install only what is missing:\n\n```bash\npython3 -c \"import vertexai, google.genai, pandas, requests\" \\\n || pip install 'google-cloud-aiplatform[evaluation]>=1.163.0' 'google-genai>=1.0.0'\n```\n\nThe version specifiers must stay quoted: unquoted, bash reads `>=1.154.0` as a\nredirect and silently writes an empty file instead of constraining the install.\n\nNeed `GOOGLE_CLOUD_PROJECT` and `GOOGLE_CLOUD_LOCATION`.\n\n- **Preserve User Project and Location**: Always prioritize the user's\n explicitly provided project and location (e.g. `project='\u003CPROJECT_NUMBER>'`,\n `location='us-central1'`). Never change or override the user's requested\n location to 'global' unless the user explicitly requested 'global'.\n- **Missing Parameters**: If the user's request omits the project or location,\n you MUST pause in your response and ask the user for the missing\n location\u002Fproject before preparing or running the evaluation.\n\n### Correct SDK entrypoints\n\n```python\nimport agentplatform\nclient = agentplatform.Client(project=PROJECT, location=LOCATION)\n\nclient.evals.run_inference(model=..., src=...)\nclient.evals.evaluate(dataset=..., metrics=...)\nclient.evals.generate_conversation_scenarios(...)\n```\n\nTwo imports that look plausible and are not:\n\n- `from agentplatform.types import evals` -- `ModuleNotFoundError`. `types` is\n a module, not a package; use `from agentplatform import types`.\n- `from vertexai.evaluation import PointwiseMetric, EvalTask` -- the\n superseded SDK. Its classes take different arguments (`PointwiseMetric` has\n no `system_instruction`), so code written against it fails with `TypeError`\n rather than an import error. Use `agentplatform` throughout.\n\n## The Quality Flywheel\n\nFive stages, run in order on the first pass, then loop 2 → 5 until quality\ntargets are met.\n\n### Shortcuts that waste time\n\n| Shortcut | Why it fails |\n| ------------------------------------ | ------------------------------------ |\n| \"I'll tune the metric threshold down | Hides real failures. Fix the agent, |\n: so it passes.\" : not the bar. :\n| \"This case is flaky, I'll skip it.\" | Flakiness reveals non-determinism in |\n: : the agent. Fix with `temperature=0` :\n: : or stricter instructions. :\n| \"I just need to fix the eval | If expected outputs keep moving, the |\n: dataset, not the agent.\" : agent has a behavior problem. :\n| \"I can tell from the trace it works | Self-grading doesn't generalize. |\n: — skip Stage 3.\" : Always run `evaluate()` and read :\n: : scores. :\n| \"One iteration is enough.\" | Expect 5–10+ iterations. Stopping |\n: : early leaves regressions on other :\n: : metrics undetected. :\n\n### 1. Prepare Data\n\nProduce an `EvaluationDataset`. There are three input shapes, pick the one that\nmatches the data the user already has:\n\n- **`EvalCase` list (single-turn or multi-turn):**\n\n ```python\n from agentplatform import types\n from google.genai import types as genai_types\n\n # prompt\u002Freference\u002Fresponse values are Content, not str. UserContent and\n # ModelContent wrap a plain string and set the right role.\n dataset = types.EvaluationDataset(eval_cases=[\n types.EvalCase(\n prompt=genai_types.UserContent(\"What is 2+2?\"),\n responses=[types.ResponseCandidate(\n response=genai_types.ModelContent(\"4\"))],\n reference=types.ResponseCandidate(\n response=genai_types.ModelContent(\"4\")),\n ),\n # For multi-turn agent traces, set agent_data instead of prompt\u002Fresponses.\n ])\n ```\n\n Multi-turn agent traces wrap each conversation in `AgentData` →\n `ConversationTurn` → `AgentEvent`. See\n [references\u002Fdataset_schema.md](references\u002Fdataset_schema.md) for the full\n type hierarchy.\n\n- **Pandas DataFrame (tabular sources — CSV, BigQuery, Sheets):**\n\n ```python\n import pandas as pd\n from agentplatform import types\n\n df = pd.DataFrame({\n \"prompt\": [\"What is 2+2?\", \"Capital of France?\"],\n \"response\": [\"4\", \"Paris\"],\n \"reference\": [\"4\", \"Paris\"],\n })\n dataset = types.EvaluationDataset(eval_dataset_df=df)\n ```\n\n Column names must match the fields the chosen metrics expect (see\n [references\u002Fdataset_schema.md](references\u002Fdataset_schema.md) for the\n per-metric requirements table).\n\n- **Cold start (no data at all):** synthesize scenarios server-side with\n `client.evals.generate_conversation_scenarios(agent=..., config=...)` -- the\n parameter is `agent` or `agent_info`, not `agents`, and `config` is\n required. The config class is `types.evals.UserScenarioGenerationConfig`,\n not `types.UserScenarioGenerationConfig`. Set its `user_scenario_count`\n (1-100): it defaults to None, the client accepts that, and the server\n rejects the call with `400 INVALID_ARGUMENT`. `count` is a separate field\n and does not substitute for it. Stage 2 plays the scenarios out.\n * **CRITICAL - Multi-turn user-simulation scenarios must use this API**:\n When the user asks to synthesize multi-turn **user scenarios** to\n simulate customers interacting with an agent (as opposed to writing a\n local single-turn test dataset in the sandbox), never author the\n conversation scenarios yourself in prose or a handwritten file. Calling\n `client.evals.generate_conversation_scenarios` is a server-side Tier M\n API call (using the user's project, location, model, and environment\n data) and requires a confirmation card; local single-turn dataset files\n written in the sandbox without calling Agent Platform APIs are Tier R.\n * **CRITICAL - Underspecified Requests**: When asked to synthesize\n scenarios, if the request omits required parameters (such as `location`,\n `environment_data`, `simulation_instruction`, or `model_name`), do NOT\n assume defaults or guess values. You MUST pause in your first turn and\n explicitly ask the user for the missing information (e.g., \"Please\n provide the missing simulation instructions, environment data, model\n name, and location\"). Only proceed with the dry-run preview after the\n user provides them. `location` has no safe default: neither\n `us-central1` nor a location you assumed on your own for another call is\n a substitute for a location the user has not given.\n * **Friction & Parameter Changes**: When asked to generate synthetic user\n scenarios, if the user modifies requested parameters (such as scenario\n count, model, or instructions) or pushes back, you MUST present a\n revised dry-run confirmation card with the updated parameters and wait\n for explicit user approval before executing generation code via\n `run_command`.\n\n- **Managed Agents (Gemini Agents API):** evaluate agents created with the\n [Managed Agents API](https:\u002F\u002Fdocs.cloud.google.com\u002Fgemini-enterprise-agent-platform\u002Fbuild\u002Fmanaged-agents).\n Use `generate_conversation_scenarios` to create test scenarios from the\n agent's configuration, `run_inference` to execute the agent, and `evaluate`\n to score the traces. These functions now accept managed agents and\n interaction ids as input. You can also evaluate existing interactions\n recorded via the Interactions API using `InteractionsDataSource`. See\n [references\u002Fsdk_patterns.md](references\u002Fsdk_patterns.md) Pattern 8 for the\n full code pattern.\n\nFor ADK session dumps, use `scripts\u002Fparse_adk_traces.py` instead of writing the\nconversion by hand.\n\n### 2. Run Inference\n\nPopulate responses\u002Ftraces on the dataset. **Skip this stage** if traces are\nalready complete (e.g., production logs or replay).\n\n```python\n# Agent eval — pass a callable wrapping the user's ADK Agent\u002FApp.\nclient.evals.run_inference(model=agent_callable, src=dataset)\n\n# Model eval — pass a model ID directly.\nclient.evals.run_inference(model=\"gemini-2.5-flash\", src=dataset)\n\n# Synthesized scenarios — let the simulator drive.\nclient.evals.run_inference(\n model=agent_callable,\n src=dataset,\n user_simulator_config=UserSimulatorConfig(max_turn=10),\n)\n\n# DataFrame also works as src= — no EvalCase wrapping needed.\nclient.evals.run_inference(model=\"gemini-2.5-flash\", src=df)\n\n# Managed Agent — pass an agent resource name.\nAGENT_RESOURCE = f\"projects\u002F{PROJECT_ID}\u002Flocations\u002Fglobal\u002Fagents\u002F{AGENT_ID}\"\nclient.evals.run_inference(\n agent=AGENT_RESOURCE,\n src=scenarios,\n config={\"user_simulator_config\": {\"max_turn\": 3}},\n)\n```\n\n### 3. Grade (always run)\n\n```python\nresult = client.evals.evaluate(dataset=dataset, metrics=[...])\nresult.show() # Interactive HTML report with scores, rubrics, and traces.\n```\n\n**Pick metrics by what you want to measure.** Full catalog in\n[references\u002Fmetric_registry.md](references\u002Fmetric_registry.md).\n\n**If the user names a metric, use it directly.** Every identifier in the tables\nbelow (`general_quality`, `text_quality`, `instruction_following`,\n`hallucination`, `grounding`, `safety`, `multi_turn_*`, `final_response_*`,\n`tool_use_quality`) is a `types.RubricMetric.\u003CUPPERCASE_NAME>` accessor — pass\nit straight into `metrics=[types.RubricMetric.GENERAL_QUALITY, ...]`. Do not\nscaffold a custom `LLMMetric` for a name that appears here, and do not reach for\n`vertexai.evaluation.EvalTask` \u002F `PointwiseMetric` \u002F\n`MetricPromptTemplateExamples` — that SDK is superseded (see Setup).\n\n**`judge_model` format and handling user-named models:**\n\n* **Full resource name required for `judge_model`**: Both `LLMMetric` and\n `RubricMetric` send `judge_model` as `autorater_config.autorater_model` to\n the backend (even though `agentplatform` logs a client-side \"ignored\"\n warning on `RubricMetric`). Short names like `\"gemini-2.5-flash\"` or\n `\"gemini-2.5-pro\"` fail with `400 INVALID_ARGUMENT (\"Invalid autorater model\n resource name\")`. Always pass the full publisher model resource name:\n `f\"projects\u002F{PROJECT_ID}\u002Flocations\u002F{LOCATION}\u002Fpublishers\u002Fgoogle\u002Fmodels\u002F{MODEL_NAME}\"`\n (prefer `gemini-2.5-flash` as autorater; `gemini-2.5-pro` can intermittently\n fail server-side with `\"Rubric results could not be reliably computed\"`).\n* **Datasets without vs. with responses**:\n * If the dataset contains **prompts only** (no `response` column), call\n `run_inference(model=\"\u003Cthe named model>\", src=dataset)` to generate the\n candidate model's responses first, then grade them with `evaluate()`.\n * If the user **already supplies `response`s** in the dataset, **do NOT\n call `run_inference`** — `run_inference` overwrites the existing\n `response` column with the model's generated output and discards the\n user's responses! Instead, grade the user's supplied `response`s\n directly with `evaluate(dataset=dataset, ...)`, passing the named model\n as the full resource path in `judge_model` (or if a predefined\n `RubricMetric` cannot reliably compute rubric results with that\n autorater model, fall back to the default autorater and explain why).\n\n**Agent metrics (multi-turn, adaptive rubrics)** — start here for agent eval.\n\nGoal | Metric\n--------------------------------------------- | -------------------------------\nDid the agent achieve the user's goal? | `multi_turn_task_success`\nWas the reasoning path logical and efficient? | `multi_turn_trajectory_quality`\nTool\u002Ffunction calling quality across turns | `multi_turn_tool_use_quality`\nOverall conversational quality | `multi_turn_general_quality`\nFinal response quality (no reference needed) | `final_response_quality`\nFinal response vs. a golden reference | `final_response_match`\nSingle-turn tool use | `tool_use_quality`\n\n**General quality metrics (single-turn, adaptive rubrics)** — for model eval.\n\nGoal | Metric\n----------------------------------------------------- | -----------------------\nOverall response quality (recommended starting point) | `general_quality`\nLinguistic quality (fluency, coherence, grammar) | `text_quality`\nAdherence to specific constraints \u002F instructions | `instruction_following`\n\n**Static rubric metrics (fixed criteria)** — apply alongside the above.\n\nGoal | Metric\n------------------------------------------------- | ---------------\nCatch hallucinated claims (RAG, factual answers) | `hallucination`\nFactuality \u002F consistency against provided context | `grounding`\nSafety policy compliance | `safety`\n\n**Domain-specific check no built-in covers:** write a custom metric.\n\n- **Predefined:** `types.RubricMetric.\u003CNAME>` — server-side AutoRater, no\n judge model needed.\n- **Custom LLM-as-a-judge:** `types.LLMMetric` with `prompt_template` or\n `types.MetricPromptBuilder` for structured rubrics. Always set\n `judge_model`; it defaults to `None` and every case then fails with `400\n INVALID_ARGUMENT: Error parsing JSON`.\n * **Judge Model Selection**: If the user specifies a judge model (e.g.\n `gemini-2.5-pro`), use it. If the user omits the judge model or states\n they do not have information \u002F preference for one, default to\n `gemini-2.5-flash` as the judge model in the dry-run preview card and\n ask for confirmation to run the evaluation. Do NOT halt or refuse to\n evaluate when the user does not specify a judge model.\n- **Custom code:** `types.CodeExecutionMetric` with a `custom_function` string\n containing `def evaluate(instance: dict)` for remote sandboxed execution; or\n `types.Metric` with `custom_function=\u003Ccallable>` for local execution. The\n `instance` dict's `prompt` \u002F `response` \u002F `reference` are `Content` dicts,\n not strings; read\n [references\u002Fmetric_registry.md](references\u002Fmetric_registry.md) (\"What the\n function receives\") before writing the body, rather than discovering the\n shape by running the evaluation.\n\n**Always persist the result** so Stage 4 and 5 can read it. Save both JSON\n(machine-readable, diffable) and HTML (human-readable, linkable):\n\n```python\nimport datetime\nfrom pathlib import Path\n\nfrom agentplatform._genai import _evals_visualization\n\nout_dir = Path(\"artifacts\u002Fgrade_results\")\nout_dir.mkdir(parents=True, exist_ok=True)\nts = datetime.datetime.now().strftime(\"%Y%m%d_%H%M%S\")\n\n# fallback=str, or a DataFrame-backed dataset raises PydanticSerializationError.\nresult_json = result.model_dump_json(fallback=str)\n(out_dir \u002F f\"results_{ts}.json\").write_text(result_json)\n\nhtml = _evals_visualization.get_evaluation_html(result_json)\n(out_dir \u002F f\"results_{ts}.html\").write_text(str(html))\n```\n\nOr after the fact: `scripts\u002Frender_html_report.py --type evaluation` or\n`scripts\u002Finspect_results.py --save-html`.\n\n### 4. Analyze Failures\n\nRead `summary_metrics` and `eval_case_results` — never fabricate scores. Use\n`scripts\u002Finspect_results.py --failing-only` to filter to failures.\n\nFor each failed metric, see\n[references\u002Ffailure_patterns.md](references\u002Ffailure_patterns.md) for deeper\ndiagnoses. The compact mapping:\n\n| Failing metric | What to change |\n| ----------------------------------- | -------------------------------------- |\n| `multi_turn_task_success` low | The agent isn't completing the goal — |\n: : fix orchestration, missing tool calls, :\n: : premature termination, wrong tool :\n: : selection. :\n| `multi_turn_trajectory_quality` low | The agent reaches the goal |\n: : inefficiently — refine planning :\n: : prompts, remove redundant tool calls. :\n| `multi_turn_tool_use_quality` low | Fix tool descriptions, parameter |\n: : docstrings, or agent instructions for :\n: : tool selection. :\n| `final_response_quality` low | Read auto-generated rubric verdicts; |\n: : refine instructions to address the :\n: : worst-scoring criterion. :\n| `final_response_match` low | The agent's final answer doesn't match |\n: : the golden reference — adjust response :\n: : format or update the reference. :\n| `hallucination` low | Tighten instructions to stay grounded |\n: : in tool output; verify the tool :\n: : actually returned the claimed data. :\n| `grounding` low | The response contradicts the provided |\n: : context — add explicit \"cite only from :\n: : context\" instructions. :\n| `safety` low | Add safety guardrails; review the |\n: : violating content category in the :\n: : rubric verdict. :\n| `general_quality` \u002F `text_quality` | Adjust system instruction wording; the |\n: low : model's default phrasing is too :\n: : generic for the task. :\n| `instruction_following` low | The agent is ignoring constraints — |\n: : restate them in the system instruction :\n: : or use stricter wording. :\n| Agent calls wrong tools | Fix tool descriptions, agent |\n: : instructions, or `tool_config`. :\n| Agent calls extra tools | Add explicit stop instructions, or |\n: : switch to :\n: : `multi_turn_tool_use_quality` to :\n: : surface the extra calls in the rubric. :\n\n**For 10+ failures on the same metric**, use the **Error Analysis service** to\ncluster failures into themes (L1\u002FL2 taxonomy categories) instead of reading\nevery trace:\n\n```python\n# Only supports multi_turn_task_success and multi_turn_tool_use_quality.\n# Service runs in the global region.\nanalysis_client = agentplatform.Client(project=\"PROJECT_ID\", location=\"global\")\nresponse = analysis_client.evals.generate_loss_clusters(\n eval_result=result,\n metric=\"multi_turn_task_success\",\n config={\"max_top_cluster_count\": 5},\n)\nfor r in response.results:\n for cluster in r.clusters:\n print(\n f\"[{cluster.taxonomy_entry.l1_category}\u002F\"\n f\"{cluster.taxonomy_entry.l2_category}] \"\n f\"{cluster.item_count} cases — {cluster.taxonomy_entry.description}\"\n )\n```\n\nSave `response.model_dump_json()` and render with `scripts\u002Frender_html_report.py\n--type loss-analysis`.\n\n### 5. Optimize & Iterate\n\nApply a fix targeting the failing metric. Re-run Stage 3. Compare with\n`scripts\u002Fcompare_results.py --baseline \u003Cprev> --candidate \u003Cnew>` to confirm the\ntarget improved AND no other metric regressed.\n\nTrack progress across iterations:\n\nIteration | Metric A | Metric B | Change made\n--------- | -------- | -------- | ----------------------\nBaseline | 0.62 | 0.55 | —\nv2 | 0.78 | 0.68 | Added grounding prompt\nv3 | 0.81 | 0.72 | Fixed tool selection\n\nExpect 5–10+ iterations per failing case. Only after a case passes should you\nexpand coverage with more eval cases.\n\n## Proving your work\n\nNever claim eval results you didn't read from an actual `result` object.\n\n- After running eval, print the `summary_metrics` table\n (`scripts\u002Finspect_results.py`).\n- After a fix, show before\u002Fafter via `scripts\u002Fcompare_results.py`.\n- Before declaring success, confirm ALL cases pass — not just the one you were\n working on.\n\nIf you can't produce the evidence (SDK call failed, result truncated, metric\nunsupported), say so explicitly. Don't paper over gaps.\n\n## Rules of Engagement\n\n1. **Always Plan First:** Before writing a script, output a `\u003Cplan>` block\n detailing the steps you are about to take.\n2. **Step-by-Step Execution:** Prepare the data and evaluation script, present\n the dry-run confirmation card with full parameters, wait for user approval,\n execute only after explicit confirmation, then inspect and analyze results.\n Do NOT run evaluation calls before user confirmation.\n3. **Standard Python:** Use standard Python imports (`import agentplatform`,\n `from google.genai import types`). Don't use internal import paths.\n4. **Verify Before Guessing:** When unsure about SDK types or metrics, check\n the SDK source code rather than guessing or hallucinating.\n5. **Never End a Turn Silently:** Every turn must end with a non-empty,\n informative text reply to the user summarizing the actions taken or\n presenting the next steps. Returning nothing reads as a failure no matter\n what the tools did.\n\n## SDK Quick Reference\n\n```python\nimport agentplatform\nfrom agentplatform import types\nfrom google.genai import types as genai_types\nimport pandas as pd\n\n# Initialize client\nclient = agentplatform.Client(project=\"PROJECT_ID\", location=\"LOCATION\")\n\n# --- SINGLE-TURN EVAL (pandas DataFrame) -- RECOMMENDED ---\n# The converter wraps plain strings for you.\ndf = pd.DataFrame({\n \"prompt\": [\"Q1\", \"Q2\"],\n \"response\": [\"A1\", \"A2\"],\n})\ndataset = types.EvaluationDataset(eval_dataset_df=df)\n\n# --- SINGLE-TURN EVAL (direct EvalCase) ---\n# Verbose and easy to get wrong; see references\u002Fdataset_schema.md for the\n# exact types before using this form.\ndataset = types.EvaluationDataset(eval_cases=[\n types.EvalCase(\n prompt=genai_types.UserContent(\"Query here\"),\n responses=[types.ResponseCandidate(\n response=genai_types.ModelContent(\"Model response here\"))],\n reference=types.ResponseCandidate(\n response=genai_types.ModelContent(\"Ground truth here\")),\n ),\n])\n\n# --- MULTI-TURN AGENT EVAL ---\nagent_data = types.evals.AgentData(\n agents={\"my_agent\": types.evals.AgentConfig(\n agent_id=\"my_agent\", instruction=\"You are helpful.\")},\n turns=[types.evals.ConversationTurn(turn_index=0, events=[\n types.evals.AgentEvent(author=\"user\",\n content=genai_types.Content(role=\"user\",\n parts=[genai_types.Part(text=\"Hello\")])),\n types.evals.AgentEvent(author=\"my_agent\",\n content=genai_types.Content(role=\"model\",\n parts=[genai_types.Part(text=\"Hi! How can I help?\")])),\n ])],\n)\ndataset = types.EvaluationDataset(\n eval_cases=[types.EvalCase(agent_data=agent_data)])\n\n# --- METRICS ---\npredefined = types.RubricMetric.MULTI_TURN_TRAJECTORY_QUALITY\ncustom_llm = types.LLMMetric(name=\"tone\",\n prompt_template=\"Is this polite? Response: {response}\")\ncustom_code = types.CodeExecutionMetric(name=\"check\",\n custom_function='def evaluate(instance): return {\"score\": 1.0}')\n\n# --- EVALUATE ---\nresult = client.evals.evaluate(dataset=dataset, metrics=[predefined])\n\n# --- RESULTS ---\nfor s in result.summary_metrics:\n print(f\"{s.metric_name}: mean={s.mean_score}, pass_rate={s.pass_rate}\")\nfor case in result.eval_case_results:\n for cand in case.response_candidate_results:\n for name, r in cand.metric_results.items():\n print(f\" {name}: score={r.score}, explanation={r.explanation}\")\n```\n\nSee [references\u002Fsdk_patterns.md](references\u002Fsdk_patterns.md) for advanced\npatterns: synthetic data generation, pairwise comparison, `MetricPromptBuilder`,\nmulti-agent evaluation.\n\n## Bundled scripts\n\nScript | When to use\n------------------------ | -----------\n`validate_dataset.py` | Before Stage 3 — catch malformed `EvaluationDataset` JSON.\n`parse_adk_traces.py` | Stage 1 — convert ADK session dumps to the canonical dataset shape.\n`inspect_results.py` | Stages 3\u002F4 — render summary + per-case scores. `--save-html` for a browsable report.\n`compare_results.py` | Stage 5 — diff baseline vs. candidate, detect regressions.\n`render_html_report.py` | Render HTML from a saved result JSON or loss-clusters JSON.\n`endpoint_evaluation.py` | Stages 2\u002F3 against a deployed Agent Platform endpoint (BYOM). See [references\u002Fdeployment.md](references\u002Fdeployment.md).\n`maas_evaluation.py` | Stages 2\u002F3 against a Model-as-a-Service model by ID. See [references\u002Fdeployment.md](references\u002Fdeployment.md).\n",{"name":5,"metadata":52,"description":7},{"version":53,"category":54},"1.0.4","AiAndMachineLearning","main","071c4f7eba880767a647ce778650db8dbf7e16fc","skills\u002Fcloud\u002Fagent-platform-eval-flywheel",2005,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills","https:\u002F\u002Fgithub.com\u002Fgoogle","https:\u002F\u002Fopensource.google\u002F","Google ❤️ Open Source","2026-11-14 01:51:35",{"source":8,"skillName":5,"commands":65,"prompt":75,"zipAvailable":76},[66,69,72],{"tool":67,"command":68},"npx","npx skills add google\u002Fskills --skill agent-platform-eval-flywheel",{"tool":70,"command":71},"bunx","bunx skills add google\u002Fskills --skill agent-platform-eval-flywheel",{"tool":73,"command":74},"pnpm","pnpm dlx skills add google\u002Fskills --skill agent-platform-eval-flywheel","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fagent-platform-eval-flywheel\u002FSKILL.md",true,[78,84,88,92,96,100,104,110,114,118,122,126,130],{"path":79,"resourceType":80,"resourceTypeDesc":81,"sizeBytes":82,"sourceHtmlUrl":83},"references\u002Fattribution.md","REFERENCE","参考文档",1958,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fagent-platform-eval-flywheel\u002Freferences\u002Fattribution.md",{"path":85,"resourceType":80,"resourceTypeDesc":81,"sizeBytes":86,"sourceHtmlUrl":87},"references\u002Fdataset_schema.md",11281,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fagent-platform-eval-flywheel\u002Freferences\u002Fdataset_schema.md",{"path":89,"resourceType":80,"resourceTypeDesc":81,"sizeBytes":90,"sourceHtmlUrl":91},"references\u002Fdeployment.md",7244,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fagent-platform-eval-flywheel\u002Freferences\u002Fdeployment.md",{"path":93,"resourceType":80,"resourceTypeDesc":81,"sizeBytes":94,"sourceHtmlUrl":95},"references\u002Ffailure_patterns.md",6356,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fagent-platform-eval-flywheel\u002Freferences\u002Ffailure_patterns.md",{"path":97,"resourceType":80,"resourceTypeDesc":81,"sizeBytes":98,"sourceHtmlUrl":99},"references\u002Fmetric_registry.md",16457,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fagent-platform-eval-flywheel\u002Freferences\u002Fmetric_registry.md",{"path":101,"resourceType":80,"resourceTypeDesc":81,"sizeBytes":102,"sourceHtmlUrl":103},"references\u002Fsdk_patterns.md",11460,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fagent-platform-eval-flywheel\u002Freferences\u002Fsdk_patterns.md",{"path":105,"resourceType":106,"resourceTypeDesc":107,"sizeBytes":108,"sourceHtmlUrl":109},"scripts\u002Fcompare_results.py","SCRIPT","脚本",6328,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fagent-platform-eval-flywheel\u002Fscripts\u002Fcompare_results.py",{"path":111,"resourceType":106,"resourceTypeDesc":107,"sizeBytes":112,"sourceHtmlUrl":113},"scripts\u002Fendpoint_evaluation.py",5410,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fagent-platform-eval-flywheel\u002Fscripts\u002Fendpoint_evaluation.py",{"path":115,"resourceType":106,"resourceTypeDesc":107,"sizeBytes":116,"sourceHtmlUrl":117},"scripts\u002Finspect_results.py",6824,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fagent-platform-eval-flywheel\u002Fscripts\u002Finspect_results.py",{"path":119,"resourceType":106,"resourceTypeDesc":107,"sizeBytes":120,"sourceHtmlUrl":121},"scripts\u002Fmaas_evaluation.py",2344,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fagent-platform-eval-flywheel\u002Fscripts\u002Fmaas_evaluation.py",{"path":123,"resourceType":106,"resourceTypeDesc":107,"sizeBytes":124,"sourceHtmlUrl":125},"scripts\u002Fparse_adk_traces.py",8639,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fagent-platform-eval-flywheel\u002Fscripts\u002Fparse_adk_traces.py",{"path":127,"resourceType":106,"resourceTypeDesc":107,"sizeBytes":128,"sourceHtmlUrl":129},"scripts\u002Frender_html_report.py",3768,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fagent-platform-eval-flywheel\u002Fscripts\u002Frender_html_report.py",{"path":131,"resourceType":106,"resourceTypeDesc":107,"sizeBytes":132,"sourceHtmlUrl":133},"scripts\u002Fvalidate_dataset.py",11205,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fagent-platform-eval-flywheel\u002Fscripts\u002Fvalidate_dataset.py",[135,170,198,227,253,276],{"skillNo":136,"name":137,"nameZh":6,"description":138,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":139,"categories":140,"keywords":147,"publishedAt":47},"01M6998S2EZKD3GEJ7F943RZTE","sign-in-with-google-web","Implement, configure, and secure Sign In With Google (SiwG) using Google Identity Services (GIS \u002F `https:\u002F\u002Faccounts.google.com\u002Fgsi\u002Fclient`) across web architectures. Use when creating Google sign-in buttons, implementing Google One Tap with FedCM, integrating GIS in React\u002FNext.js\u002FAngular\u002FHTML, verifying ID tokens on backend runtimes (Python\u002FNode.js\u002FGo\u002FJava), enforcing Google Workspace domain restrictions (`hd`), securing client-side ID tokens via WebCrypto nonces or encrypted IndexedDB, embedding in cross-origin iframes via the Intermediate Iframe API (`https:\u002F\u002Faccounts.google.com\u002Fgsi\u002Fintermediate`), configuring Content Security Policy (CSP), COOP, and Permissions Policy (`identity-credentials-get`) headers, or handling sign-out and revocation. Don't use for legacy Google Sign-In (`gapi.auth2`), Google Cloud IAM service accounts, or Android\u002FiOS native Credential Manager.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fidentity\u002Fsign-in-with-google-web\u002FSKILL.md",[141,144],{"categoryNo":142,"categoryName":143,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000009","前端开发",{"categoryNo":145,"categoryName":146,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000010","后端与接口",[9,148,149,150,151,152,153,154,155,46,156,157,158,159,160,161,162,163,164,165,166,167,168,169],"sign in with google web","sign","with","web","android","go","security","google workspace","implement","policy","python","implementing","react","iam","permissions","buttons","secure","manager","identity","architectures","domain","runtimes",{"skillNo":171,"name":172,"nameZh":6,"description":173,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":174,"categories":175,"keywords":183,"publishedAt":47},"01M6998PHPDCAD684VW5WRG83D","dpop-adoption","Implement and debug OAuth 2.0 DPoP (RFC 9449) refresh token sender-constraining for WebCrypto, Node.js ES6, and browser runtimes integrating with Google's OAuth platform. Use when configuring non-extractable asymmetric key pairs (P-256), generating DPoP Proof JWTs for authorization code exchange and token refresh, or handling 400 use_dpop_nonce challenge retry loops at oauth2.googleapis.com\u002Ftoken. Don't use for unconstrained OAuth 2.0 flows (where refresh tokens are not bound to a client key pair), or for Google Cloud IAM \u002F service account authentication.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fidentity\u002Fdpop-adoption\u002FSKILL.md",[176,177,178],{"categoryNo":145,"categoryName":146,"taxonomyType":21,"taxonomyTypeDesc":22},{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},{"categoryNo":179,"categoryName":180,"taxonomyType":181,"taxonomyTypeDesc":182},"01JBSEEDCAT000000000000030","开发工程师","PROFESSION","适用职业",[184,185,9,46,186,187,188,189,190,191,161,192,193,194,195,196,197,166,169,28,31],"dpop adoption","adoption","es6","retry","authentication","flows","challenge","code","browser","loops","at","node","token","pair",{"skillNo":199,"name":200,"nameZh":6,"description":201,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":202,"categories":203,"keywords":210,"publishedAt":47},"01M6998M72TC73EEAQH4XC7EXW","retrieving-developer-knowledge","Searches, retrieves, and synthesizes official Google developer documentation across Google Cloud, AI\u002FGemini, Android, Chrome, Web, Flutter, Go, Firebase, and other Google developer platforms. Integrates with the Developer Knowledge MCP server (search_documents, get_documents, answer_query) or the Developer Knowledge REST API fallback. Use when searching for gcloud CLI commands, API syntax, IAM permissions, official documentation, architectural comparisons, or product choice overviews. Don't use for local filesystem lookups or non-Google documentation.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fdevelopers\u002Fretrieving-developer-knowledge\u002FSKILL.md",[204,205,208,209],{"categoryNo":145,"categoryName":146,"taxonomyType":21,"taxonomyTypeDesc":22},{"categoryNo":206,"categoryName":207,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000021","资料检索与研究",{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},{"categoryNo":179,"categoryName":180,"taxonomyType":181,"taxonomyTypeDesc":182},[211,212,213,214,46,215,216,217,218,219,220,221,222,31,223,224,225,161,226,162],"retrieving developer knowledge","knowledge","developer","MCP","mcp server","official","developer documentation","get","search","local","choice","documents","query","api","across","documentation",{"skillNo":228,"name":229,"nameZh":6,"description":230,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":231,"categories":232,"keywords":233,"publishedAt":47},"01M6998J2CZEXS9BES9S3WWFKC","migrate-to-rust","Clones, analyzes, plans, and migrates legacy codebases (C++, Fortran, Java, Python, C, Shell) from a GitHub URL or local directory to a modern, high-performance, deterministic, 100% Safe Rust architecture, tracking live telemetry across every task bucket, test failure\u002Ffix cycle, and Profile-Guided Optimization (PGO) stage, and producing a first-person AI storytelling report and interactive HTML infographic. Use when migrating a GitHub repository or local codebase to Rust, porting legacy C++, Fortran, Python, or Java workloads to memory-safe Rust, designing Struct-of-Arrays (SoA) or SIMD Rust architectures, or optimizing Rust ports with LLVM PGO. Don't use for unrelated non-Rust refactoring or single-line syntax questions.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fdevelopers\u002Fmigrate-to-rust\u002FSKILL.md",[],[234,235,236,237,238,239,240,241,158,242,243,244,245,246,39,220,247,248,249,31,250,251,225,252],"migrate to rust","rust","migrate","github","memory","performance","directory","safe","ports","llvm","soa","cycle","porting","arrays","infographic","high-performance","URL","architecture","unrelated",{"skillNo":254,"name":255,"nameZh":6,"description":256,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":257,"categories":258,"keywords":262,"publishedAt":47},"01M6998GCC3K29WZ82KXMB9E02","finding-google-skills","Google platform decision and setup guidance, loaded on demand from Google's skill catalog. Use when a developer is choosing or setting up part of their stack, such as where to run a service, a database, storage, messaging, authentication, analytics, ads, or AI model serving, and a Google product is a reasonable candidate - whether or not a vendor is named - or when a request names a Google product or API. Brings in the matching Google skill so the answer can weigh Google options, their trade-offs, and when they are not the right fit. Skip when the stack is already settled on another provider and no Google product is named, or the task involves no platform choice.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fdevelopers\u002Ffinding-google-skills\u002FSKILL.md",[259,260,261],{"categoryNo":145,"categoryName":146,"taxonomyType":21,"taxonomyTypeDesc":22},{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},{"categoryNo":179,"categoryName":180,"taxonomyType":181,"taxonomyTypeDesc":182},[9,263,264,37,265,28,41,266,267,268,269,188,270,271,272,221,273,274,31,275,224],"finding google skills","storage","messaging","trade-offs","provider","answer","fit","catalog","up","decision","use","database","candidate",{"skillNo":277,"name":278,"nameZh":6,"description":279,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":280,"categories":281,"keywords":284,"publishedAt":47},"01M6998DWNHTH0BS6W1CY8ZXMP","workload-manager-basics","Use this skill to manage Google Cloud Workload Manager evaluations, rules, scanned resources, and validation results by using public client libraries and the REST API. Use when you need to inspect workload best-practice rules, create and run evaluations for Google Cloud general best practices, SAP, SQL Server, or custom organizational rules, review violations, export results to BigQuery, or automate Workload Manager through client libraries because no service-specific public CLI or MCP server is available. Don't use for general Google Compute Engine instance management, VPC configuration, or standard IAM auditing.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fworkload-manager-basics\u002FSKILL.md",[282,283],{"categoryNo":145,"categoryName":146,"taxonomyType":21,"taxonomyTypeDesc":22},{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},[285,286,165,287,9,214,288,46,215,289,290,291,292,31,293,224,161,294,295,296,297],"workload manager basics","workload","basics","cli","review","vpc","resources","inspect","you","create","libraries","server","run"]