[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6990PXZ0S6NHYV8PGGVFT6E":3,"skill-same:01M6990PXZ0S6NHYV8PGGVFT6E":114},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":28,"publishedAt":48,"linkOnly":49,"markdownContent":50,"markdownBody":51,"frontMatter":52,"refName":60,"commitSha":61,"skillDirectory":62,"forksCount":63,"repositoryHtmlUrl":64,"repositoryHomepageUrl":65,"ownerHtmlUrl":66,"ownerBlogUrl":67,"ownerDescription":68,"updatedAt":69,"install":70,"resources":83},"01M6990PXZ0S6NHYV8PGGVFT6E","google-agents-cli-eval",null,"This skill should be used when the user wants to \"run an evaluation\", \"evaluate my agent\", \"evaluate my ADK agent\", \"write an eval dataset\", \"analyze eval failures\", \"compare eval results\", \"optimize agent\", or needs guidance on the Gemini platform eval methodology and the Quality Flywheel. Covers eval metrics, dataset schema, LLM-as-judge scoring, and common failure causes. Applies to any agents-cli project, whatever framework the agent is written in. Do NOT use for agent API code patterns (ADK: use google-agents-cli-adk-code), deployment (use google-agents-cli-deploy), or project scaffolding (use google-agents-cli-scaffold).","google\u002Fagents-cli","google","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F1342004?v=4","01M698VYEFDM0VJBZDYYJ2T83S","Google","01M698ZVGDEBGAXVZGNKKKCRBQ",6349,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fagents-cli\u002Fblob\u002Fmain\u002Fsrc\u002Fgoogle\u002Fagents\u002Fcli\u002Fskills\u002Fdata\u002Fgoogle-agents-cli-eval\u002FSKILL.md",[18,23],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",{"categoryNo":24,"categoryName":25,"taxonomyType":26,"taxonomyTypeDesc":27},"01JBSEEDCAT000000000000030","开发工程师","PROFESSION","适用职业",[29,30,9,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47],"cli","google agents cli eval","eval","write","run","apache","optimize","analyze","common","api","scoring","code","code patterns","dataset","uv","scaffolding","project","project scaffolding","adk","2026-11-10 10:12:30",false,"---\nname: google-agents-cli-eval\ndescription: >\n  This skill should be used when the user wants to \"run an evaluation\",\n  \"evaluate my agent\", \"evaluate my ADK agent\", \"write an eval dataset\",\n  \"analyze eval failures\", \"compare eval results\", \"optimize agent\", or needs\n  guidance on the Gemini platform eval methodology and the Quality Flywheel.\n  Covers eval metrics, dataset schema, LLM-as-judge scoring, and common failure causes.\n  Applies to any agents-cli project, whatever framework the agent is written in.\n  Do NOT use for agent API code patterns (ADK: use google-agents-cli-adk-code), deployment\n  (use google-agents-cli-deploy), or project scaffolding (use google-agents-cli-scaffold).\nmetadata:\n  author: Google\n  license: Apache-2.0\n  version: 1.14.0\n  requires:\n    bins:\n      - agents-cli\n    install: \"uv tool install google-agents-cli\"\n---\n\n# Agent Evaluation Guide\n\n> **Requires:** `agents-cli` (`uv tool install google-agents-cli`) — [install uv](https:\u002F\u002Fdocs.astral.sh\u002Fuv\u002Fgetting-started\u002Finstallation\u002Findex.md) first if needed.\n\n> **Scaffolded project?** If you used `\u002Fgoogle-agents-cli-scaffold`, dataset and a custom metric are already scaffolded in `tests\u002Feval\u002F` (Python projects) or `eval\u002F` (Go projects). For simplicity, this skill and its references use the Python directory layout; adjust accordingly if you've scaffolded a Go agent.\n> You already have `agents-cli eval run` (chains `generate` + `grade`), `tests\u002Feval\u002Fdatasets\u002F`, and `tests\u002Feval\u002Feval_config.yaml`. Start with executing `eval run` and iterate from there.\n\n## Reference Files\n\n| File | Contents |\n|------|----------|\n| `references\u002Fdataset_schema.md` | Canonical EvaluationDataset schema — all field types, JSON examples for single-turn \u002F multi-turn \u002F multi-agent, common mistakes |\n| `references\u002Fmetrics-guide.md` | Complete metrics reference — all built-in metrics, match types, custom metrics, judge model config |\n| `references\u002Fuser-simulation.md` | Dynamic conversation testing — `eval dataset synthesize` flags, what scenarios are, compatible metrics |\n| `references\u002Fbuiltin-tools-eval.md` | google_search and model-internal tools — trajectory behavior, metric compatibility |\n| `references\u002Fadvanced-commands.md` | Opt-in commands: `eval analyze`, `eval optimize`, `eval submit` \u002F `eval results` |\n| `references\u002Fmultimodal-eval.md` | Multimodal inputs — eval dataset schema, built-in metric limitations, custom evaluator pattern |\n| `references\u002Flive-eval.md` | Live and voice agents — `--mode adk_live`, what gets graded, user-only turn authoring, the Live-model and region traps |\n\n---\n\n## The Quality Flywheel\n\nImproving agent quality is iterative. The 4 stages below describe the loop. Each stage has a Default path (you, the coding agent, do the work directly) and an Opt-in CLI command that delegates to the Eval Service in Gemini platform for better quality and scale.\n\n### 1. Prepare Data\n\n**Default:** Use or edit the scaffolded `tests\u002Feval\u002Fdatasets\u002Fbasic-dataset.json` to define single-turn eval inputs. Start with 1–2 cases.\n\n**Opt-in (ADK projects):** `agents-cli eval dataset synthesize`: user-simulate multi-turn datasets when you lack data; its output already includes traces, so Stage 2 collapses to `agents-cli eval grade` alone. See *Eval Commands* and `references\u002Fuser-simulation.md`.\n\n### 2. Run the Eval (always run)\n\n**Default:** `agents-cli eval run` runs the agent over the dataset and grades the traces, writing `results_\u003Cts>.{json,html}` to `artifacts\u002Fgrade_results\u002F`. Its last line is the verdict, e.g. `RESULT: FAIL (custom_response_quality 3.20 \u003C 4.00; custom_response_quality: 1 case errored)`, and it exits 1 when a metric misses its threshold or a case errored. Agree the thresholds with the user and record them in the eval config (see *Evaluation Configuration Schema*).\n\n**Decoupled form:** `eval generate` then `eval grade`, for a custom traces location, re-grading without re-running the agent, or traces from `synthesize` (`eval grade` alone).\n\n### 3. Analyze Failures\n\n**Default:** Start from the `Failures:` list above the `RESULT:` line, which names each metric that missed its threshold, the cases below it, and the cases that errored. Then open the latest `artifacts\u002Fgrade_results\u002Fresults_\u003Cts>.html` (or `.json`) for those cases' scores and judge rationales — see *What to fix when scores fail* below for the fix table.\n\n**Opt-in:** `agents-cli eval analyze`, LLM-based failure clustering; prefer when you have 10+ failing cases and want categorized failure modes. See `references\u002Fadvanced-commands.md`.\n\n### 4. Optimize & Code Fix\n\n**Default:** Edit the agent — adjust prompts, tool descriptions, instructions, or eval dataset based on the failure analysis. See *What to fix when scores fail* below for the failure → fix mapping.\n\n**Opt-in (ADK projects):** `agents-cli eval optimize` runs ADK GEPA prompt optimization against a target metric (see `references\u002Fadvanced-commands.md`). Suitable for prompt-only failures. The optimized prompt appears in the command output; capture it and apply it to the agent. For the full per-iteration trace, set `print_detailed_results: true` in your optimization config file.\n\n> **Long-running and expensive.** GEPA optimization makes many LLM calls and can take a long time. Do not run it unless the user explicitly asks for prompt optimization. When you do run it, iterate as far as possible with manual fixes first, then run a **single** final `eval optimize` — never loop on this command.\n\n### Running the loop\n\nIterate stages 2 → 3 → 4 → 2 (with `synthesize`, re-run Stage 1 each pass, then `eval grade`). After each fix, run `agents-cli eval compare \u003Cprev_results>.json \u003Cnew_results>.json` to confirm the target metric improved without regressing others. Expect 5–10+ iterations per case before it passes, which is normal. Only after a case passes should you expand coverage with more eval cases.\n\nWhen doing 5+ iterations, maintain a task list of which cases are fixed, which are still failing, and what fixes you've tried. Prevents re-attempting the same fix.\n\n**Hold cases back.** Keep a slice of cases out of the loop and grade them only when you think you're done — otherwise you can't tell a fix that generalizes from one fitted to the cases you iterated against.\n\n### Shortcuts That Waste Time\n\nRecognize these rationalizations and push back — they always cost more time than they save:\n\n| Shortcut | Why it fails |\n|----------|-------------|\n| \"I'll lower the bar so it passes\" | Lowering the bar hides real failures. If the agent can't meet the bar, fix the agent, don't move the bar. |\n| \"This eval case is flaky, I'll skip it\" | Flaky evals reveal non-determinism in your agent. Fix with `temperature=0`, rubric-based metrics, or more specific instructions — don't delete the signal. |\n| \"I just need to fix the eval dataset, not the agent\" | If you're always adjusting expected outputs, your agent has a behavior problem. Fix the instructions or tool logic first. |\n| \"I'll iterate until every case I have passes\" | Nothing is left to detect overfitting to your own cases. See *Hold cases back* above. |\n\n## Choosing the Right Metrics\n\nPick built-in metrics by what you want to measure. Only `multi_turn_task_success`, `multi_turn_trajectory_quality`, and `multi_turn_tool_use_quality` accept multi-turn traces; every other built-in 400s on one. When no built-in fits, write a custom metric (see *Evaluation Configuration Schema* below).\n\n| Goal | Recommended built-in metrics |\n|------|------------------------------|\n| **Did the agent achieve the user's goal?** (catch-all for multi-turn agents) | `multi_turn_task_success` |\n| **Was the agent's reasoning path logical and efficient?** | `multi_turn_trajectory_quality` |\n| **Quality of tool \u002F function calling across turns** | `multi_turn_tool_use_quality` |\n| **Final response quality** (no ground-truth reference needed) | `final_response_quality` |\n| **Factual grounding** (catch hallucinated claims, e.g., RAG agents) | `hallucination`, or `grounding` when the case carries a `context` field |\n| **Safety policy compliance** | `safety` |\n| **Match against a golden answer** | `final_response_match` (needs `reference` on the case) |\n| **Different pass\u002Ffail criteria per case** | Put them on the case as `rubric_groups` and grade with a managed rubric metric. See `references\u002Fdataset_schema.md` (*Per-Case Rubrics*). |\n| **Domain-specific check no built-in covers** | Write a custom `LLMMetric` (LLM-judge) or `CodeExecutionMetric` (deterministic Python). See *Evaluation Configuration Schema* below. |\n\nRun `agents-cli eval metric list` to see all available built-ins. For full metric definitions and rubric details, see the [Gemini platform metric docs](https:\u002F\u002Fcloud.google.com\u002Fgemini-enterprise-agent-platform\u002Foptimize\u002Fevaluation\u002Fmanage-metrics) and `references\u002Fmetrics-guide.md`.\n\n---\n\n## What to fix when scores fail\n\nAfter `agents-cli eval run` completes, inspect the latest `artifacts\u002Fgrade_results\u002Fresults_\u003Ctimestamp>.json` (or open the `.html` file) for per-case scores and judge rationales, the input to every fix decision below.\n\n| Failure | What to change |\n|---------|---------------|\n| `multi_turn_task_success` low | The agent isn't completing the user's goal — fix orchestration, missing tool calls, premature termination, or wrong tool selection |\n| `multi_turn_trajectory_quality` low | The agent reaches the goal inefficiently or takes wrong steps — refine planning prompts, tighten instruction order, or remove redundant tool calls |\n| `multi_turn_tool_use_quality` low | Fix tool descriptions, parameter docstrings, or agent instructions for tool selection |\n| `final_response_quality` low | Read the auto-generated rubric verdicts; refine agent instructions to address the worst-scoring criterion (often clarity, completeness, or instruction-following) |\n| `hallucination` low | Tighten agent instructions to stay grounded in tool output; verify the tool actually returned the data the agent claimed |\n| `safety` low | Add safety guardrails to instructions; review the violating content category in the rubric verdict |\n| Agent calls wrong tools | Fix tool descriptions, agent instructions, or the model's tool-choice config (**ADK:** `tool_config`) |\n| Agent calls extra tools | Add strict stop instructions, or switch to `multi_turn_tool_use_quality` |\n\nAfter applying a fix, rerun `agents-cli eval run` and use `agents-cli eval compare \u003Cprev_results>.json \u003Cnew_results>.json` to confirm the fix improved the target metric without regressing others.\n\n---\n\n## Eval Commands\n\n`agents-cli eval \u003Csubcommand> --help` is the authoritative flag list; the examples below are the common invocations.\n\n### `eval run` (default)\n\nRuns the agent over the dataset and grades the traces in one command.\n\n```bash\n# Basic: dataset from tests\u002Feval\u002Fdatasets\u002F, results to artifacts\u002Fgrade_results\u002F,\n# metrics from tests\u002Feval\u002Feval_config.yaml\nagents-cli eval run\n\n# Advanced: pick the dataset, metrics, and output dir\nagents-cli eval run --dataset tests\u002Feval\u002Fdatasets\u002Fcustom.json --metrics final_response_quality,safety --output .\u002Fout\u002F\n\n# Gate: exit 1 if the 1-5 judge averages below 4\nagents-cli eval run --fail-under custom_response_quality=4\n```\n\n### `eval generate`\n\nRuns an agent over an evaluation dataset and writes traces to disk.\n\nBy default, runs the agent locally and records a trace per evaluation case. You can generate traces from an already-running agent by passing its HTTP endpoint and app name to `--url` and `--app-name`.\n\n> **ADK projects.** The built-in generator serves the agent over HTTP and drives it over ADK's `\u002Fapps\u002F...` and `\u002Frun_sse` routes — the same shape `--url` \u002F `--app-name` expect. What it launches depends on the project's language: Python uses the project's `fast_api_app.py` if it exists, else `adk api_server`; Go runs `go run ...`. Extensions for other frameworks replace `eval generate` with their own generator, which may not serve HTTP at all; `--url` and `--app-name` are then unsupported.\n\n```bash\n# Basic — uses tests\u002Feval\u002Fdatasets\u002F, writes to artifacts\u002Ftraces\u002F\nagents-cli eval generate\n\n# Advanced — custom dataset and output dir\nagents-cli eval generate --dataset tests\u002Feval\u002Fdatasets\u002Fcustom.json -o .\u002Fcustom_traces\u002F\n\n# Against a deployed agent (or one you started manually)\nagents-cli eval generate --url https:\u002F\u002Fmy-agent.run.app --app-name app\n\n# Live agent — stream each case over ADK's \u002Frun_live WebSocket\nagents-cli eval generate --mode adk_live\n```\n\n#### Evaluating Live agents\n\nLive agents run over a WebSocket, not `\u002Frun_sse`: add `--mode adk_live` to `generate` or `run`. The dataset, the traces, and grading are unchanged, and the audio reply is transcribed so the transcript is what gets graded. Two things must already be true or the socket connects and *then* fails mid-session: the agent uses a Live model (the scaffold default is not one), and on Gemini platform its region is pinned on the model rather than left to `GOOGLE_CLOUD_LOCATION`. Both, plus dataset authoring rules: `references\u002Flive-eval.md`.\n\n### `eval grade`\n\nScores traces (from `eval generate`, `eval dataset synthesize`, or hand-authored) against built-in or custom metrics. Writes timestamped `results_\u003CYYYYMMDD_HHMMSS>.json` (consumed by `eval compare`) and `.html` (open in a browser) into the output dir, and prints a summary table to the console. The output ends with a `RESULT: PASS|FAIL (...)` line; the command exits 1 when a metric's mean score is below its threshold (`thresholds` in the config, or `--fail-under`) or any case errored, so CI can gate on the exit code.\n\n```bash\n# Basic — defaults: traces from artifacts\u002Ftraces\u002F, results to artifacts\u002Fgrade_results\u002F,\n# metrics from tests\u002Feval\u002Feval_config.yaml's metrics_to_run\nagents-cli eval grade\n\n# Advanced 1 — grade traces from a non-default location (the canonical\n# pairing for `eval generate --output custom_traces\u002F`)\nagents-cli eval grade --traces custom_traces\u002F\n\n# Advanced 2: load metrics to run from a config file (YAML or JSON) on a specified trace file.\nagents-cli eval grade --traces .\u002Fartifacts\u002Ftraces\u002Ftrace_1.json --config tests\u002Feval\u002Feval_config.yaml\n\n# Advanced 3: dispatch rate, 15 metric computations per second by default. Lower it when the\n# judge model or the eval service rate-limits you, raise it when they have headroom.\nagents-cli eval grade --qps 5\n```\n\nSee *Evaluation Configuration Schema* below for the config file format.\n\n### `eval compare`\n\nDiffs two `results_*.json` files from an eval run. Run it after a fix to confirm the target metric improved without regressing others.\n\n```bash\nagents-cli eval compare baseline.json candidate.json\n```\n\n### `eval dataset synthesize`\n\n> **ADK projects.** It loads and runs the agent through ADK, so it is unavailable on other frameworks.\n\nGenerates user scenarios from your agent's tools and instructions, plays each against an LLM-backed user simulator, and writes graded-ready traces to `artifacts\u002Ftraces\u002F` (feed straight to `eval grade`, skip `eval generate`). Invocations, flags, and compatible metrics: `references\u002Fuser-simulation.md`.\n\n### Advanced commands\n\n`eval analyze` (cluster failure modes), `eval optimize` (GEPA prompt tuning), and `eval submit` \u002F `eval results` (managed cloud-side runs for CI or large datasets) are documented in `references\u002Fadvanced-commands.md`.\n\n---\n\n## Evaluation Dataset Format\n\nAn `EvaluationDataset` is a JSON file with an `eval_cases` array. Cases come in two shapes depending on how they're used:\n\n- **Inference input** (what you give to `eval generate`) — a single user prompt, or a multi-turn set of **user turns** (for live) \u002F a continuation ending in a user turn (for SSE). The agent runs and produces traces. Don't pre-author agent replies for live inference.\n- **Grading input** (what you give to `eval grade`) — a complete trace including the agent's responses and tool calls. Normally produced by `eval generate` or `eval dataset synthesize`; you don't write these by hand.\n\nSee `references\u002Fdataset_schema.md` for the full canonical schema, all field types, and common mistakes.\n\n### Inference input format\n\nTwo shapes are supported.\n\n**(a) Simple single-turn prompt** — what the scaffolded `tests\u002Feval\u002Fdatasets\u002Fbasic-dataset.json` uses. The agent runs from scratch.\n\n```json\n{\n  \"eval_cases\": [\n    {\n      \"eval_case_id\": \"greeting\",\n      \"prompt\": {\n        \"role\": \"user\",\n        \"parts\": [{\"text\": \"Hello, what can you help me with?\"}]\n      }\n    }\n  ]\n}\n```\n\n**(b) Multi-turn via `agent_data`** — the shape depends on the transport:\n\n- **Live (`--mode adk_live`):** author **user-only** turns; the agent generates every reply over one live session (authored agent turns are ignored, with a warning).\n- **SSE:** continuation form — prior turns are seeded as history and only the trailing user turn is answered.\n\nSee `references\u002Fdataset_schema.md` (*Multi-Turn \u002F Multi-Agent Dataset*) for the JSON shapes.\n\n### Grading input format (traces)\n\nA complete trace — agent responses plus `function_call` \u002F `function_response` parts — normally produced by `eval generate` \u002F `eval dataset synthesize` (you don't write these by hand). Authors are `\"user\"`, an agent ID from the `agents` map, or `\"tool\"`. See `references\u002Fdataset_schema.md` for the trace shape, multi-agent examples, and the full type reference.\n\n---\n\n## Evaluation Configuration Schema\n\n`agents-cli eval run --config \u003Cpath>` (and `eval grade --config \u003Cpath>`) accepts a single configuration file in either **YAML** (`.yaml` \u002F `.yml`) or **JSON** (`.json`). The file declares two parts:\n\n- `metrics_to_run`: the **selection list** of metric names to execute on this run. A name resolves to a `custom_metrics` entry when one matches, otherwise to the built-in metric of that name.\n- `custom_metrics` — a **definition pool** of custom metrics available to this project. Defining a metric here does **not** run it; it must also appear in `metrics_to_run` (or be passed via `--metrics name1,name2` on the CLI, which is equivalent to overriding `metrics_to_run` for that invocation).\n- `thresholds` (optional) — the minimum **mean score** per metric, in that metric's own scale (built-ins score 0-1, the scaffolded `custom_response_quality` judge 1-5). A run with a metric below its threshold exits 1 and lists the cases scoring below it; `pass_rate` in the results is then the share of cases at or above it, unless the eval service already reported one. `--fail-under METRIC=SCORE` (repeatable, one metric each) overrides them per run. Errored cases fail the run with or without thresholds.\n\n**Minimal example (YAML preferred — human-readable, no JSON escaping for prompts and Python):**\n\n```yaml\nmetrics_to_run:\n  - multi_turn_task_success     # built-in\n  - example_llm_metric          # selected from custom_metrics pool below\n  - agent_turn_count            # selected from custom_metrics pool below\n\ncustom_metrics:\n  - name: example_llm_metric\n    prompt_template: |\n      Rate the agent's response 1-5 for helpfulness and accuracy.\n      Prompt: {prompt}\n      Final response: {response}\n      Full trace (for tool-call and reasoning context): {agent_data}\n      Return JSON: {\"score\": \u003C1|2|3|4|5>, \"explanation\": \"\u003Creason>\"}\n\n  - name: agent_turn_count\n    custom_function: |\n      def evaluate(instance):\n          turns = (instance.get(\"agent_data\") or {}).get(\"turns\", [])\n          return {'score': len(turns)}\n\nthresholds:\n  multi_turn_task_success: 0.8\n  example_llm_metric: 4\n```\n\nJSON is also accepted (same field names, with `prompt_template` and `custom_function` as escaped strings) — but **always prefer YAML** for human-readable configs.\n\nDispatch by field: `custom_function` → Python metric; `prompt_template` → `LLMMetric` (LLM-as-judge); neither, on a built-in name → parameterizes that built-in (e.g. `metric_spec_parameters.rubric_group_key`). Field reference: `references\u002Fmetrics-guide.md`.\n\n**Agent trace field model.** For datasets produced by `agents-cli eval generate` (or `eval dataset synthesize`), each eval case exposes three standard fields to a metric:\n\n- `{prompt}` — the user message (or first user turn).\n- `{response}` — the agent's final text response, extracted from the last text-bearing event. In `custom_function` callbacks this is `instance['response']` with shape `{\"role\": \"model\", \"parts\": [{\"text\": \"...\"}]}`.\n- `{agent_data}` — the full structured `turns`\u002F`events` trace, useful when the judge needs to reason about tool calls or intermediate reasoning.\n\n`reference`, `context`, and `rubric_groups` are yours to author on the case: `eval generate` carries them onto the trace but never invents them, so `{reference}` \u002F `{context}` resolve only where you wrote them. `rubric_groups` is not a placeholder at all: managed rubric metrics read it off the case, and a `custom_function` sees `instance['rubric_groups']`. See `references\u002Fdataset_schema.md` (*Per-Case Rubrics*).\n\nCode-based metrics default to **local in-process execution** (no GCP project or region required, but the `evaluate(instance)` function runs with the CLI's privileges). Set `execution: \"remote\"` on the metric to run it server-side in the `CodeExecutionMetric` sandbox in Gemini platform instead — that path requires a configured GCP project + region.\n\n---\n\n\n## Common Gotchas\n\n### Use Rubric-Based Tool Evaluation instead of Hardcoded Sequences\n\nEvaluating agent tool usage using strict sequence matching is fragile because agents may call helper tools (like searches or geocoding) in different orders or perform extra proactive steps.\n\nInstead, use **`multi_turn_tool_use_quality`** \u002F **`multi_turn_trajectory_quality`**. These metrics automatically generate content-based and intent-based adaptive rubrics, assessing technical correctness and technical sequence logic semantically using an LLM judge rather than forcing a rigid match.\n\n### App name must match directory name\n\n> **ADK projects.**\n\nThe `App` object's `name` parameter MUST match the directory containing your agent:\n\n```python\n# CORRECT - matches the \"app\" directory\napp = App(root_agent=root_agent, name=\"app\")\n\n# WRONG - causes \"Session not found\" errors\napp = App(root_agent=root_agent, name=\"flight_booking_assistant\")\n```\n\n### Gemini platform eval region\n\n`eval run`, `eval grade`, and `eval submit` **default to the `global` endpoint**. They don't inherit the manifest `region` (the eval services support only a subset of regions), and `eval analyze` is `global`-only. Override these per run with `--region \u003CREGION>` (e.g. data residency); the service rejects an unsupported one:\n\n```\n400 FAILED_PRECONDITION: Unsupported region for Vertex Evaluation Service: \u003Cregion>\n```\n\n`eval generate` (without the `--url` flag) and `eval dataset synthesize` run your agent locally, so they honor the agent's own `.env` — notably `GOOGLE_CLOUD_LOCATION`, which selects the model endpoint **when the agent uses Gemini platform** (`GOOGLE_GENAI_USE_ENTERPRISE=true`); it's unused with a `GEMINI_API_KEY` (AI Studio). They take **no** `--region` and never override your `.env` with the manifest `region`; change the model region by editing `.env` — or, better for a single agent, pin it in code with `Gemini(model=…, client_kwargs={\"location\": …})`, which beats the env var and leaves it free for everything else. One caveat for `synthesize`: its scenario-generation step is a **server-side** eval call at `GOOGLE_CLOUD_LOCATION`, so keep that an eval-supported region (`global` by default) even though the agent itself could run elsewhere.\n\n**No eval region fits your data-residency rules?** Fall back to **local custom metrics** — a `custom_metrics` entry with a `custom_function` (`execution: local`, the default) grades in-process with no GCP region required. You lose the managed built-in metrics, but your `custom_function` can still call an LLM judge in a compliant region itself — so LLM-as-judge grading stays available anywhere.\n\n### The `before_agent_callback` Pattern (State Initialization)\n\n> **ADK projects.**\n\nAlways use a callback to initialize session state variables used in your instruction template. This prevents `KeyError` crashes on the first turn:\n\n```python\nasync def initialize_state(callback_context: CallbackContext) -> None:\n    state = callback_context.state\n    if \"user_preferences\" not in state:\n        state[\"user_preferences\"] = {}\n\nroot_agent = Agent(\n    name=\"my_agent\",\n    before_agent_callback=initialize_state,\n    instruction=\"Based on preferences: {user_preferences}...\",\n)\n```\n\n### Model thinking mode may bypass tools\n\nModels with \"thinking\" enabled may skip tool calls. Force tool usage through the model's tool-choice config (**ADK:** `tool_config` with `mode=\"ANY\"`), or switch to a non-thinking model for predictable tool calling.\n\n---\n\n## Common Eval Failure Causes\n\n| Symptom | Cause | Fix |\n|---------|-------|-----|\n| Score fluctuates between runs | Non-deterministic model | Set `temperature=0` or use rubric-based eval with multiple samples |\n| LLM judge ignores image\u002Faudio in eval | `get_text_from_content()` skips non-text parts | Use custom metric with vision-capable judge (see `references\u002Fmultimodal-eval.md`) |\n\n---\n\n## Proving Your Work\n\nDon't assert that eval passes — show the evidence. Concrete output prevents false confidence and catches issues early.\n\n- **After running eval:** Paste the scores table output so the user can see exactly what passed and failed.\n- **After fixing a failure:** Show before\u002Fafter scores for the specific case you fixed, and confirm no other cases regressed.\n- **Before deploy:** Rerun `agents-cli eval run` and show the scores for every case, not just the one you fixed, plus the final `RESULT:` line. The exit code gates only on the `thresholds` you set (and errored cases), so confirm they match the bar you agreed with the user.\n\n---\n\n## Related Skills\n\n- `\u002Fgoogle-agents-cli-workflow` — Development workflow and the spec-driven build-evaluate-deploy lifecycle\n- `\u002Fgoogle-agents-cli-adk-code` — ADK API quick reference for writing agent code (ADK projects only)\n- `\u002Fgoogle-agents-cli-scaffold` — Project creation and enhancement with `agents-cli scaffold create` \u002F `scaffold enhance`\n- `\u002Fgoogle-agents-cli-deploy` — Deployment targets, CI\u002FCD pipelines, and production workflows\n- `\u002Fgoogle-agents-cli-observability` — Cloud Trace, logging, and monitoring for debugging agent behavior\n","# Agent Evaluation Guide\n\n> **Requires:** `agents-cli` (`uv tool install google-agents-cli`) — [install uv](https:\u002F\u002Fdocs.astral.sh\u002Fuv\u002Fgetting-started\u002Finstallation\u002Findex.md) first if needed.\n\n> **Scaffolded project?** If you used `\u002Fgoogle-agents-cli-scaffold`, dataset and a custom metric are already scaffolded in `tests\u002Feval\u002F` (Python projects) or `eval\u002F` (Go projects). For simplicity, this skill and its references use the Python directory layout; adjust accordingly if you've scaffolded a Go agent.\n> You already have `agents-cli eval run` (chains `generate` + `grade`), `tests\u002Feval\u002Fdatasets\u002F`, and `tests\u002Feval\u002Feval_config.yaml`. Start with executing `eval run` and iterate from there.\n\n## Reference Files\n\n| File | Contents |\n|------|----------|\n| `references\u002Fdataset_schema.md` | Canonical EvaluationDataset schema — all field types, JSON examples for single-turn \u002F multi-turn \u002F multi-agent, common mistakes |\n| `references\u002Fmetrics-guide.md` | Complete metrics reference — all built-in metrics, match types, custom metrics, judge model config |\n| `references\u002Fuser-simulation.md` | Dynamic conversation testing — `eval dataset synthesize` flags, what scenarios are, compatible metrics |\n| `references\u002Fbuiltin-tools-eval.md` | google_search and model-internal tools — trajectory behavior, metric compatibility |\n| `references\u002Fadvanced-commands.md` | Opt-in commands: `eval analyze`, `eval optimize`, `eval submit` \u002F `eval results` |\n| `references\u002Fmultimodal-eval.md` | Multimodal inputs — eval dataset schema, built-in metric limitations, custom evaluator pattern |\n| `references\u002Flive-eval.md` | Live and voice agents — `--mode adk_live`, what gets graded, user-only turn authoring, the Live-model and region traps |\n\n---\n\n## The Quality Flywheel\n\nImproving agent quality is iterative. The 4 stages below describe the loop. Each stage has a Default path (you, the coding agent, do the work directly) and an Opt-in CLI command that delegates to the Eval Service in Gemini platform for better quality and scale.\n\n### 1. Prepare Data\n\n**Default:** Use or edit the scaffolded `tests\u002Feval\u002Fdatasets\u002Fbasic-dataset.json` to define single-turn eval inputs. Start with 1–2 cases.\n\n**Opt-in (ADK projects):** `agents-cli eval dataset synthesize`: user-simulate multi-turn datasets when you lack data; its output already includes traces, so Stage 2 collapses to `agents-cli eval grade` alone. See *Eval Commands* and `references\u002Fuser-simulation.md`.\n\n### 2. Run the Eval (always run)\n\n**Default:** `agents-cli eval run` runs the agent over the dataset and grades the traces, writing `results_\u003Cts>.{json,html}` to `artifacts\u002Fgrade_results\u002F`. Its last line is the verdict, e.g. `RESULT: FAIL (custom_response_quality 3.20 \u003C 4.00; custom_response_quality: 1 case errored)`, and it exits 1 when a metric misses its threshold or a case errored. Agree the thresholds with the user and record them in the eval config (see *Evaluation Configuration Schema*).\n\n**Decoupled form:** `eval generate` then `eval grade`, for a custom traces location, re-grading without re-running the agent, or traces from `synthesize` (`eval grade` alone).\n\n### 3. Analyze Failures\n\n**Default:** Start from the `Failures:` list above the `RESULT:` line, which names each metric that missed its threshold, the cases below it, and the cases that errored. Then open the latest `artifacts\u002Fgrade_results\u002Fresults_\u003Cts>.html` (or `.json`) for those cases' scores and judge rationales — see *What to fix when scores fail* below for the fix table.\n\n**Opt-in:** `agents-cli eval analyze`, LLM-based failure clustering; prefer when you have 10+ failing cases and want categorized failure modes. See `references\u002Fadvanced-commands.md`.\n\n### 4. Optimize & Code Fix\n\n**Default:** Edit the agent — adjust prompts, tool descriptions, instructions, or eval dataset based on the failure analysis. See *What to fix when scores fail* below for the failure → fix mapping.\n\n**Opt-in (ADK projects):** `agents-cli eval optimize` runs ADK GEPA prompt optimization against a target metric (see `references\u002Fadvanced-commands.md`). Suitable for prompt-only failures. The optimized prompt appears in the command output; capture it and apply it to the agent. For the full per-iteration trace, set `print_detailed_results: true` in your optimization config file.\n\n> **Long-running and expensive.** GEPA optimization makes many LLM calls and can take a long time. Do not run it unless the user explicitly asks for prompt optimization. When you do run it, iterate as far as possible with manual fixes first, then run a **single** final `eval optimize` — never loop on this command.\n\n### Running the loop\n\nIterate stages 2 → 3 → 4 → 2 (with `synthesize`, re-run Stage 1 each pass, then `eval grade`). After each fix, run `agents-cli eval compare \u003Cprev_results>.json \u003Cnew_results>.json` to confirm the target metric improved without regressing others. Expect 5–10+ iterations per case before it passes, which is normal. Only after a case passes should you expand coverage with more eval cases.\n\nWhen doing 5+ iterations, maintain a task list of which cases are fixed, which are still failing, and what fixes you've tried. Prevents re-attempting the same fix.\n\n**Hold cases back.** Keep a slice of cases out of the loop and grade them only when you think you're done — otherwise you can't tell a fix that generalizes from one fitted to the cases you iterated against.\n\n### Shortcuts That Waste Time\n\nRecognize these rationalizations and push back — they always cost more time than they save:\n\n| Shortcut | Why it fails |\n|----------|-------------|\n| \"I'll lower the bar so it passes\" | Lowering the bar hides real failures. If the agent can't meet the bar, fix the agent, don't move the bar. |\n| \"This eval case is flaky, I'll skip it\" | Flaky evals reveal non-determinism in your agent. Fix with `temperature=0`, rubric-based metrics, or more specific instructions — don't delete the signal. |\n| \"I just need to fix the eval dataset, not the agent\" | If you're always adjusting expected outputs, your agent has a behavior problem. Fix the instructions or tool logic first. |\n| \"I'll iterate until every case I have passes\" | Nothing is left to detect overfitting to your own cases. See *Hold cases back* above. |\n\n## Choosing the Right Metrics\n\nPick built-in metrics by what you want to measure. Only `multi_turn_task_success`, `multi_turn_trajectory_quality`, and `multi_turn_tool_use_quality` accept multi-turn traces; every other built-in 400s on one. When no built-in fits, write a custom metric (see *Evaluation Configuration Schema* below).\n\n| Goal | Recommended built-in metrics |\n|------|------------------------------|\n| **Did the agent achieve the user's goal?** (catch-all for multi-turn agents) | `multi_turn_task_success` |\n| **Was the agent's reasoning path logical and efficient?** | `multi_turn_trajectory_quality` |\n| **Quality of tool \u002F function calling across turns** | `multi_turn_tool_use_quality` |\n| **Final response quality** (no ground-truth reference needed) | `final_response_quality` |\n| **Factual grounding** (catch hallucinated claims, e.g., RAG agents) | `hallucination`, or `grounding` when the case carries a `context` field |\n| **Safety policy compliance** | `safety` |\n| **Match against a golden answer** | `final_response_match` (needs `reference` on the case) |\n| **Different pass\u002Ffail criteria per case** | Put them on the case as `rubric_groups` and grade with a managed rubric metric. See `references\u002Fdataset_schema.md` (*Per-Case Rubrics*). |\n| **Domain-specific check no built-in covers** | Write a custom `LLMMetric` (LLM-judge) or `CodeExecutionMetric` (deterministic Python). See *Evaluation Configuration Schema* below. |\n\nRun `agents-cli eval metric list` to see all available built-ins. For full metric definitions and rubric details, see the [Gemini platform metric docs](https:\u002F\u002Fcloud.google.com\u002Fgemini-enterprise-agent-platform\u002Foptimize\u002Fevaluation\u002Fmanage-metrics) and `references\u002Fmetrics-guide.md`.\n\n---\n\n## What to fix when scores fail\n\nAfter `agents-cli eval run` completes, inspect the latest `artifacts\u002Fgrade_results\u002Fresults_\u003Ctimestamp>.json` (or open the `.html` file) for per-case scores and judge rationales, the input to every fix decision below.\n\n| Failure | What to change |\n|---------|---------------|\n| `multi_turn_task_success` low | The agent isn't completing the user's goal — fix orchestration, missing tool calls, premature termination, or wrong tool selection |\n| `multi_turn_trajectory_quality` low | The agent reaches the goal inefficiently or takes wrong steps — refine planning prompts, tighten instruction order, or remove redundant tool calls |\n| `multi_turn_tool_use_quality` low | Fix tool descriptions, parameter docstrings, or agent instructions for tool selection |\n| `final_response_quality` low | Read the auto-generated rubric verdicts; refine agent instructions to address the worst-scoring criterion (often clarity, completeness, or instruction-following) |\n| `hallucination` low | Tighten agent instructions to stay grounded in tool output; verify the tool actually returned the data the agent claimed |\n| `safety` low | Add safety guardrails to instructions; review the violating content category in the rubric verdict |\n| Agent calls wrong tools | Fix tool descriptions, agent instructions, or the model's tool-choice config (**ADK:** `tool_config`) |\n| Agent calls extra tools | Add strict stop instructions, or switch to `multi_turn_tool_use_quality` |\n\nAfter applying a fix, rerun `agents-cli eval run` and use `agents-cli eval compare \u003Cprev_results>.json \u003Cnew_results>.json` to confirm the fix improved the target metric without regressing others.\n\n---\n\n## Eval Commands\n\n`agents-cli eval \u003Csubcommand> --help` is the authoritative flag list; the examples below are the common invocations.\n\n### `eval run` (default)\n\nRuns the agent over the dataset and grades the traces in one command.\n\n```bash\n# Basic: dataset from tests\u002Feval\u002Fdatasets\u002F, results to artifacts\u002Fgrade_results\u002F,\n# metrics from tests\u002Feval\u002Feval_config.yaml\nagents-cli eval run\n\n# Advanced: pick the dataset, metrics, and output dir\nagents-cli eval run --dataset tests\u002Feval\u002Fdatasets\u002Fcustom.json --metrics final_response_quality,safety --output .\u002Fout\u002F\n\n# Gate: exit 1 if the 1-5 judge averages below 4\nagents-cli eval run --fail-under custom_response_quality=4\n```\n\n### `eval generate`\n\nRuns an agent over an evaluation dataset and writes traces to disk.\n\nBy default, runs the agent locally and records a trace per evaluation case. You can generate traces from an already-running agent by passing its HTTP endpoint and app name to `--url` and `--app-name`.\n\n> **ADK projects.** The built-in generator serves the agent over HTTP and drives it over ADK's `\u002Fapps\u002F...` and `\u002Frun_sse` routes — the same shape `--url` \u002F `--app-name` expect. What it launches depends on the project's language: Python uses the project's `fast_api_app.py` if it exists, else `adk api_server`; Go runs `go run ...`. Extensions for other frameworks replace `eval generate` with their own generator, which may not serve HTTP at all; `--url` and `--app-name` are then unsupported.\n\n```bash\n# Basic — uses tests\u002Feval\u002Fdatasets\u002F, writes to artifacts\u002Ftraces\u002F\nagents-cli eval generate\n\n# Advanced — custom dataset and output dir\nagents-cli eval generate --dataset tests\u002Feval\u002Fdatasets\u002Fcustom.json -o .\u002Fcustom_traces\u002F\n\n# Against a deployed agent (or one you started manually)\nagents-cli eval generate --url https:\u002F\u002Fmy-agent.run.app --app-name app\n\n# Live agent — stream each case over ADK's \u002Frun_live WebSocket\nagents-cli eval generate --mode adk_live\n```\n\n#### Evaluating Live agents\n\nLive agents run over a WebSocket, not `\u002Frun_sse`: add `--mode adk_live` to `generate` or `run`. The dataset, the traces, and grading are unchanged, and the audio reply is transcribed so the transcript is what gets graded. Two things must already be true or the socket connects and *then* fails mid-session: the agent uses a Live model (the scaffold default is not one), and on Gemini platform its region is pinned on the model rather than left to `GOOGLE_CLOUD_LOCATION`. Both, plus dataset authoring rules: `references\u002Flive-eval.md`.\n\n### `eval grade`\n\nScores traces (from `eval generate`, `eval dataset synthesize`, or hand-authored) against built-in or custom metrics. Writes timestamped `results_\u003CYYYYMMDD_HHMMSS>.json` (consumed by `eval compare`) and `.html` (open in a browser) into the output dir, and prints a summary table to the console. The output ends with a `RESULT: PASS|FAIL (...)` line; the command exits 1 when a metric's mean score is below its threshold (`thresholds` in the config, or `--fail-under`) or any case errored, so CI can gate on the exit code.\n\n```bash\n# Basic — defaults: traces from artifacts\u002Ftraces\u002F, results to artifacts\u002Fgrade_results\u002F,\n# metrics from tests\u002Feval\u002Feval_config.yaml's metrics_to_run\nagents-cli eval grade\n\n# Advanced 1 — grade traces from a non-default location (the canonical\n# pairing for `eval generate --output custom_traces\u002F`)\nagents-cli eval grade --traces custom_traces\u002F\n\n# Advanced 2: load metrics to run from a config file (YAML or JSON) on a specified trace file.\nagents-cli eval grade --traces .\u002Fartifacts\u002Ftraces\u002Ftrace_1.json --config tests\u002Feval\u002Feval_config.yaml\n\n# Advanced 3: dispatch rate, 15 metric computations per second by default. Lower it when the\n# judge model or the eval service rate-limits you, raise it when they have headroom.\nagents-cli eval grade --qps 5\n```\n\nSee *Evaluation Configuration Schema* below for the config file format.\n\n### `eval compare`\n\nDiffs two `results_*.json` files from an eval run. Run it after a fix to confirm the target metric improved without regressing others.\n\n```bash\nagents-cli eval compare baseline.json candidate.json\n```\n\n### `eval dataset synthesize`\n\n> **ADK projects.** It loads and runs the agent through ADK, so it is unavailable on other frameworks.\n\nGenerates user scenarios from your agent's tools and instructions, plays each against an LLM-backed user simulator, and writes graded-ready traces to `artifacts\u002Ftraces\u002F` (feed straight to `eval grade`, skip `eval generate`). Invocations, flags, and compatible metrics: `references\u002Fuser-simulation.md`.\n\n### Advanced commands\n\n`eval analyze` (cluster failure modes), `eval optimize` (GEPA prompt tuning), and `eval submit` \u002F `eval results` (managed cloud-side runs for CI or large datasets) are documented in `references\u002Fadvanced-commands.md`.\n\n---\n\n## Evaluation Dataset Format\n\nAn `EvaluationDataset` is a JSON file with an `eval_cases` array. Cases come in two shapes depending on how they're used:\n\n- **Inference input** (what you give to `eval generate`) — a single user prompt, or a multi-turn set of **user turns** (for live) \u002F a continuation ending in a user turn (for SSE). The agent runs and produces traces. Don't pre-author agent replies for live inference.\n- **Grading input** (what you give to `eval grade`) — a complete trace including the agent's responses and tool calls. Normally produced by `eval generate` or `eval dataset synthesize`; you don't write these by hand.\n\nSee `references\u002Fdataset_schema.md` for the full canonical schema, all field types, and common mistakes.\n\n### Inference input format\n\nTwo shapes are supported.\n\n**(a) Simple single-turn prompt** — what the scaffolded `tests\u002Feval\u002Fdatasets\u002Fbasic-dataset.json` uses. The agent runs from scratch.\n\n```json\n{\n  \"eval_cases\": [\n    {\n      \"eval_case_id\": \"greeting\",\n      \"prompt\": {\n        \"role\": \"user\",\n        \"parts\": [{\"text\": \"Hello, what can you help me with?\"}]\n      }\n    }\n  ]\n}\n```\n\n**(b) Multi-turn via `agent_data`** — the shape depends on the transport:\n\n- **Live (`--mode adk_live`):** author **user-only** turns; the agent generates every reply over one live session (authored agent turns are ignored, with a warning).\n- **SSE:** continuation form — prior turns are seeded as history and only the trailing user turn is answered.\n\nSee `references\u002Fdataset_schema.md` (*Multi-Turn \u002F Multi-Agent Dataset*) for the JSON shapes.\n\n### Grading input format (traces)\n\nA complete trace — agent responses plus `function_call` \u002F `function_response` parts — normally produced by `eval generate` \u002F `eval dataset synthesize` (you don't write these by hand). Authors are `\"user\"`, an agent ID from the `agents` map, or `\"tool\"`. See `references\u002Fdataset_schema.md` for the trace shape, multi-agent examples, and the full type reference.\n\n---\n\n## Evaluation Configuration Schema\n\n`agents-cli eval run --config \u003Cpath>` (and `eval grade --config \u003Cpath>`) accepts a single configuration file in either **YAML** (`.yaml` \u002F `.yml`) or **JSON** (`.json`). The file declares two parts:\n\n- `metrics_to_run`: the **selection list** of metric names to execute on this run. A name resolves to a `custom_metrics` entry when one matches, otherwise to the built-in metric of that name.\n- `custom_metrics` — a **definition pool** of custom metrics available to this project. Defining a metric here does **not** run it; it must also appear in `metrics_to_run` (or be passed via `--metrics name1,name2` on the CLI, which is equivalent to overriding `metrics_to_run` for that invocation).\n- `thresholds` (optional) — the minimum **mean score** per metric, in that metric's own scale (built-ins score 0-1, the scaffolded `custom_response_quality` judge 1-5). A run with a metric below its threshold exits 1 and lists the cases scoring below it; `pass_rate` in the results is then the share of cases at or above it, unless the eval service already reported one. `--fail-under METRIC=SCORE` (repeatable, one metric each) overrides them per run. Errored cases fail the run with or without thresholds.\n\n**Minimal example (YAML preferred — human-readable, no JSON escaping for prompts and Python):**\n\n```yaml\nmetrics_to_run:\n  - multi_turn_task_success     # built-in\n  - example_llm_metric          # selected from custom_metrics pool below\n  - agent_turn_count            # selected from custom_metrics pool below\n\ncustom_metrics:\n  - name: example_llm_metric\n    prompt_template: |\n      Rate the agent's response 1-5 for helpfulness and accuracy.\n      Prompt: {prompt}\n      Final response: {response}\n      Full trace (for tool-call and reasoning context): {agent_data}\n      Return JSON: {\"score\": \u003C1|2|3|4|5>, \"explanation\": \"\u003Creason>\"}\n\n  - name: agent_turn_count\n    custom_function: |\n      def evaluate(instance):\n          turns = (instance.get(\"agent_data\") or {}).get(\"turns\", [])\n          return {'score': len(turns)}\n\nthresholds:\n  multi_turn_task_success: 0.8\n  example_llm_metric: 4\n```\n\nJSON is also accepted (same field names, with `prompt_template` and `custom_function` as escaped strings) — but **always prefer YAML** for human-readable configs.\n\nDispatch by field: `custom_function` → Python metric; `prompt_template` → `LLMMetric` (LLM-as-judge); neither, on a built-in name → parameterizes that built-in (e.g. `metric_spec_parameters.rubric_group_key`). Field reference: `references\u002Fmetrics-guide.md`.\n\n**Agent trace field model.** For datasets produced by `agents-cli eval generate` (or `eval dataset synthesize`), each eval case exposes three standard fields to a metric:\n\n- `{prompt}` — the user message (or first user turn).\n- `{response}` — the agent's final text response, extracted from the last text-bearing event. In `custom_function` callbacks this is `instance['response']` with shape `{\"role\": \"model\", \"parts\": [{\"text\": \"...\"}]}`.\n- `{agent_data}` — the full structured `turns`\u002F`events` trace, useful when the judge needs to reason about tool calls or intermediate reasoning.\n\n`reference`, `context`, and `rubric_groups` are yours to author on the case: `eval generate` carries them onto the trace but never invents them, so `{reference}` \u002F `{context}` resolve only where you wrote them. `rubric_groups` is not a placeholder at all: managed rubric metrics read it off the case, and a `custom_function` sees `instance['rubric_groups']`. See `references\u002Fdataset_schema.md` (*Per-Case Rubrics*).\n\nCode-based metrics default to **local in-process execution** (no GCP project or region required, but the `evaluate(instance)` function runs with the CLI's privileges). Set `execution: \"remote\"` on the metric to run it server-side in the `CodeExecutionMetric` sandbox in Gemini platform instead — that path requires a configured GCP project + region.\n\n---\n\n\n## Common Gotchas\n\n### Use Rubric-Based Tool Evaluation instead of Hardcoded Sequences\n\nEvaluating agent tool usage using strict sequence matching is fragile because agents may call helper tools (like searches or geocoding) in different orders or perform extra proactive steps.\n\nInstead, use **`multi_turn_tool_use_quality`** \u002F **`multi_turn_trajectory_quality`**. These metrics automatically generate content-based and intent-based adaptive rubrics, assessing technical correctness and technical sequence logic semantically using an LLM judge rather than forcing a rigid match.\n\n### App name must match directory name\n\n> **ADK projects.**\n\nThe `App` object's `name` parameter MUST match the directory containing your agent:\n\n```python\n# CORRECT - matches the \"app\" directory\napp = App(root_agent=root_agent, name=\"app\")\n\n# WRONG - causes \"Session not found\" errors\napp = App(root_agent=root_agent, name=\"flight_booking_assistant\")\n```\n\n### Gemini platform eval region\n\n`eval run`, `eval grade`, and `eval submit` **default to the `global` endpoint**. They don't inherit the manifest `region` (the eval services support only a subset of regions), and `eval analyze` is `global`-only. Override these per run with `--region \u003CREGION>` (e.g. data residency); the service rejects an unsupported one:\n\n```\n400 FAILED_PRECONDITION: Unsupported region for Vertex Evaluation Service: \u003Cregion>\n```\n\n`eval generate` (without the `--url` flag) and `eval dataset synthesize` run your agent locally, so they honor the agent's own `.env` — notably `GOOGLE_CLOUD_LOCATION`, which selects the model endpoint **when the agent uses Gemini platform** (`GOOGLE_GENAI_USE_ENTERPRISE=true`); it's unused with a `GEMINI_API_KEY` (AI Studio). They take **no** `--region` and never override your `.env` with the manifest `region`; change the model region by editing `.env` — or, better for a single agent, pin it in code with `Gemini(model=…, client_kwargs={\"location\": …})`, which beats the env var and leaves it free for everything else. One caveat for `synthesize`: its scenario-generation step is a **server-side** eval call at `GOOGLE_CLOUD_LOCATION`, so keep that an eval-supported region (`global` by default) even though the agent itself could run elsewhere.\n\n**No eval region fits your data-residency rules?** Fall back to **local custom metrics** — a `custom_metrics` entry with a `custom_function` (`execution: local`, the default) grades in-process with no GCP region required. You lose the managed built-in metrics, but your `custom_function` can still call an LLM judge in a compliant region itself — so LLM-as-judge grading stays available anywhere.\n\n### The `before_agent_callback` Pattern (State Initialization)\n\n> **ADK projects.**\n\nAlways use a callback to initialize session state variables used in your instruction template. This prevents `KeyError` crashes on the first turn:\n\n```python\nasync def initialize_state(callback_context: CallbackContext) -> None:\n    state = callback_context.state\n    if \"user_preferences\" not in state:\n        state[\"user_preferences\"] = {}\n\nroot_agent = Agent(\n    name=\"my_agent\",\n    before_agent_callback=initialize_state,\n    instruction=\"Based on preferences: {user_preferences}...\",\n)\n```\n\n### Model thinking mode may bypass tools\n\nModels with \"thinking\" enabled may skip tool calls. Force tool usage through the model's tool-choice config (**ADK:** `tool_config` with `mode=\"ANY\"`), or switch to a non-thinking model for predictable tool calling.\n\n---\n\n## Common Eval Failure Causes\n\n| Symptom | Cause | Fix |\n|---------|-------|-----|\n| Score fluctuates between runs | Non-deterministic model | Set `temperature=0` or use rubric-based eval with multiple samples |\n| LLM judge ignores image\u002Faudio in eval | `get_text_from_content()` skips non-text parts | Use custom metric with vision-capable judge (see `references\u002Fmultimodal-eval.md`) |\n\n---\n\n## Proving Your Work\n\nDon't assert that eval passes — show the evidence. Concrete output prevents false confidence and catches issues early.\n\n- **After running eval:** Paste the scores table output so the user can see exactly what passed and failed.\n- **After fixing a failure:** Show before\u002Fafter scores for the specific case you fixed, and confirm no other cases regressed.\n- **Before deploy:** Rerun `agents-cli eval run` and show the scores for every case, not just the one you fixed, plus the final `RESULT:` line. The exit code gates only on the `thresholds` you set (and errored cases), so confirm they match the bar you agreed with the user.\n\n---\n\n## Related Skills\n\n- `\u002Fgoogle-agents-cli-workflow` — Development workflow and the spec-driven build-evaluate-deploy lifecycle\n- `\u002Fgoogle-agents-cli-adk-code` — ADK API quick reference for writing agent code (ADK projects only)\n- `\u002Fgoogle-agents-cli-scaffold` — Project creation and enhancement with `agents-cli scaffold create` \u002F `scaffold enhance`\n- `\u002Fgoogle-agents-cli-deploy` — Deployment targets, CI\u002FCD pipelines, and production workflows\n- `\u002Fgoogle-agents-cli-observability` — Cloud Trace, logging, and monitoring for debugging agent behavior\n",{"name":5,"metadata":53,"description":59},{"author":12,"license":15,"version":54,"requires":55},"1.14.0",{"bins":56,"install":58},[57],"agents-cli","uv tool install google-agents-cli","This skill should be used when the user wants to \"run an evaluation\", \"evaluate my agent\", \"evaluate my ADK agent\", \"write an eval dataset\", \"analyze eval failures\", \"compare eval results\", \"optimize agent\", or needs guidance on the Gemini platform eval methodology and the Quality Flywheel. Covers eval metrics, dataset schema, LLM-as-judge scoring, and common failure causes. Applies to any agents-cli project, whatever framework the agent is written in. Do NOT use for agent API code patterns (ADK: use google-agents-cli-adk-code), deployment (use google-agents-cli-deploy), or project scaffolding (use google-agents-cli-scaffold).\n","main","6eedbdc2f44c388cbfc5f305eb83b9924cfefebb","src\u002Fgoogle\u002Fagents\u002Fcli\u002Fskills\u002Fdata\u002Fgoogle-agents-cli-eval",731,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fagents-cli","https:\u002F\u002Fgoogle.github.io\u002Fagents-cli\u002F","https:\u002F\u002Fgithub.com\u002Fgoogle","https:\u002F\u002Fopensource.google\u002F","Google ❤️ Open Source","2026-11-12 23:04:58",{"source":8,"skillName":5,"commands":71,"prompt":81,"zipAvailable":82},[72,75,78],{"tool":73,"command":74},"npx","npx skills add google\u002Fagents-cli --skill google-agents-cli-eval",{"tool":76,"command":77},"bunx","bunx skills add google\u002Fagents-cli --skill google-agents-cli-eval",{"tool":79,"command":80},"pnpm","pnpm dlx skills add google\u002Fagents-cli --skill google-agents-cli-eval","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fagents-cli\u002Fblob\u002Fmain\u002Fsrc\u002Fgoogle\u002Fagents\u002Fcli\u002Fskills\u002Fdata\u002Fgoogle-agents-cli-eval\u002FSKILL.md",true,[84,90,94,98,102,106,110],{"path":85,"resourceType":86,"resourceTypeDesc":87,"sizeBytes":88,"sourceHtmlUrl":89},"references\u002Fadvanced-commands.md","REFERENCE","参考文档",2395,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fagents-cli\u002Fblob\u002Fmain\u002Fsrc\u002Fgoogle\u002Fagents\u002Fcli\u002Fskills\u002Fdata\u002Fgoogle-agents-cli-eval\u002Freferences\u002Fadvanced-commands.md",{"path":91,"resourceType":86,"resourceTypeDesc":87,"sizeBytes":92,"sourceHtmlUrl":93},"references\u002Fbuiltin-tools-eval.md",6194,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fagents-cli\u002Fblob\u002Fmain\u002Fsrc\u002Fgoogle\u002Fagents\u002Fcli\u002Fskills\u002Fdata\u002Fgoogle-agents-cli-eval\u002Freferences\u002Fbuiltin-tools-eval.md",{"path":95,"resourceType":86,"resourceTypeDesc":87,"sizeBytes":96,"sourceHtmlUrl":97},"references\u002Fdataset_schema.md",8540,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fagents-cli\u002Fblob\u002Fmain\u002Fsrc\u002Fgoogle\u002Fagents\u002Fcli\u002Fskills\u002Fdata\u002Fgoogle-agents-cli-eval\u002Freferences\u002Fdataset_schema.md",{"path":99,"resourceType":86,"resourceTypeDesc":87,"sizeBytes":100,"sourceHtmlUrl":101},"references\u002Flive-eval.md",2710,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fagents-cli\u002Fblob\u002Fmain\u002Fsrc\u002Fgoogle\u002Fagents\u002Fcli\u002Fskills\u002Fdata\u002Fgoogle-agents-cli-eval\u002Freferences\u002Flive-eval.md",{"path":103,"resourceType":86,"resourceTypeDesc":87,"sizeBytes":104,"sourceHtmlUrl":105},"references\u002Fmetrics-guide.md",10612,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fagents-cli\u002Fblob\u002Fmain\u002Fsrc\u002Fgoogle\u002Fagents\u002Fcli\u002Fskills\u002Fdata\u002Fgoogle-agents-cli-eval\u002Freferences\u002Fmetrics-guide.md",{"path":107,"resourceType":86,"resourceTypeDesc":87,"sizeBytes":108,"sourceHtmlUrl":109},"references\u002Fmultimodal-eval.md",5432,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fagents-cli\u002Fblob\u002Fmain\u002Fsrc\u002Fgoogle\u002Fagents\u002Fcli\u002Fskills\u002Fdata\u002Fgoogle-agents-cli-eval\u002Freferences\u002Fmultimodal-eval.md",{"path":111,"resourceType":86,"resourceTypeDesc":87,"sizeBytes":112,"sourceHtmlUrl":113},"references\u002Fuser-simulation.md",6415,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fagents-cli\u002Fblob\u002Fmain\u002Fsrc\u002Fgoogle\u002Fagents\u002Fcli\u002Fskills\u002Fdata\u002Fgoogle-agents-cli-eval\u002Freferences\u002Fuser-simulation.md",[115,140,161,184,207,231],{"skillNo":116,"name":117,"nameZh":6,"description":118,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":119,"categories":120,"keywords":126,"publishedAt":48},"01M6990ZGJV6CHR6CZK81S58CN","google-agents-cli-workflow","This skill should be used when the user wants to \"develop an agent\", \"build an agent using ADK\", \"run the agent locally\", \"debug agent code\", \"test an agent\", \"deploy an agent\", \"publish an agent\", \"monitor an agent\", or needs the ADK (Agent Development Kit) development lifecycle and coding guidelines. Entrypoint for building ADK agents. Always active — provides the full workflow (scaffold, build, evaluate, deploy, publish, observe), code preservation rules, model selection guidance, and troubleshooting steps for ADK or any agent development.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fagents-cli\u002Fblob\u002Fmain\u002Fsrc\u002Fgoogle\u002Fagents\u002Fcli\u002Fskills\u002Fdata\u002Fgoogle-agents-cli-workflow\u002FSKILL.md",[121,124,125],{"categoryNo":122,"categoryName":123,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000011","运维与部署",{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},{"categoryNo":24,"categoryName":25,"taxonomyType":26,"taxonomyTypeDesc":27},[9,127,29,128,47,129,130,131,132,133,134,135,136,43,40,137,138,34,139,33],"google agents cli workflow","workflow","agent development kit","agent development","development lifecycle","model","observe","selection","model selection","kit","debug agent","publish","active",{"skillNo":141,"name":142,"nameZh":6,"description":143,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":144,"categories":145,"keywords":148,"publishedAt":48},"01M6990XH0S4MNS7YTDAMNAAEW","google-agents-cli-scaffold","This skill should be used when the user wants to \"create an agent project\", \"start a new ADK project\", \"build me a new agent\", \"add CI\u002FCD to my project\", \"add deployment\", \"enhance my project\", or \"upgrade my project\". Part of the agents-cli skills suite. Covers `agents-cli scaffold create`, `scaffold enhance`, and `scaffold upgrade` commands, template options, deployment targets, and the prototype-first workflow. Do NOT use for writing agent code (ADK projects: use google-agents-cli-adk-code) or deployment operations (use google-agents-cli-deploy).","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fagents-cli\u002Fblob\u002Fmain\u002Fsrc\u002Fgoogle\u002Fagents\u002Fcli\u002Fskills\u002Fdata\u002Fgoogle-agents-cli-scaffold\u002FSKILL.md",[146,147],{"categoryNo":122,"categoryName":123,"taxonomyType":21,"taxonomyTypeDesc":22},{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},[149,9,29,150,47,151,152,153,45,154,43,155,128,40,156,157,34,158,159,160],"google agents cli scaffold","scaffold","writing","projects","new agent","ci","upgrade","create","me","options","template","CI\u002FCD",{"skillNo":162,"name":163,"nameZh":6,"description":164,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":165,"categories":166,"keywords":169,"publishedAt":48},"01M6990VP634A9G6052KQCZNYJ","google-agents-cli-publish","This skill should be used when the user wants to \"publish an agent\", \"publish my ADK agent\", \"register an agent with Gemini\", \"publish to Gemini\" (formerly Gemini Enterprise), or needs guidance on the agents-cli publish gemini command. Also use when the user wants to \"manage agents in Agent Registry\", \"list\u002Fupdate\u002Fdelete registered agents\", or \"register an MCP server\". Covers ADK vs A2A registration modes, programmatic and interactive usage, flag reference, auto-detection from deployment metadata, Agent Registry fleet management, and troubleshooting. Part of the agents-cli skills suite. Do NOT use for deployment (use google-agents-cli-deploy).","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fagents-cli\u002Fblob\u002Fmain\u002Fsrc\u002Fgoogle\u002Fagents\u002Fcli\u002Fskills\u002Fdata\u002Fgoogle-agents-cli-publish\u002FSKILL.md",[167,168],{"categoryNo":122,"categoryName":123,"taxonomyType":21,"taxonomyTypeDesc":22},{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},[9,29,170,138,47,171,172,43,173,174,175,176,177,34,178,179,180,181,182,183],"google agents cli publish","MCP","programmatic","auto","mcp server","agent registry","server","delete","command","update","install","suite","fleet","registry",{"skillNo":185,"name":186,"nameZh":6,"description":187,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":188,"categories":189,"keywords":198,"publishedAt":48},"01M6990SFDY0JKXYP921BKRJYE","google-agents-cli-observability","This skill should be used when the user wants to \"set up tracing\", \"monitor my agent\", \"configure logging\", \"add observability\", \"debug production traffic\", or needs guidance on monitoring deployed agents, including ADK (Agent Development Kit) agents. Covers Cloud Trace, prompt-response logging, BigQuery Agent Analytics, third-party integrations (AgentOps, Phoenix, MLflow, etc.), and troubleshooting. Part of the agents-cli skills suite. Do NOT use for deployment setup (use google-agents-cli-deploy) or API code patterns (use google-agents-cli-adk-code).","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fagents-cli\u002Fblob\u002Fmain\u002Fsrc\u002Fgoogle\u002Fagents\u002Fcli\u002Fskills\u002Fdata\u002Fgoogle-agents-cli-observability\u002FSKILL.md",[190,193,194,195],{"categoryNo":191,"categoryName":192,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000013","分析与可视化",{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},{"categoryNo":24,"categoryName":25,"taxonomyType":26,"taxonomyTypeDesc":27},{"categoryNo":196,"categoryName":197,"taxonomyType":26,"taxonomyTypeDesc":27},"01JBSEEDCAT000000000000033","数据分析师",[9,29,199,200,47,129,130,201,202,136,43,203,204,41,40,38,205,34,180,206],"google agents cli observability","observability","SET","up","traffic","configure","monitoring","mlflow",{"skillNo":208,"name":209,"nameZh":6,"description":210,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":211,"categories":212,"keywords":219,"publishedAt":48},"01M6990MAHH0X6XCNVB74Q67P6","google-agents-cli-deploy","This skill should be used when the user wants to \"deploy an agent\", \"deploy my ADK agent\", \"set up CI\u002FCD\", \"configure secrets\", \"troubleshoot a deployment\", or needs guidance on Agent Runtime, Cloud Run, or GKE deployment targets, or binding an agent to an Agent Gateway. Covers deployment workflows, service accounts, rollback, and production infrastructure. Applies to any framework agents-cli deploys (ADK, LangChain, ...). Part of the agents-cli skills suite. Do NOT use for agent API code patterns (ADK: use google-agents-cli-adk-code), evaluation (use google-agents-cli-eval), or project scaffolding (use google-agents-cli-scaffold).","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fagents-cli\u002Fblob\u002Fmain\u002Fsrc\u002Fgoogle\u002Fagents\u002Fcli\u002Fskills\u002Fdata\u002Fgoogle-agents-cli-deploy\u002FSKILL.md",[213,216,217,218],{"categoryNo":214,"categoryName":215,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000010","后端与接口",{"categoryNo":122,"categoryName":123,"taxonomyType":21,"taxonomyTypeDesc":22},{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},{"categoryNo":24,"categoryName":25,"taxonomyType":26,"taxonomyTypeDesc":27},[9,220,29,221,47,45,154,31,43,222,41,223,33,224,225,226,227,228,229,230],"google agents cli deploy","deploy","workflows","gke","Evaluation","langchain","infrastructure","do","cloud run","for","deployment",{"skillNo":232,"name":233,"nameZh":6,"description":234,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":235,"categories":236,"keywords":239,"publishedAt":48},"01M6990HY6R7MS0J02P09ZTT89","google-agents-cli-adk-code","This skill should be used when the user wants to \"write agent code\", \"build an agent with ADK\", \"add a tool\", \"create a callback\", \"define an agent\", \"use state management\" — in a project that needs ADK (Agent Development Kit) API patterns and code examples. It provides a quick reference for agent types, tool definitions, orchestration patterns, callbacks, state management, the graph Workflow API, and reference recipes to study. Do NOT use for scaffolding (use google-agents-cli-scaffold) or deployment (use google-agents-cli-deploy).","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fagents-cli\u002Fblob\u002Fmain\u002Fsrc\u002Fgoogle\u002Fagents\u002Fcli\u002Fskills\u002Fdata\u002Fgoogle-agents-cli-adk-code\u002FSKILL.md",[237,238],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},{"categoryNo":24,"categoryName":25,"taxonomyType":26,"taxonomyTypeDesc":27},[9,47,29,240,40,129,130,241,242,243,45,44,136,43,244,128,38,156,34,32],"google agents cli adk code","quick","quick reference","types","orchestration"]