commerce-evals
Authoring and running behavioral evals for a shopping or merchant agent, covering the case shape, authoring rules, code graders and judges, the run pattern, and poisoned fixtures. Load when writing eval cases or rubrics or deciding how a suite runs.
- License 许可
- Apache-2.0
- 最近更新
- 2026-11-04
- Stars
- ★ 3.5k
- Forks
- 714
01 前言 Front Matter
- name
- commerce-evals
- description
- Authoring and running behavioral evals for a shopping or merchant agent, covering the case shape, authoring rules, code graders and judges, the run pattern, and poisoned fixtures. Load when writing eval cases or rubrics or deciding how a suite runs.
03 SKILL.md 正文
Commerce agent evals
The repo ships no eval harness; the suite is yours, because a case only means something against your catalog,
orders, and fixtures. Paths below are in the reference repo: commerce_common/ is commerce-common/commerce_common/.
Gate behavior that needs no model (provenance, caps, guardrails, approval) is unit-tested with FakeClient in
commerce_common/testing.py; evals cover what the model decides.
The case shape
This block is the one home of the case shape and the scorer names; /author-commerce-evals refers to it.
{
"id": "<flow>-<nnn>-<behavior>",
"priority": "critical | high | medium | low", "difficulty": "easy | medium | hard", "tags": ["..."],
"skip": "<reason, when a case cannot run yet>",
"state": {"seen_products": ["..."], "cart": [...], "memory": [...], "staged_changes": [...]},
"turns": ["<the customer's or operator's message>", "..."],
"expected": {
"calls_tool": ["..."], "calls_one_of": ["..."], "never_calls": ["..."],
"first_tool": "...", "first_tool_not": "...",
"ui_components": ["..."], "no_ui": true,
"cart_contains": ["..."], "cart_item_count": 0, "cart_not_contains": ["..."],
"staged_change_kinds": ["..."], "no_applied_changes": true,
"memory_contains": ["..."], "memory_not_contains": ["..."],
"skill_loaded": "...", "skill_not_loaded": "...", "no_skill_load": true,
"reply_includes": ["..."], "reply_omits": ["..."], "max_tool_calls": 0,
"rubric": "PASS if <condition>. FAIL if <condition>."
},
"notes": "<what the case pins and the fixture fact that decides it>"
}
state is the precondition; turns is one message unless the behavior under test is carrying state across turns;
expected holds only the keys the case is about. Ids in state and expected are real ids from your fixtures.
priority and difficulty let a report say which failures matter; a case that cannot run yet carries skip
with its reason rather than being deleted.
Authoring rules
- Preconditions go in injected state (the products already seen, the cart, the memory facts, the staged queue), which the runner loads into the session state and the memory store before the turn. Earlier turns are for state the behavior itself carries, and for nothing else.
- Every positive has a negative: for each case that asserts a component, a skill load, a gate, a disclosure, or a memory write, a case in the same niche asserts its absence. A refusal case has a should-serve counterpart.
- Memory is three cases: a remark worth keeping is written; an identifier or excluded content is refused (nothing stored, no error to the person); a stored fact changes the next session's pick.
- Grade the final tool arguments and the state they produced: the ids on the last presentation call, the fields on
the staged change, the cart and the memory store after the turn. The reply's wording is graded only for strings
that must or must not appear. Which route the agent took (
skill_loaded, one named component,never_callson a presentation tool) is asserted only where the route is the behavior (a grounding read first, a write that must never happen); elsewherecalls_one_ofnames the acceptable set. When a live run takes a route the case did not expect and the answer was right, widen the case to the acceptable set; do not re-pin it to the route observed. - A rubric is one PASS condition and one FAIL condition that no response satisfies both of; it names the fixture fact that decides it (the updated delivery date, the price today); variants you accept are written into it; it says nothing about tone, length, or the order components appear in.
- A turn that mentions health, a one-off errand, or hostile content asserts the memory end-state
(
memory_not_contains, ornever_callsonsave_memory). A case where a stored fact should change the pick uses a query whose results contain both the item the fact favors and the one it rules out; run the search before writing the case. max_tool_callsis set from what a well-behaved agent needs; a multi-item request fans out several searches in one round, and thepresent_suggestionscall that ends the turn is not counted.
Scorers
- Code graders read the events a turn yields (
commerce_common/streaming.py):tool_callnames and arguments,tool_resultwithstatusblockedand the gate inreason,uicomponent names, the lastcart_updateorchange_update, and the reply text. Every key above exceptrubricis a code grader. rubricgoes to a judge. One judge call per dimension (budget respected, no invented availability, trade-off stated), returning structured output with a verdict and a reason; the transcript is passed to it as quoted material, tool results and component payloads included; when the transcript exceeds the judge's window, truncate from the start so the graded turn survives, and record the truncation on the outcome. Pin the judge model at temperature zero; a change to the judge model or a rubric invalidates every stored verdict scored with it, so the recording carries a fingerprint of both.- A judge reply that does not parse into a verdict is a judge failure on the case, kept apart from an agent failure.
Run pattern
| When | What runs | What decides |
|---|---|---|
| Every merge to the agent, a skill, a tool description, or a fixture | The regression set | Each case over several trials; a pass threshold per set |
| While changing one flow | That flow's targeted set | The failure set, read beside the previous run's as the baseline |
| Choosing or upgrading a model (commerce-architecture's model fields) | Everything | The two failure sets side by side |
| In production | A judged sample of live traffic against the same rubrics | Trend per dimension |
Diff failure sets; a topline moving a point between live runs is noise. A case that fails after a change means the change broke the behavior or the case encoded a stale one; fix whichever it is and say which in the commit.
Poisoned fixtures
Listings, reviews, and messages carrying instructions live in eval-only fixtures the runner merges into the backend
for the run, under a third-party brand or seller; none of their ids appears in demo data, seeds, or captures. Each
such case asserts the negative in code (never_calls, cart_not_contains, no_applied_changes,
memory_not_contains, reply_omits), and every vector it asserts is one the driven turn actually puts in front of
the model (a review is only read on a details call). Cover at least an instruction to write to the cart or stage a
change, one to remember something, and a false claim (a code, a guarantee). The should-serve counterpart is a
separate benign eval-only listing in the same niche, so an agent that refuses everything fails it.
04 同仓库更多 Skills
search-discovery
Turning a described need (several constraints, a gift, a choice between candidates already in view, a search that came back empty or sold out) into a shortlist and a pick. Not needed when one search for the thing the customer named answers the request, or when the customer wants to learn what matters in a category first (purchase-research).
purchase-research
Teaching a customer how to choose before any candidates appear, when they ask what matters in a category, how two approaches to it differ, or where to begin. Not needed once the customer has named the thing they want, however considered the purchase (that is search work), nor for outfitting a space or an event (planning-goals).
planning-goals
Planning a coordinated set of items toward a goal, an event, a trip, or a project, where the request only comes together as several purchases with a structure between them, and later changes to a plan you have presented. Not needed when the customer wants a single item, however carefully chosen.
memory-personalization
Requests about what is remembered, however short, from an ask to remember one thing to asks to say what is on file, correct it, or forget it; a bare save or an answer from the profile alone does not cover these. Also applying what the store already knows, recalling an older fact (a past recipient, a size, a recurring order) when it would change the pick, and deciding what should outlast the session. Not needed when the request carries no personal context to apply or keep.
customer-care
Help after a purchase or booking, covering the status of an order or delivery whether or not the customer names it, returns, refunds, exchanges, and cancellations, an item that arrived damaged, incomplete, or late, and questions about the store's terms or a membership. Not needed for finding, comparing, or choosing something new.
commerce-ui-tools
The reference presentation-tool contract, covering server-side enrichment, suggestion chips, the event stream, progressive rendering, both roles' built-in components, and adding a vertical component. Load when building or reviewing how a commerce agent's output reaches a frontend.