[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6M93X858CBMM4Q0BFBXQDTS":3,"skill-same:01M6M93X858CBMM4Q0BFBXQDTS":94},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":23,"publishedAt":45,"linkOnly":46,"markdownContent":47,"markdownBody":48,"frontMatter":49,"refName":52,"commitSha":53,"skillDirectory":54,"forksCount":55,"repositoryHtmlUrl":56,"repositoryHomepageUrl":57,"ownerHtmlUrl":58,"ownerBlogUrl":59,"ownerDescription":60,"updatedAt":61,"install":62,"resources":75},"01M6M93X858CBMM4Q0BFBXQDTS","suede-ai-eval",null,"Suede AI Eval Skills. Use when asked to \"write evals for this AI feature\" or assess LLM, RAG, agent, classifier or generated-media behavior: AI-SPEC, failure rubric, executable cases, coverage and acceptance evidence. NOT FOR: implementation review (suede-code), wiring existing evals into CI (suede-ci-gate), or coordinating builder lanes (suede-agent-teams).","hashgraph-online\u002Fawesome-codex-plugins","hashgraph-online","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F160371962?v=4","01M6CNB6NRJVNNGN14TR01Q6D3","HOL (Hashgraph Online)","01M6JT1HSKKG9V68KQAQGBA7XP",1575,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FJasonColapietro\u002Fsuede-creator-skills\u002Fskills\u002Fsuede-ai-eval\u002FSKILL.md",[18],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44],"suede ai eval","eval","rubric","when","wiring","agent-teams","implementation review","feature","classifier","builder","evals","evidence","review","generated","coverage","write","Media","ci","acceptance","code","rag","2026-11-10 10:12:30",false,"---\nname: suede-ai-eval\ndescription: \"Suede AI Eval Skills. Use when asked to \\\"write evals for this AI feature\\\" or assess LLM, RAG, agent, classifier or generated-media behavior: AI-SPEC, failure rubric, executable cases, coverage and acceptance evidence. NOT FOR: implementation review (suede-code), wiring existing evals into CI (suede-ci-gate), or coordinating builder lanes (suede-agent-teams).\"\nmetadata:\n  display-name: \"Suede AI Eval Skills\"\n---\n\n# Suede AI Eval Skills\n\nRead [the worked delivery contract](references\u002Fworked-contract.md) for a concrete\ninput, expected artifact, failure response and validation example.\n\n## Compatibility\n\nRequires the AI surface contract or source and test data; a project runtime\u002Feval runner for execution. Model\u002FAPI access is needed only for actual model runs, with existing authorization and cost scope.\n\n## Gate policy: advisory, not blocking\n\nEvery claim-verification step, check, quality gate, and ship verdict in this\nskill is a **recommendation to the user, not a control on the agent**. This\npolicy governs every gate, check, verdict, and \"do not ship \u002F publish \u002F\nproceed\" line elsewhere in this skill:\n\n- Run every check and report the results honestly. Verdicts (`ship`,\n  `ship-with-caveats`, `hold`, letter grades, BLOCKED or OPEN items) are\n  advice attached to the work, not orders that change it.\n- Never block, delay, skip, rewrite, or refuse the action the user asked for\n  because a check failed or a gate said hold. Complete the requested action\n  as asked, and deliver the gate output alongside it as a clearly labeled\n  recommendation.\n- A failed gate changes what you report, never what you do.\n- Single exception: if a finding is extremely risky (data loss, security or\n  credential exposure, legal or rights violations, payment mistakes, or\n  irreversible public damage), pause, tell the user exactly what the risk is\n  and what the options are, and let them pick. Their choice is final.\n\n\nMake AI behavior testable before it becomes a vague product promise. **No eval plan, no `ship` recommendation: for an AI feature without one, the recommended verdict stays below `ship`: report that gap and let the user decide.**\n\nThe deliverable is an eval plan or coverage audit, not a model benchmark leaderboard. Keep it grounded in the actual product surface, user promise, data sources, prompts, tools, logs, tests, and failure modes available now.\n\n## Hard Gates\n\n- No AI-SPEC → no eval plan. Write the one-paragraph spec first; cases written without a spec test nothing.\n- No eval plan → no `ship` recommendation. Do not recommend `ship` or `ship-with-caveats` for an AI feature that lacks a failure-mode map and eval cases; name the gap and leave the ship decision with the user.\n- A failure mode without an eval case, an owner, and a gate is uncovered, regardless of how unlikely it feels.\n- A live surface that was never sampled gets the output stamped `source-only`; do not present source-only review as runtime evidence.\n- A model grading its own output is not evidence. LLM-as-judge scores count only after spot-checked agreement with a human-reviewed sample.\n\n## Source Truth\n\nInspect the current target before writing the eval. Do not evaluate from memory or product copy alone.\n\nRead or verify:\n\n- repo, branch, remote, dirty state, local instructions, and touched files;\n- the AI surface: route, API, worker, prompt, system message, tool call, model config, retrieval path, classifier, agent loop, generated media path, or recommendation logic;\n- user-facing promise, allowed claims, forbidden claims, safety boundaries, fallback behavior, and support path;\n- input data, retrieval corpus, schemas, tool contracts, metadata, logs, telemetry, and persisted outputs;\n- existing tests, fixtures, eval scripts, prompt snapshots, golden examples, analytics, bug reports, screenshots, or live\u002FAPI readbacks.\n\nWhen the surface is already live, sample real behavior with safe inputs and record exact commands or URLs. When live checks are not appropriate, mark the eval as source-only and name the missing runtime evidence.\n\n## Workflow\n\n1. **Define the AI-SPEC.** State the AI job in one paragraph: user, trigger, input, output, allowed sources, disallowed behavior, fallback, latency\u002Fcost expectation, and success signal.\n2. **Map the failure modes.** List the ways the AI can harm the user, product truth, rights\u002Fprovenance, security, privacy, brand trust, cost, or workflow completion.\n3. **Build the rubric.** Score each failure mode with severity, likelihood, detectability, owner, gate, and required evidence.\n4. **Write eval cases.** Produce concrete pass\u002Ffail cases with inputs, setup data, expected output traits, forbidden output traits, and the reason the case exists.\n5. **Set acceptance gates.** Decide what blocks ship, what allows ship-with-caveats, and what can become follow-up work.\n6. **Audit coverage.** Compare existing tests, logs, metrics, and manual checks against the failure-mode map. Score coverage and infrastructure using the method under Tooling and Infrastructure below. Name every uncovered high-risk behavior regardless of the numeric score.\n7. **Return the artifact.** Give the AI-SPEC, rubric, eval table, coverage gaps, required tests, and next implementation step. Name the exact command that runs the cases and its expected exit status (the repo's own eval script if one exists, otherwise the tool's invocation, e.g. `npx promptfoo eval -c \u003Cconfig>`) and record the run's pass\u002Ffail counts under \"Commands or evidence checked\". An eval plan with no runnable command is a document, not coverage, and suede-ci-gate cannot wire it into CI without that string.\n\n## Eval Dimensions By System Type\n\nStart the failure-mode map from the canonical dimensions for the surface's system type, then add product-specific failure modes on top. Always include safety (user-facing) and task completion (agentic) regardless of type.\n\n| System type | Canonical dimensions |\n|---|---|\n| RAG \u002F retrieval | context faithfulness, hallucination, answer relevance, retrieval precision, source citation |\n| Multi-agent | task decomposition, inter-agent handoff correctness, goal completion, loop detection |\n| Conversational | tone\u002Fstyle, safety, instruction following, escalation accuracy |\n| Extraction \u002F structured output | schema compliance, field accuracy, format validity |\n| Autonomous \u002F tool-using agent | safety guardrails, tool-use correctness, cost\u002Ftoken adherence, task completion |\n| Content generation | factual accuracy, brand voice, tone, originality |\n| Code generation | correctness, safety, test pass rate, instruction following |\n\nFor each dimension, assign a measurement approach before writing the eval case:\n\n- **Code-based**: schema validation, required-field presence, performance thresholds, regex checks. Fast, deterministic, cheap to run in CI.\n- **LLM judge**: tone, reasoning quality, safety-violation detection. Requires calibration against a human-reviewed sample before the score counts as evidence (see Hard Gates).\n- **Human review**: edge cases, LLM-judge calibration itself, high-stakes sampling that cannot be automated yet.\n\n## Tooling and Infrastructure\n\nDetect existing eval\u002Ftracing tooling before recommending anything new:\n\n```bash\ngrep -rl \"langfuse\\|langsmith\\|arize\\|phoenix\\|braintrust\\|promptfoo\\|ragas\" \\\n  --include=\"*.py\" --include=\"*.ts\" --include=\"*.toml\" --include=\"*.json\" . \\\n  2>\u002Fdev\u002Fnull | grep -v node_modules | head -10\n```\n\nIf nothing is detected, these are the default starting points, not a mandate to install all four:\n\n| Concern | Default | Why |\n|---|---|---|\n| Tracing \u002F observability | Arize Phoenix | Open-source, self-hostable, framework-agnostic via OpenTelemetry |\n| RAG eval metrics | RAGAS | Faithfulness, answer relevance, context precision\u002Frecall out of the box |\n| Prompt regression in CI | Promptfoo | CLI-first, no platform account required |\n| LangChain\u002FLangGraph pipelines | LangSmith | Overrides Phoenix when the project is already in that ecosystem |\n\n**Reference dataset spec:** minimum 10 examples to start, 20+ before treating coverage as production-grade. Composition: critical paths, edge cases, known failure modes, and adversarial inputs, not just happy-path samples. Labeling: domain expert where stakes are high, LLM judge with calibration otherwise. Start building the dataset during implementation, not after the feature ships.\n\n**Production monitoring split:** classify every covered failure mode as either an online guardrail (catastrophic risk, runs on every request in the hot path, must be fast) or an offline flywheel check (quality signal, sampled batch, feeds the improvement loop, not latency-sensitive). Keep online guardrails minimal since each one adds latency to every request.\n\n**Coverage scoring:** for each dimension, mark COVERED (implementation exists, targets the rubric behavior, actually runs), PARTIAL (exists but incomplete, not automated, or has known gaps), or MISSING (no implementation found). Audit infrastructure separately, ok\u002Fpartial\u002Fmissing: eval tooling is installed and actually called (not just a listed dependency), the reference dataset file exists and meets the spec above, a CI\u002FCD command runs the eval suite, each planned online guardrail is implemented in the request path (not stubbed), and tracing is configured and wrapping the real AI calls. Score `coverage = covered \u002F total_dimensions × 100` and `infra = (tooling + dataset + cicd + guardrails + tracing) \u002F 5 × 100`, then `overall = coverage × 0.6 + infra × 0.4`.\n\n## Eval Case Design\n\nHow to build the case set (golden cases, adversarial cases, failure-mode coverage,\nand what makes a case gradeable) is in `references\u002Feval-case-design.md`. Read it\nbefore writing cases. Skip it when you are only reviewing an existing suite or\nsizing infrastructure.\n\n## Rubric\n\nUse this table shape:\n\n| Failure mode | Severity | Likelihood | Detectability | Evidence now | Ship gate | Required fix |\n|---|---:|---:|---:|---|---|---|\n| Hallucinates a rights claim | 5 | 3 | 2 | none | block | add refusal eval + source citation check |\n\nScoring:\n\n- **Severity 5:** legal, financial, rights\u002Fprovenance, privacy, security, payment, irreversible user harm, or public trust collapse.\n- **Severity 4:** user-visible wrong outcome on a core workflow, broken agent action, major cost spike, or misleading published statement.\n- **Severity 3:** recoverable user confusion, incomplete answer, or degraded workflow quality.\n- **Severity 2:** minor formatting, tone, or non-core quality miss.\n- **Severity 1:** cosmetic or informational.\n\nGate defaults:\n\n- Any uncovered severity 5 behavior blocks release.\n- Severity 4 requires an eval case, fallback behavior, and named owner before release.\n- Regressions from real observed failures require a fixture or scripted check.\n- Product copy cannot claim eval coverage that does not exist.\n\n## AI-SPEC Template\n\n```text\nAI-SPEC: [surface\u002Fname]\nDate:\nTarget repo\u002Froute\u002FAPI:\nOwner:\n\nUser promise:\nInputs:\nOutputs:\nAllowed sources:\nDisallowed behavior:\nFallback behavior:\nPrivacy\u002Fsecurity boundaries:\nRights\u002Fprovenance boundaries:\nLatency\u002Fcost budget:\nSuccess metrics:\nKnown non-goals:\n\nFailure modes:\nEval suite:\nAcceptance gates:\nCoverage gaps:\nNext implementation step:\n```\n\n## Red Flags: Stop\n\n- \"It looked good in the demo\": a demo is one happy-path sample, not coverage.\n- \"We'll eval after launch\": after launch, the eval set is your users.\n- \"The model seems smart\": vibes are not a rubric row; write the failure mode down and score it.\n- \"We tested the prompt by hand\": prompt review and happy-path poking are not eval coverage.\n- \"It passed once\": a pass with no fixture or scripted check protects nothing on the next model or prompt change.\n- \"The judge model approved it\": self-judgment without human-agreement spot checks is not evidence.\n\n## Output\n\nReturn:\n\n```text\nTarget:\nAI-SPEC:\nFailure-mode rubric:\nEval cases:\nExisting coverage:\nMissing coverage:\nShip gate: ship | ship-with-caveats | hold\nRequired next step:\nCommands or evidence checked:\n```\n\nShip gate is mechanical: **hold** = any severity-5 failure mode uncovered, or no eval plan exists; **ship-with-caveats** = all severity-5 modes covered, remaining severity-4 gaps each have a named owner and follow-up; **ship** = every severity 4-5 failure mode has a case, a gate, and evidence.\n\n## Boundaries\n\n- Do not claim legal, rights, licensing, medical, financial, or compliance clearance.\n- Do not invent private datasets, logs, scores, or customer outcomes.\n- Do not upload data, call private services, or run destructive workflows unless the user explicitly asks and the repo\u002Ftooling supports it.\n- Do not treat a model's self-judgment as sufficient evidence.\n- Do not mark eval coverage complete when only prompt review or happy-path manual testing exists.\n\n## Routing\n\n- The AI surface's implementation needs review or a ship grade → **suede-code**\n- Eval cases written and passing → **suede-ci-gate** to wire them into CI as a required check\n- The eval work is one lane of a bigger coordinated build → **suede-agent-teams**\n","# Suede AI Eval Skills\n\nRead [the worked delivery contract](references\u002Fworked-contract.md) for a concrete\ninput, expected artifact, failure response and validation example.\n\n## Compatibility\n\nRequires the AI surface contract or source and test data; a project runtime\u002Feval runner for execution. Model\u002FAPI access is needed only for actual model runs, with existing authorization and cost scope.\n\n## Gate policy: advisory, not blocking\n\nEvery claim-verification step, check, quality gate, and ship verdict in this\nskill is a **recommendation to the user, not a control on the agent**. This\npolicy governs every gate, check, verdict, and \"do not ship \u002F publish \u002F\nproceed\" line elsewhere in this skill:\n\n- Run every check and report the results honestly. Verdicts (`ship`,\n  `ship-with-caveats`, `hold`, letter grades, BLOCKED or OPEN items) are\n  advice attached to the work, not orders that change it.\n- Never block, delay, skip, rewrite, or refuse the action the user asked for\n  because a check failed or a gate said hold. Complete the requested action\n  as asked, and deliver the gate output alongside it as a clearly labeled\n  recommendation.\n- A failed gate changes what you report, never what you do.\n- Single exception: if a finding is extremely risky (data loss, security or\n  credential exposure, legal or rights violations, payment mistakes, or\n  irreversible public damage), pause, tell the user exactly what the risk is\n  and what the options are, and let them pick. Their choice is final.\n\n\nMake AI behavior testable before it becomes a vague product promise. **No eval plan, no `ship` recommendation: for an AI feature without one, the recommended verdict stays below `ship`: report that gap and let the user decide.**\n\nThe deliverable is an eval plan or coverage audit, not a model benchmark leaderboard. Keep it grounded in the actual product surface, user promise, data sources, prompts, tools, logs, tests, and failure modes available now.\n\n## Hard Gates\n\n- No AI-SPEC → no eval plan. Write the one-paragraph spec first; cases written without a spec test nothing.\n- No eval plan → no `ship` recommendation. Do not recommend `ship` or `ship-with-caveats` for an AI feature that lacks a failure-mode map and eval cases; name the gap and leave the ship decision with the user.\n- A failure mode without an eval case, an owner, and a gate is uncovered, regardless of how unlikely it feels.\n- A live surface that was never sampled gets the output stamped `source-only`; do not present source-only review as runtime evidence.\n- A model grading its own output is not evidence. LLM-as-judge scores count only after spot-checked agreement with a human-reviewed sample.\n\n## Source Truth\n\nInspect the current target before writing the eval. Do not evaluate from memory or product copy alone.\n\nRead or verify:\n\n- repo, branch, remote, dirty state, local instructions, and touched files;\n- the AI surface: route, API, worker, prompt, system message, tool call, model config, retrieval path, classifier, agent loop, generated media path, or recommendation logic;\n- user-facing promise, allowed claims, forbidden claims, safety boundaries, fallback behavior, and support path;\n- input data, retrieval corpus, schemas, tool contracts, metadata, logs, telemetry, and persisted outputs;\n- existing tests, fixtures, eval scripts, prompt snapshots, golden examples, analytics, bug reports, screenshots, or live\u002FAPI readbacks.\n\nWhen the surface is already live, sample real behavior with safe inputs and record exact commands or URLs. When live checks are not appropriate, mark the eval as source-only and name the missing runtime evidence.\n\n## Workflow\n\n1. **Define the AI-SPEC.** State the AI job in one paragraph: user, trigger, input, output, allowed sources, disallowed behavior, fallback, latency\u002Fcost expectation, and success signal.\n2. **Map the failure modes.** List the ways the AI can harm the user, product truth, rights\u002Fprovenance, security, privacy, brand trust, cost, or workflow completion.\n3. **Build the rubric.** Score each failure mode with severity, likelihood, detectability, owner, gate, and required evidence.\n4. **Write eval cases.** Produce concrete pass\u002Ffail cases with inputs, setup data, expected output traits, forbidden output traits, and the reason the case exists.\n5. **Set acceptance gates.** Decide what blocks ship, what allows ship-with-caveats, and what can become follow-up work.\n6. **Audit coverage.** Compare existing tests, logs, metrics, and manual checks against the failure-mode map. Score coverage and infrastructure using the method under Tooling and Infrastructure below. Name every uncovered high-risk behavior regardless of the numeric score.\n7. **Return the artifact.** Give the AI-SPEC, rubric, eval table, coverage gaps, required tests, and next implementation step. Name the exact command that runs the cases and its expected exit status (the repo's own eval script if one exists, otherwise the tool's invocation, e.g. `npx promptfoo eval -c \u003Cconfig>`) and record the run's pass\u002Ffail counts under \"Commands or evidence checked\". An eval plan with no runnable command is a document, not coverage, and suede-ci-gate cannot wire it into CI without that string.\n\n## Eval Dimensions By System Type\n\nStart the failure-mode map from the canonical dimensions for the surface's system type, then add product-specific failure modes on top. Always include safety (user-facing) and task completion (agentic) regardless of type.\n\n| System type | Canonical dimensions |\n|---|---|\n| RAG \u002F retrieval | context faithfulness, hallucination, answer relevance, retrieval precision, source citation |\n| Multi-agent | task decomposition, inter-agent handoff correctness, goal completion, loop detection |\n| Conversational | tone\u002Fstyle, safety, instruction following, escalation accuracy |\n| Extraction \u002F structured output | schema compliance, field accuracy, format validity |\n| Autonomous \u002F tool-using agent | safety guardrails, tool-use correctness, cost\u002Ftoken adherence, task completion |\n| Content generation | factual accuracy, brand voice, tone, originality |\n| Code generation | correctness, safety, test pass rate, instruction following |\n\nFor each dimension, assign a measurement approach before writing the eval case:\n\n- **Code-based**: schema validation, required-field presence, performance thresholds, regex checks. Fast, deterministic, cheap to run in CI.\n- **LLM judge**: tone, reasoning quality, safety-violation detection. Requires calibration against a human-reviewed sample before the score counts as evidence (see Hard Gates).\n- **Human review**: edge cases, LLM-judge calibration itself, high-stakes sampling that cannot be automated yet.\n\n## Tooling and Infrastructure\n\nDetect existing eval\u002Ftracing tooling before recommending anything new:\n\n```bash\ngrep -rl \"langfuse\\|langsmith\\|arize\\|phoenix\\|braintrust\\|promptfoo\\|ragas\" \\\n  --include=\"*.py\" --include=\"*.ts\" --include=\"*.toml\" --include=\"*.json\" . \\\n  2>\u002Fdev\u002Fnull | grep -v node_modules | head -10\n```\n\nIf nothing is detected, these are the default starting points, not a mandate to install all four:\n\n| Concern | Default | Why |\n|---|---|---|\n| Tracing \u002F observability | Arize Phoenix | Open-source, self-hostable, framework-agnostic via OpenTelemetry |\n| RAG eval metrics | RAGAS | Faithfulness, answer relevance, context precision\u002Frecall out of the box |\n| Prompt regression in CI | Promptfoo | CLI-first, no platform account required |\n| LangChain\u002FLangGraph pipelines | LangSmith | Overrides Phoenix when the project is already in that ecosystem |\n\n**Reference dataset spec:** minimum 10 examples to start, 20+ before treating coverage as production-grade. Composition: critical paths, edge cases, known failure modes, and adversarial inputs, not just happy-path samples. Labeling: domain expert where stakes are high, LLM judge with calibration otherwise. Start building the dataset during implementation, not after the feature ships.\n\n**Production monitoring split:** classify every covered failure mode as either an online guardrail (catastrophic risk, runs on every request in the hot path, must be fast) or an offline flywheel check (quality signal, sampled batch, feeds the improvement loop, not latency-sensitive). Keep online guardrails minimal since each one adds latency to every request.\n\n**Coverage scoring:** for each dimension, mark COVERED (implementation exists, targets the rubric behavior, actually runs), PARTIAL (exists but incomplete, not automated, or has known gaps), or MISSING (no implementation found). Audit infrastructure separately, ok\u002Fpartial\u002Fmissing: eval tooling is installed and actually called (not just a listed dependency), the reference dataset file exists and meets the spec above, a CI\u002FCD command runs the eval suite, each planned online guardrail is implemented in the request path (not stubbed), and tracing is configured and wrapping the real AI calls. Score `coverage = covered \u002F total_dimensions × 100` and `infra = (tooling + dataset + cicd + guardrails + tracing) \u002F 5 × 100`, then `overall = coverage × 0.6 + infra × 0.4`.\n\n## Eval Case Design\n\nHow to build the case set (golden cases, adversarial cases, failure-mode coverage,\nand what makes a case gradeable) is in `references\u002Feval-case-design.md`. Read it\nbefore writing cases. Skip it when you are only reviewing an existing suite or\nsizing infrastructure.\n\n## Rubric\n\nUse this table shape:\n\n| Failure mode | Severity | Likelihood | Detectability | Evidence now | Ship gate | Required fix |\n|---|---:|---:|---:|---|---|---|\n| Hallucinates a rights claim | 5 | 3 | 2 | none | block | add refusal eval + source citation check |\n\nScoring:\n\n- **Severity 5:** legal, financial, rights\u002Fprovenance, privacy, security, payment, irreversible user harm, or public trust collapse.\n- **Severity 4:** user-visible wrong outcome on a core workflow, broken agent action, major cost spike, or misleading published statement.\n- **Severity 3:** recoverable user confusion, incomplete answer, or degraded workflow quality.\n- **Severity 2:** minor formatting, tone, or non-core quality miss.\n- **Severity 1:** cosmetic or informational.\n\nGate defaults:\n\n- Any uncovered severity 5 behavior blocks release.\n- Severity 4 requires an eval case, fallback behavior, and named owner before release.\n- Regressions from real observed failures require a fixture or scripted check.\n- Product copy cannot claim eval coverage that does not exist.\n\n## AI-SPEC Template\n\n```text\nAI-SPEC: [surface\u002Fname]\nDate:\nTarget repo\u002Froute\u002FAPI:\nOwner:\n\nUser promise:\nInputs:\nOutputs:\nAllowed sources:\nDisallowed behavior:\nFallback behavior:\nPrivacy\u002Fsecurity boundaries:\nRights\u002Fprovenance boundaries:\nLatency\u002Fcost budget:\nSuccess metrics:\nKnown non-goals:\n\nFailure modes:\nEval suite:\nAcceptance gates:\nCoverage gaps:\nNext implementation step:\n```\n\n## Red Flags: Stop\n\n- \"It looked good in the demo\": a demo is one happy-path sample, not coverage.\n- \"We'll eval after launch\": after launch, the eval set is your users.\n- \"The model seems smart\": vibes are not a rubric row; write the failure mode down and score it.\n- \"We tested the prompt by hand\": prompt review and happy-path poking are not eval coverage.\n- \"It passed once\": a pass with no fixture or scripted check protects nothing on the next model or prompt change.\n- \"The judge model approved it\": self-judgment without human-agreement spot checks is not evidence.\n\n## Output\n\nReturn:\n\n```text\nTarget:\nAI-SPEC:\nFailure-mode rubric:\nEval cases:\nExisting coverage:\nMissing coverage:\nShip gate: ship | ship-with-caveats | hold\nRequired next step:\nCommands or evidence checked:\n```\n\nShip gate is mechanical: **hold** = any severity-5 failure mode uncovered, or no eval plan exists; **ship-with-caveats** = all severity-5 modes covered, remaining severity-4 gaps each have a named owner and follow-up; **ship** = every severity 4-5 failure mode has a case, a gate, and evidence.\n\n## Boundaries\n\n- Do not claim legal, rights, licensing, medical, financial, or compliance clearance.\n- Do not invent private datasets, logs, scores, or customer outcomes.\n- Do not upload data, call private services, or run destructive workflows unless the user explicitly asks and the repo\u002Ftooling supports it.\n- Do not treat a model's self-judgment as sufficient evidence.\n- Do not mark eval coverage complete when only prompt review or happy-path manual testing exists.\n\n## Routing\n\n- The AI surface's implementation needs review or a ship grade → **suede-code**\n- Eval cases written and passing → **suede-ci-gate** to wire them into CI as a required check\n- The eval work is one lane of a bigger coordinated build → **suede-agent-teams**\n",{"name":5,"metadata":50,"description":7},{"display-name":51},"Suede AI Eval Skills","main","2b63f856ce19c2a797929cacabd9e0fc936812fa","plugins\u002FJasonColapietro\u002Fsuede-creator-skills\u002Fskills\u002Fsuede-ai-eval",491,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins","https:\u002F\u002Fhol.org\u002Fplugins\u002Fbest-codex-plugins","https:\u002F\u002Fgithub.com\u002Fhashgraph-online","https:\u002F\u002Fhol.org","Open-source standards and developer tools for interoperable AI agents: discovery, identity, communication, payments, privacy and security.","2026-11-14 14:06:13",{"source":8,"skillName":5,"commands":63,"prompt":73,"zipAvailable":74},[64,67,70],{"tool":65,"command":66},"npx","npx skills add hashgraph-online\u002Fawesome-codex-plugins --skill suede-ai-eval",{"tool":68,"command":69},"bunx","bunx skills add hashgraph-online\u002Fawesome-codex-plugins --skill suede-ai-eval",{"tool":71,"command":72},"pnpm","pnpm dlx skills add hashgraph-online\u002Fawesome-codex-plugins --skill suede-ai-eval","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FJasonColapietro\u002Fsuede-creator-skills\u002Fskills\u002Fsuede-ai-eval\u002FSKILL.md",true,[76,82,86,90],{"path":77,"resourceType":78,"resourceTypeDesc":79,"sizeBytes":80,"sourceHtmlUrl":81},"agents\u002Fopenai.yaml","REFERENCE","参考文档",309,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FJasonColapietro\u002Fsuede-creator-skills\u002Fskills\u002Fsuede-ai-eval\u002Fagents\u002Fopenai.yaml",{"path":83,"resourceType":78,"resourceTypeDesc":79,"sizeBytes":84,"sourceHtmlUrl":85},"CARD.md",4340,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FJasonColapietro\u002Fsuede-creator-skills\u002Fskills\u002Fsuede-ai-eval\u002FCARD.md",{"path":87,"resourceType":78,"resourceTypeDesc":79,"sizeBytes":88,"sourceHtmlUrl":89},"references\u002Feval-case-design.md",10760,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FJasonColapietro\u002Fsuede-creator-skills\u002Fskills\u002Fsuede-ai-eval\u002Freferences\u002Feval-case-design.md",{"path":91,"resourceType":78,"resourceTypeDesc":79,"sizeBytes":92,"sourceHtmlUrl":93},"references\u002Fworked-contract.md",2054,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FJasonColapietro\u002Fsuede-creator-skills\u002Fskills\u002Fsuede-ai-eval\u002Freferences\u002Fworked-contract.md",[95,105,113,121,129,137],{"skillNo":96,"name":97,"nameZh":6,"description":98,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":99,"licenseFileUrl":100,"sourceHtmlUrl":101,"categories":102,"keywords":103,"publishedAt":104},"01M7DV332YQMGD8WTRJ9DZQXTB","worktable-review-with-annotations","Create, inspect, reply to, update, or resolve situated Worktable annotations. Use when feedback belongs to a Doc section, block, text range, or HTML Doc, or when the user asks to address existing review feedback and coordinate a requested artifact edit.","MIT","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-review-with-annotations\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-review-with-annotations\u002FSKILL.md",[],[],"2026-11-15 14:19:17",{"skillNo":106,"name":107,"nameZh":6,"description":108,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":99,"licenseFileUrl":109,"sourceHtmlUrl":110,"categories":111,"keywords":112,"publishedAt":104},"01M7DV2YGHCZVBEB1AHYB6CGHE","worktable-find-and-synthesize-context","Find, assess, and synthesize context from Worktable Docs and Records. Use when the user asks what a workspace contains, wants workspace-backed research, needs current facts reconciled across artifacts, or requests a cited synthesis without changing workspace state.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-find-and-synthesize-context\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-find-and-synthesize-context\u002FSKILL.md",[],[],{"skillNo":114,"name":115,"nameZh":6,"description":116,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":99,"licenseFileUrl":117,"sourceHtmlUrl":118,"categories":119,"keywords":120,"publishedAt":104},"01M7DV2TQN1H11ME2SAF9MXN7V","worktable-create-or-update-docs","Create, edit, rename, or maintain narrative Worktable Docs. Use for plans, decisions, briefs, notes, research syntheses, and other durable prose when the user requests a new artifact or a safe update to an existing one.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-update-docs\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-update-docs\u002FSKILL.md",[],[],{"skillNo":122,"name":123,"nameZh":6,"description":124,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":99,"licenseFileUrl":125,"sourceHtmlUrl":126,"categories":127,"keywords":128,"publishedAt":104},"01M7DV2NE452NEA8TXATKSNAJH","worktable-create-or-manage-records","Organize repeated structured information as durable Worktable Records. Use for research datasets, inventories, directories, pipelines, trackers, and other collections whose items need identity, fields, filtering, links, validation, or individual updates.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-records\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-records\u002FSKILL.md",[],[],{"skillNo":130,"name":131,"nameZh":6,"description":132,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":99,"licenseFileUrl":133,"sourceHtmlUrl":134,"categories":135,"keywords":136,"publishedAt":104},"01M7DV2HENRBR2KSCVM064WH0Y","worktable-create-or-manage-interactive-html","Create or improve durable visual and interactive artifacts in Worktable. Use for dashboards, trackers, calculators, mini apps, slide presentations, and other requests that benefit from a polished HTML Doc.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-interactive-html\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-interactive-html\u002FSKILL.md",[],[],{"skillNo":138,"name":139,"nameZh":6,"description":140,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":99,"licenseFileUrl":141,"sourceHtmlUrl":142,"categories":143,"keywords":144,"publishedAt":104},"01M7DV2CADR5R7RQFFYPNAGR9Y","worktable-collaborate-threads","Read, answer, and hand off messages through durable Worktable threads with users or other authenticated participants. Use when the user asks to check addressed messages, reply to a participant, continue a thread, request another agent's help, wait briefly for a response, or watch for comments and messages sent to the agent.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-collaborate-threads\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-collaborate-threads\u002FSKILL.md",[],[]]