askit-build-samples
Creates and validates a skill's sample sets and eval sets (golden examples, anti-examples, and triggering cases) and detects drift against current behavior. Use when generating samples for a skill, building an eval set, or checking samples for drift.
- License 许可
- Apache-2.0
- 最近更新
- 2026-11-14
- Stars
- ★ 3
- Forks
- 0
01 前言 Front Matter
- name
- askit-build-samples
- metadata
- { "tier": "universal", "version": "0.1.0", "audience": "advanced" }
- description
- Creates and validates a skill's sample sets and eval sets (golden examples, anti-examples, and triggering cases) and detects drift against current behavior. Use when generating samples for a skill, building an eval set, or checking samples for drift.
02 资源文件 3
- 参考文档examples/golden-1-statusline-evidence-set.md11.6 KB
- 参考文档README.md605 B
- 参考文档references/samples-format.md2.0 KB
03 SKILL.md 正文
askit-build-samples
Purpose
Author and validate the evidence a skill carries, following the builder pattern (../../docs/reference/builder-pattern.md). create generates a skill's samples (at least 3 golden examples plus at least 1 anti-example, Standard sec 7.2) under examples/, and its triggering eval set (at least 20 {query, should_trigger} cases, sec 8.3) plus any chain/hook behavior cases under evals/ in the eval-set format the G3 library-regression check and the askit-evaluate behavioral mode consume. validate detects drift: a sample or eval that no longer matches the skill's current behavior is an error, not a silent staleness. Format and the example-threads convention are in references/samples-format.md.
When to use
When generating samples or an eval set for a skill, or checking existing samples and evals for drift after a behavior change.
create mode
- Read the target skill (its description, triggers, and behavior).
- Generate
examples/golden samples (>= 3 realistic input/output pairs) and an anti-example (>= 1 case the skill should NOT handle), per sec 7.2. - Generate
evals/<name>.eval.json: a triggering set of >= 20{query, should_trigger}cases (fires when it should, silent when it should not, sec 8.3), plus{given, expect}behavior cases for any chain the skill participates in (so the G3 check finds coverage).
validate mode
- Re-run the samples and evals against the skill's current behavior.
- Flag drift: a golden sample whose output changed, an anti-example that now triggers, or an eval whose expectation no longer holds. Drift is an error so samples stay honest (sec 7.2, 8.3).
Scope
Samples and eval sets are the evidence layer. The deterministic G3 baseline (presence + the regression signal) is enforced by library-regression; behavioral judging of the cases is the opt-in askit-evaluate behavioral mode (delegated to askit-quality-grader), never the CI gate (Design Principle 3). Example-threads (the bounded validation triad of ADR 0021: a greenfield Bronze plugin, the pm-skills adopt-and-grade thread, and the toolkit itself as Gold) anchor samples to real end-to-end arcs rather than isolated snippets.
04 同仓库更多 Skills
fx-demo
A fixture skill that exists so the re-derived manifest has a component list to compare. Use only from tests/unit/self-consistency.test.mjs.
fx-demo
A fixture skill that exists so the re-derived manifest has a component list to compare. Use only from tests/unit/self-consistency.test.mjs.
fx-demo
A fixture skill that exists so the re-derived manifest has a component list to compare. Use only from tests/unit/self-consistency.test.mjs.
sf-caller
Calls the worker subagent to do a thing. Use when delegating work to sf-worker.
sf-do-thing
Converts CSV input into a formatted summary. Use when the user asks to summarize or tabulate spreadsheet data.
root-skill-plugin
Summarizes a changelog section into one release-note sentence. Use when the user asks to condense a list of changelog bullets into a single announcement line.