waza-runner
Run evaluations on Agent Skills to measure their effectiveness. USE FOR: "run skill evals", "evaluate my skill", "test skill quality", "check skill triggers", "skill compliance check", "measure skill performance", "run evals on [skill-name]", "grade skill execution". DO NOT USE FOR: writing skills (use skill-authoring), improving frontmatter (use sensei), or general testing unrelated to skills.
- License 许可
- MIT
- 所属仓库
- microsoft/waza
- 最近更新
- 2026-11-15
- Stars
- ★ 1.5k
- Forks
- 101
01 前言 Front Matter
- name
- waza-runner
- metadata
- { "author": "spboyer", "version": "1.0" }
- description
- Run evaluations on Agent Skills to measure their effectiveness. USE FOR: "run skill evals", "evaluate my skill", "test skill quality", "check skill triggers", "skill compliance check", "measure skill performance", "run evals on [skill-name]", "grade skill execution". DO NOT USE FOR: writing skills (use skill-authoring), improving frontmatter (use sensei), or general testing unrelated to skills.
02 资源文件 1
- 参考文档references/EVAL-SPEC.md3.1 KB
03 SKILL.md 正文
Skill Eval Runner
Evaluate Agent Skills like you evaluate AI Agents
This skill runs evaluations on other skills to measure their effectiveness using the same patterns that power AI agent evaluations.
When to Use
- Running quality evaluations on a skill
- Testing if a skill triggers on correct prompts
- Measuring skill behavior quality
- Generating eval reports for CI/CD
Commands
Run Evals
Run evals on <skill-name>
Initialize Eval Suite
Create evals for <skill-name>
Generate Report
Generate eval report for <skill-name>
Workflow
- Check for Eval Suite: Look for
eval.yamlin the skill directory - Load Tasks: Parse task definitions from
tasks/*.yaml - Execute: Run each task through the configured graders
- Report: Output results in JSON or Markdown format
Metrics Measured
| Metric | Description | Default Threshold |
|---|---|---|
| Task Completion | Did the skill accomplish the goal? | 80% |
| Trigger Accuracy | Was skill invoked on correct prompts? | 90% |
| Behavior Quality | Tool calls, efficiency, reasoning | 70% |
Grader Types
- Code Graders: Deterministic assertions, regex matching
- LLM Graders: Model-as-judge with configurable rubrics
- Human Graders: Manual review workflow
Example Usage
Running Evals
# From CLI
waza run ./my-skill/eval.yaml
# Output to file
waza run ./my-skill/eval.yaml -o results.json
Interpreting Results
{
"summary": {
"pass_rate": 0.85,
"composite_score": 0.82
},
"metrics": {
"task_completion": { "score": 0.9, "passed": true },
"trigger_accuracy": { "score": 0.95, "passed": true }
}
}
References
- Eval Specification - Full eval.yaml schema
- Writing Tasks - Task definition guide
- Grader Reference - Available graders
04 同仓库更多 Skills
waza
**WORKFLOW SKILL** - Evaluate AI agent skills using structured benchmarks with YAML specs, fixture isolation, and pluggable validators. USE FOR: run waza, waza help, run eval, run benchmark, evaluate skill, test agent, generate eval suite, init eval, compare results, score agent, agent evaluation, skill testing, cross-model comparison. DO NOT USE FOR: improving skill frontmatter (use waza dev), creating new skills from scratch (use skill-creator), token counting or budget checks (use waza tokens). INVOKES: Copilot SDK executor, mock engine, code/regex validators. FOR SINGLE OPERATIONS: use waza run directly for a single benchmark.
waza-interactive
Interactive workflow partner for creating, testing, and improving AI agent skills with waza. USE FOR: run my evals, check my skill, compare models, create eval suite, debug failing tests, is my skill ready, ship readiness, interpret results, improve score. DO NOT USE FOR: general coding, non-skill work, writing skill content (use skill-authoring), improving frontmatter only (use sensei).
code-explainer
Explains code snippets in plain English, breaking down what the code does step by step. Perfect for learning, code reviews, or documentation.
compatibility-skill
Read fixture status and report whether it is ready.
evidence-judge
Read a supplied report and summarize its recorded status and numeric items.
code-explainer
**UTILITY SKILL** - Explain code snippets, functions, and algorithms in plain language. USE FOR: explain code, what does this code do, break down this function, help me understand this, walk through this algorithm, clarify this logic, explain this snippet, describe what happens here. DO NOT USE FOR: writing new code (use code generation), fixing bugs (use debugging), refactoring (use refactoring skills), code review with action items. INVOKES: file reading tools to access code, language detection for tailored explanations. FOR SINGLE OPERATIONS: If the user just needs to see file contents, use file reading tools directly.