AGENT SKILL

waza-runner

Run evaluations on Agent Skills to measure their effectiveness. USE FOR: "run skill evals", "evaluate my skill", "test skill quality", "check skill triggers", "skill compliance check", "measure skill performance", "run evals on [skill-name]", "grade skill execution". DO NOT USE FOR: writing skills (use skill-authoring), improving frontmatter (use sensei), or general testing unrelated to skills.

License 许可
MIT
所属仓库
microsoft/waza
最近更新
2026-11-15
Stars
★ 1.5k
Forks
101

01 前言 Front Matter

name
waza-runner
metadata
{ "author": "spboyer", "version": "1.0" }
description
Run evaluations on Agent Skills to measure their effectiveness. USE FOR: "run skill evals", "evaluate my skill", "test skill quality", "check skill triggers", "skill compliance check", "measure skill performance", "run evals on [skill-name]", "grade skill execution". DO NOT USE FOR: writing skills (use skill-authoring), improving frontmatter (use sensei), or general testing unrelated to skills.

02 资源文件 1

03 SKILL.md 正文

Skill Eval Runner

Evaluate Agent Skills like you evaluate AI Agents

This skill runs evaluations on other skills to measure their effectiveness using the same patterns that power AI agent evaluations.

When to Use

  • Running quality evaluations on a skill
  • Testing if a skill triggers on correct prompts
  • Measuring skill behavior quality
  • Generating eval reports for CI/CD

Commands

Run Evals

Run evals on <skill-name>

Initialize Eval Suite

Create evals for <skill-name>

Generate Report

Generate eval report for <skill-name>

Workflow

  1. Check for Eval Suite: Look for eval.yaml in the skill directory
  2. Load Tasks: Parse task definitions from tasks/*.yaml
  3. Execute: Run each task through the configured graders
  4. Report: Output results in JSON or Markdown format

Metrics Measured

Metric Description Default Threshold
Task Completion Did the skill accomplish the goal? 80%
Trigger Accuracy Was skill invoked on correct prompts? 90%
Behavior Quality Tool calls, efficiency, reasoning 70%

Grader Types

  • Code Graders: Deterministic assertions, regex matching
  • LLM Graders: Model-as-judge with configurable rubrics
  • Human Graders: Manual review workflow

Example Usage

Running Evals

# From CLI
waza run ./my-skill/eval.yaml

# Output to file
waza run ./my-skill/eval.yaml -o results.json

Interpreting Results

{
  "summary": {
    "pass_rate": 0.85,
    "composite_score": 0.82
  },
  "metrics": {
    "task_completion": { "score": 0.9, "passed": true },
    "trigger_accuracy": { "score": 0.95, "passed": true }
  }
}

References

04 同仓库更多 Skills

microsoft
microsoft/waza

waza

**WORKFLOW SKILL** - Evaluate AI agent skills using structured benchmarks with YAML specs, fixture isolation, and pluggable validators. USE FOR: run waza, waza help, run eval, run benchmark, evaluate skill, test agent, generate eval suite, init eval, compare results, score agent, agent evaluation, skill testing, cross-model comparison. DO NOT USE FOR: improving skill frontmatter (use waza dev), creating new skills from scratch (use skill-creator), token counting or budget checks (use waza tokens). INVOKES: Copilot SDK executor, mock engine, code/regex validators. FOR SINGLE OPERATIONS: use waza run directly for a single benchmark.

测试与质量Skill与智能体开发
★ 1.5k
microsoft
microsoft/waza

waza-interactive

Interactive workflow partner for creating, testing, and improving AI agent skills with waza. USE FOR: run my evals, check my skill, compare models, create eval suite, debug failing tests, is my skill ready, ship readiness, interpret results, improve score. DO NOT USE FOR: general coding, non-skill work, writing skill content (use skill-authoring), improving frontmatter only (use sensei).

开发工程师内容创作者Skill与智能体开发
★ 1.5k
microsoft
microsoft/waza

code-explainer

Explains code snippets in plain English, breaking down what the code does step by step. Perfect for learning, code reviews, or documentation.

开发工程师
★ 1.5k
microsoft
microsoft/waza

compatibility-skill

Read fixture status and report whether it is ready.

★ 1.5k
microsoft
microsoft/waza

evidence-judge

Read a supplied report and summarize its recorded status and numeric items.

★ 1.5k
microsoft
microsoft/waza

code-explainer

**UTILITY SKILL** - Explain code snippets, functions, and algorithms in plain language. USE FOR: explain code, what does this code do, break down this function, help me understand this, walk through this algorithm, clarify this logic, explain this snippet, describe what happens here. DO NOT USE FOR: writing new code (use code generation), fixing bugs (use debugging), refactoring (use refactoring skills), code review with action items. INVOKES: file reading tools to access code, language detection for tailored explanations. FOR SINGLE OPERATIONS: If the user just needs to see file contents, use file reading tools directly.

开发工程师代码审查
★ 1.5k