eval-gate-authoring
Turn observed run outputs into eval-spec Artifacts, paired Gates, and policy bindings. Use when creating or calibrating automated, human, or LLM-as-judge eval gates for processkit workflows.
- License 许可
- MIT
- 最近更新
- 2026-11-11
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- eval-gate-authoring
- metadata
- { "processkit": { "id": "SKILL-eval-gate-authoring", "layer": 3, "created": 1777507200000, "version": "2.0.0-alpha.1", "category": "processkit", "provides": { "mcp_tools": [ "collect_run_outputs", "codify_eval", "calibrate_judge", "bind_eval_to_runs" ], "primitives": [ "Artifact", "Gate", "Binding", "LogEntry" ] }, "apiVersion": "processkit.projectious.work/v2" } }
- description
- Turn observed run outputs into eval-spec Artifacts, paired Gates, and policy bindings. Use when creating or calibrating automated, human, or LLM-as-judge eval gates for processkit workflows.
02 资源文件 4
- 参考文档mcp/mcp-config.json221 B
- 参考文档mcp/SERVER.md429 B
- 脚本mcp/server.py7.9 KB
- 脚本scripts/test_eval_gate_authoring.py2.2 KB
03 SKILL.md 正文
Eval Gate Authoring
Intro
Eval gate authoring promotes repeated run observations into enforceable
checks. Capture examples, codify an eval-spec Artifact, create the
paired Gate, calibrate any LLM judge against human labels, then bind
the eval to the target run surface with a policy-application Binding.
Overview
The workflow starts from observed run output and ends with an auditable Gate plus bindings that describe where the evaluation applies. LLM-as-judge evals require calibration evidence before they should be treated as enforceable.
Gotchas
- Do not create an eval gate from one example unless the gate is clearly marked exploratory.
- Do not treat an LLM judge as calibrated until human labels and a calibration LogEntry exist.
- Bind evals to the narrowest useful run surface so a local experiment does not accidentally become a global policy.
Full reference
MCP tools
collect_run_outputscodify_evalcalibrate_judgebind_eval_to_runs
Produced entities
Artifact(spec.kind=eval-spec)GateBinding(type=policy-application)- calibration
LogEntryrecords
04 同仓库更多 Skills
standard-ui-design
暂无简介
standard-agent-use
Company standard for working with AI agents. Use when deciding what to delegate to agents, capturing open work, keeping agent memory or knowledge notes, improving an agent workflow, or introducing agent work in a team or for a client.
standard-agent-building
暂无简介
skill-management
Manage Skill package manifests as v1 ontology entities.
schema-management
Manage generated processkit schema contracts through deterministic MCP rebuild and inspection workflows.
proposition-management
Record claims, risks, beliefs, world facts, WSJF estimates, and assumptions through the Proposition interface.