evaluation
暂无简介
- License 许可
- MIT
- 最近更新
- 2026-05-26
- Stars
- ★ 0
- Forks
- 0
03 SKILL.md 正文
karpathy-evaluation
Purpose
Produce a structured EvalResult JSON for a given SKILL.md by applying the weighted rubric, running binary checks, and estimating token count.
Use when
- You need a machine-readable quality score for a skill before deciding whether to mutate it
- You are comparing baseline vs candidate scores and need consistent, reproducible numbers
- You are running a CI gate that must fail if the skill scores below 0.60
- You are building a report and need per-dimension breakdown alongside the final score
Do NOT use when
- You only want a qualitative review without numeric output — use
karpathy-guidelinesinstead — use a more specific skill instead. - You are generating mutations — that is the
karpathy-mutationskill's responsibility — use a more specific skill instead. - You need to check regression safety — use
karpathy-regressioninstead — use a more specific skill instead. - The skill does not follow the 13-section SKILL.md format (e.g., foundation skills use README.md)
Inputs required
- Full content of the SKILL.md to evaluate
- The parsed rubric dict from
evals/rubrics/coding_task_rubric.v1.json - A stable
skill_id(typically derived from the directory name) - A
candidate_id(use"baseline"for unmodified skills)
Constraints
- Use the exact weights from the rubric: correctness 0.30, scope_discipline 0.15, simplicity 0.15, verifiability 0.15, architecture_alignment 0.10, token_efficiency 0.10, docs_hygiene 0.05
- Token count must be estimated as
ceil(word_count × 1.3)for consistency across runs quality_per_1k_tokensmust equalfinal_score / (token_count / 1000)- Never set
regression_passed = Truein this skill — that field is set by the regression agent - Binary checks are pass/fail only, not partial credit
Step-by-step workflow
- Load the SKILL.md content and the rubric from
evals/rubrics/coding_task_rubric.v1.json. - Count tokens: split content on whitespace, multiply word count by 1.3, ceiling-round.
- Score each of the 7 dimensions using the rubric criteria and scoring levels.
- Compute
final_score = sum(weight × score)across all 7 dimensions. - Compute
quality_per_1k_tokens = final_score / (token_count / 1000). - Run all 6 binary checks (has_purpose_section, has_workflow_steps, no_placeholder_text, has_validation_checklist, has_output_contract, has_real_example).
- Set
binary_checks_passed = all(binary_checks.values()). - Set
regression_passed = Trueas a placeholder — downstream regression agent overwrites this. - Assemble and return the EvalResult dict matching
evals/schemas/skill_eval.schema.json.
Output contract
- An EvalResult dict with fields: skill_id, candidate_id, mutation_type, scores (7 dimensions), final_score, token_count, quality_per_1k_tokens, binary_checks (6 keys), binary_checks_passed, regression_passed, evaluator_notes, timestamp
- The dict must validate against
evals/schemas/skill_eval.schema.json evaluator_notesmust name the two lowest-scoring dimensions if any score below 0.5
Validation checklist
- Confirm
sum(weight × score) == final_scoreto within floating-point tolerance (1e-6) - Confirm all 7 dimension score keys are present in the
scoresdict - Confirm
quality_per_1k_tokens > 0whentoken_count > 0 - Run
python -m runners.eval_runner --skill <path>and confirm valid JSON is returned - Confirm
timestampis a valid ISO 8601 datetime string
Related skills
karpathy-guidelines— manual qualitative application of the same criteriakarpathy-mutation— uses EvalResult to decide which dimensions to mutatekarpathy-optimizer— uses EvalResult to rank candidates and apply token policy
References
../../evals/schemas/skill_eval.schema.json../../evals/rubrics/coding_task_rubric.v1.json../../agents/evaluator_agent.py
Real example
Historical static snapshot: running the evaluator on flutter-go-router-deeplink-wireup/SKILL.md returns final_score 0.7234 with correctness 0.80, scope_discipline 0.70, simplicity 0.90, verifiability 0.60 (two vague checklist items), architecture_alignment 0.70, token_efficiency 0.85, docs_hygiene 1.0. The evaluator_notes field reads: "Low-scoring dimensions: verifiability. Failed binary checks: none."
Real file output sample
memory/score_history.json
reports/latest_report.md
04 同仓库更多 Skills
toolkit-i18n-operator
Use the toolkit-i18n CLI for repeated Flutter localization workflows with compact outputs, CSV validation, key coverage reporting, generated JSON exports, and safe non-mutating behavior.
toolkit-design-operator
Use the toolkit-design CLI for repeated repository workflows.
toolkit-ci-operator
Use the toolkit-ci CLI for repeated repository workflows.
toolkit-arch-operator
Use the toolkit-arch CLI for repeated repository workflows, including Flutter i18n usage scans, layer checks, and localization coverage.
web-testing
暂无简介
web-state
暂无简介