[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6P38DH3QBB5YP1SXY8K6CDM":3,"skill-same:01M6P38DH3QBB5YP1SXY8K6CDM":43},{"skillNo":4,"name":5,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":6,"repositoryNo":11,"starsCount":12,"licenseSpdxId":13,"licenseFileUrl":6,"sourceHtmlUrl":14,"categories":15,"keywords":16,"publishedAt":18,"linkOnly":19,"markdownContent":20,"markdownBody":20,"frontMatter":21,"refName":22,"commitSha":23,"skillDirectory":24,"forksCount":12,"repositoryHtmlUrl":25,"repositoryHomepageUrl":6,"ownerHtmlUrl":26,"ownerBlogUrl":27,"ownerDescription":6,"updatedAt":28,"install":29,"resources":42},"01M6P38DH3QBB5YP1SXY8K6CDM","evaluation",null,"dexter-cnx\u002Fagent-engineering-toolkit","dexter-cnx","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F73522389?v=4","01M6K09WYAAXJXMR26H00WJ2AT","01M6P370Z0XCZ7NC67MJ2MRW94",0,"MIT","https:\u002F\u002Fgithub.com\u002Fdexter-cnx\u002Fagent-engineering-toolkit\u002Fblob\u002Fmain\u002Foverlays\u002Fagent-karpathy\u002Fskills\u002Fevaluation\u002FSKILL.md",[],[17],"Evaluation","2026-11-10 10:12:30",false,"# karpathy-evaluation\n\n## Purpose\nProduce a structured EvalResult JSON for a given SKILL.md by applying the weighted rubric, running binary checks, and estimating token count.\n\n## Use when\n- You need a machine-readable quality score for a skill before deciding whether to mutate it\n- You are comparing baseline vs candidate scores and need consistent, reproducible numbers\n- You are running a CI gate that must fail if the skill scores below 0.60\n- You are building a report and need per-dimension breakdown alongside the final score\n\n## Do NOT use when\n- You only want a qualitative review without numeric output — use `karpathy-guidelines` instead — use a more specific skill instead.\n- You are generating mutations — that is the `karpathy-mutation` skill's responsibility — use a more specific skill instead.\n- You need to check regression safety — use `karpathy-regression` instead — use a more specific skill instead.\n- The skill does not follow the 13-section SKILL.md format (e.g., foundation skills use README.md)\n\n## Inputs required\n- Full content of the SKILL.md to evaluate\n- The parsed rubric dict from `evals\u002Frubrics\u002Fcoding_task_rubric.v1.json`\n- A stable `skill_id` (typically derived from the directory name)\n- A `candidate_id` (use `\"baseline\"` for unmodified skills)\n\n## Constraints\n- Use the exact weights from the rubric: correctness 0.30, scope_discipline 0.15, simplicity 0.15, verifiability 0.15, architecture_alignment 0.10, token_efficiency 0.10, docs_hygiene 0.05\n- Token count must be estimated as `ceil(word_count × 1.3)` for consistency across runs\n- `quality_per_1k_tokens` must equal `final_score \u002F (token_count \u002F 1000)`\n- Never set `regression_passed = True` in this skill — that field is set by the regression agent\n- Binary checks are pass\u002Ffail only, not partial credit\n\n## Step-by-step workflow\n1. Load the SKILL.md content and the rubric from `evals\u002Frubrics\u002Fcoding_task_rubric.v1.json`.\n2. Count tokens: split content on whitespace, multiply word count by 1.3, ceiling-round.\n3. Score each of the 7 dimensions using the rubric criteria and scoring levels.\n4. Compute `final_score = sum(weight × score)` across all 7 dimensions.\n5. Compute `quality_per_1k_tokens = final_score \u002F (token_count \u002F 1000)`.\n6. Run all 6 binary checks (has_purpose_section, has_workflow_steps, no_placeholder_text, has_validation_checklist, has_output_contract, has_real_example).\n7. Set `binary_checks_passed = all(binary_checks.values())`.\n8. Set `regression_passed = True` as a placeholder — downstream regression agent overwrites this.\n9. Assemble and return the EvalResult dict matching `evals\u002Fschemas\u002Fskill_eval.schema.json`.\n\n## Output contract\n- An EvalResult dict with fields: skill_id, candidate_id, mutation_type, scores (7 dimensions), final_score, token_count, quality_per_1k_tokens, binary_checks (6 keys), binary_checks_passed, regression_passed, evaluator_notes, timestamp\n- The dict must validate against `evals\u002Fschemas\u002Fskill_eval.schema.json`\n- `evaluator_notes` must name the two lowest-scoring dimensions if any score below 0.5\n\n## Validation checklist\n- Confirm `sum(weight × score) == final_score` to within floating-point tolerance (1e-6)\n- Confirm all 7 dimension score keys are present in the `scores` dict\n- Confirm `quality_per_1k_tokens > 0` when `token_count > 0`\n- Run `python -m runners.eval_runner --skill \u003Cpath>` and confirm valid JSON is returned\n- Confirm `timestamp` is a valid ISO 8601 datetime string\n\n## Related skills\n- `karpathy-guidelines` — manual qualitative application of the same criteria\n- `karpathy-mutation` — uses EvalResult to decide which dimensions to mutate\n- `karpathy-optimizer` — uses EvalResult to rank candidates and apply token policy\n\n## References\n- [`..\u002F..\u002Fevals\u002Fschemas\u002Fskill_eval.schema.json`](..\u002F..\u002Fevals\u002Fschemas\u002Fskill_eval.schema.json)\n- [`..\u002F..\u002Fevals\u002Frubrics\u002Fcoding_task_rubric.v1.json`](..\u002F..\u002Fevals\u002Frubrics\u002Fcoding_task_rubric.v1.json)\n- [`..\u002F..\u002Fagents\u002Fevaluator_agent.py`](..\u002F..\u002Fagents\u002Fevaluator_agent.py)\n\n## Real example\nHistorical static snapshot: running the evaluator on `flutter-go-router-deeplink-wireup\u002FSKILL.md` returns final_score 0.7234 with correctness 0.80, scope_discipline 0.70, simplicity 0.90, verifiability 0.60 (two vague checklist items), architecture_alignment 0.70, token_efficiency 0.85, docs_hygiene 1.0. The `evaluator_notes` field reads: \"Low-scoring dimensions: verifiability. Failed binary checks: none.\"\n\n## Real file output sample\n```text\nmemory\u002Fscore_history.json\nreports\u002Flatest_report.md\n```\n",{},"main","3035171e2986d5fdb445e694b9013fbdf58b3ff6","overlays\u002Fagent-karpathy\u002Fskills\u002Fevaluation","https:\u002F\u002Fgithub.com\u002Fdexter-cnx\u002Fagent-engineering-toolkit","https:\u002F\u002Fgithub.com\u002Fdexter-cnx","https:\u002F\u002Fdexter-cnx.github.io\u002Fportfolio\u002F","2026-05-26 14:24:27",{"source":7,"skillName":5,"commands":30,"prompt":40,"zipAvailable":41},[31,34,37],{"tool":32,"command":33},"npx","npx skills add dexter-cnx\u002Fagent-engineering-toolkit --skill evaluation",{"tool":35,"command":36},"bunx","bunx skills add dexter-cnx\u002Fagent-engineering-toolkit --skill evaluation",{"tool":38,"command":39},"pnpm","pnpm dlx skills add dexter-cnx\u002Fagent-engineering-toolkit --skill evaluation","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fdexter-cnx\u002Fagent-engineering-toolkit\u002Fblob\u002Fmain\u002Foverlays\u002Fagent-karpathy\u002Fskills\u002Fevaluation\u002FSKILL.md",true,[],[44,71,91,100,112,124],{"skillNo":45,"name":46,"nameZh":6,"description":47,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":6,"repositoryNo":11,"starsCount":12,"licenseSpdxId":13,"licenseFileUrl":6,"sourceHtmlUrl":48,"categories":49,"keywords":50,"publishedAt":18},"01M6P4P9S9017DTSFMRKGA7XW2","toolkit-i18n-operator","Use the toolkit-i18n CLI for repeated Flutter localization workflows with compact outputs, CSV validation, key coverage reporting, generated JSON exports, and safe non-mutating behavior.","https:\u002F\u002Fgithub.com\u002Fdexter-cnx\u002Fagent-engineering-toolkit\u002Fblob\u002Fmain\u002Ftools\u002Ftoolkit-i18n\u002Fskills\u002Ftoolkit-i18n-operator\u002FSKILL.md",[],[51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70],"toolkit i18n operator","i18n","operator","toolkit","flutter localization","cli","safe","workflows","for","validation","with","csv","compact","reporting","json","flutter","behavior","coverage","generated","use",{"skillNo":72,"name":73,"nameZh":6,"description":74,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":6,"repositoryNo":11,"starsCount":12,"licenseSpdxId":13,"licenseFileUrl":6,"sourceHtmlUrl":75,"categories":76,"keywords":87,"publishedAt":18},"01M6P4NREKE3P0XYV3RBBWC987","toolkit-design-operator","Use the toolkit-design CLI for repeated repository workflows.","https:\u002F\u002Fgithub.com\u002Fdexter-cnx\u002Fagent-engineering-toolkit\u002Fblob\u002Fmain\u002Ftools\u002Ftoolkit-design\u002Fskills\u002Ftoolkit-design-operator\u002FSKILL.md",[77,82],{"categoryNo":78,"categoryName":79,"taxonomyType":80,"taxonomyTypeDesc":81},"01JBSEEDCAT000000000000015","视觉设计","SCENARIO","使用场景",{"categoryNo":83,"categoryName":84,"taxonomyType":85,"taxonomyTypeDesc":86},"01JBSEEDCAT000000000000032","设计师","PROFESSION","适用职业",[88,53,54,89,59,56,58,90,70],"toolkit design operator","design","repository",{"skillNo":92,"name":93,"nameZh":6,"description":94,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":6,"repositoryNo":11,"starsCount":12,"licenseSpdxId":13,"licenseFileUrl":6,"sourceHtmlUrl":95,"categories":96,"keywords":97,"publishedAt":18},"01M6P4NNSQ09QFPYG2PMPWEA94","toolkit-ci-operator","Use the toolkit-ci CLI for repeated repository workflows.","https:\u002F\u002Fgithub.com\u002Fdexter-cnx\u002Fagent-engineering-toolkit\u002Fblob\u002Fmain\u002Ftools\u002Ftoolkit-ci\u002Fskills\u002Ftoolkit-ci-operator\u002FSKILL.md",[],[98,99,53,54,90,56,58,70,59],"toolkit ci operator","ci",{"skillNo":101,"name":102,"nameZh":6,"description":103,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":6,"repositoryNo":11,"starsCount":12,"licenseSpdxId":13,"licenseFileUrl":6,"sourceHtmlUrl":104,"categories":105,"keywords":106,"publishedAt":18},"01M6P4NK636991P6WWQ6X8HDYX","toolkit-arch-operator","Use the toolkit-arch CLI for repeated repository workflows, including Flutter i18n usage scans, layer checks, and localization coverage.","https:\u002F\u002Fgithub.com\u002Fdexter-cnx\u002Fagent-engineering-toolkit\u002Fblob\u002Fmain\u002Ftools\u002Ftoolkit-arch\u002Fskills\u002Ftoolkit-arch-operator\u002FSKILL.md",[],[107,108,54,53,52,109,110,59,70,68,66,111,90,58,56],"toolkit arch operator","arch","usage","layer","localization",{"skillNo":113,"name":114,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":6,"repositoryNo":11,"starsCount":12,"licenseSpdxId":13,"licenseFileUrl":6,"sourceHtmlUrl":115,"categories":116,"keywords":120,"publishedAt":18},"01M6P4NF18XGVSG2NR01E1DX0R","web-testing","https:\u002F\u002Fgithub.com\u002Fdexter-cnx\u002Fagent-engineering-toolkit\u002Fblob\u002Fmain\u002Foverlays\u002Fweb-frontend\u002Fskills\u002Fweb-testing\u002Fskill.md",[117],{"categoryNo":118,"categoryName":119,"taxonomyType":80,"taxonomyTypeDesc":81},"01JBSEEDCAT000000000000008","测试与质量",[121,122,123],"web testing","web","testing",{"skillNo":125,"name":126,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":6,"repositoryNo":11,"starsCount":12,"licenseSpdxId":13,"licenseFileUrl":6,"sourceHtmlUrl":127,"categories":128,"keywords":129,"publishedAt":18},"01M6P4MVK07EJ75FQTFN54AGPM","web-state","https:\u002F\u002Fgithub.com\u002Fdexter-cnx\u002Fagent-engineering-toolkit\u002Fblob\u002Fmain\u002Foverlays\u002Fweb-frontend\u002Fskills\u002Fweb-state\u002Fskill.md",[],[122,130,131],"state","web state"]