regression
暂无简介
- License 许可
- MIT
- 最近更新
- 2026-05-26
- Stars
- ★ 0
- Forks
- 0
03 SKILL.md 正文
karpathy-regression
Purpose
Enforce structural and behavioral guardrails on mutation candidates before any promotion decision is made, hard-failing on any violation.
Use when
- You have generated mutation candidates and need to verify none of them break existing behavior, remove required sections, or introduce architecture violations
- You are running a CI gate and need a hard stop before a potentially broken skill reaches the promotion stage
- A candidate has an unexpectedly high score and you want to confirm it was not achieved by removing constraints or degrading the output contract
- You are manually reviewing a candidate before committing it to the repository
Do NOT use when
- You are checking the baseline skill before mutation — regression is for candidates only — use a more specific skill instead.
- You only want a quality score — use
karpathy-evaluationinstead — use a more specific skill instead. - The skill is not in SKILL.md format (regression checks assume the 13-section schema) — use a more specific skill instead.
- You are reviewing policies, templates, or documentation files (not SKILL.md content)
Inputs required
- One or more MutationCandidate dicts (output of
karpathy-mutation) - The baseline SKILL.md content (to compare section counts and output contract)
- The skill_path (to check forbidden file paths)
Constraints
- Must hard-fail (exit 1) if ANY of the five checks fail for a candidate
- Must not modify any file during a check — read-only inspection only
- Must return a RegressionResult for every candidate, even if some pass and others fail
- Forbidden mutation paths include: AGENTS.md, AGENTS.overlay.md, docs/prompt-pipeline.md, evals/rubrics/, evals/schemas/, .github/workflows/
- A candidate that removes an existing H2 section always fails — no exceptions
Step-by-step workflow
- For each candidate, extract all
##section headers and compare to the baseline set — fail if any are missing. - Scan candidate content for architecture violation patterns: direct SDK imports in presentation layer, repository bypass, logic placed directly in a widget.
- Check that the skill_path does not match any entry in the forbidden paths list.
- Compare the Purpose section between candidate and baseline — fail if candidate's Purpose is empty when baseline's was not.
- Compare Output contract bullet count between candidate and baseline — fail if candidate has fewer deliverables.
- If all checks pass, set
passed = Trueand return an emptyviolationslist. - If any check fails, set
passed = False, populateviolationswith one message per failure, and raiseRegressionViolation.
Output contract
- A RegressionResult dict per candidate: candidate_id, passed (bool), violations (list of strings)
- A summary dict with: regression_results (list), passing_candidates (list of candidates that passed), regression_failures (list of failing candidate_ids), all_passed (bool)
- Exit code 1 from
regression_runner.pyif any candidate fails
Validation checklist
- Run
python -m runners.regression_runner --skill <path> --candidates <file>and confirm exit code 0 for clean candidates - Confirm a candidate with a removed section produces
passed: falseand a non-emptyviolationslist - Confirm the
regression_failureslist matches the set of candidate_ids withpassed: false - Confirm passing candidates are forwarded to the promotion stage without modification
Related skills
karpathy-mutation— produces the candidates this skill checkskarpathy-optimizer— receives only regression-passing candidateskarpathy-evaluation— runs after regression to score passing candidates
References
Real example
Historical static snapshot: the token_budget mutation for flutter deep link removes the Constraints section in an aggressive attempt to reduce tokens. The regression check on step 1 immediately detects the missing ## Constraints header and raises RegressionViolation(candidate_id='baseline-token_budget', violations=['Missing required section: ## Constraints']). This candidate is excluded from the promotion stage.
Real file output sample
memory/candidate_archive.json
reports/latest_report.md
04 同仓库更多 Skills
toolkit-i18n-operator
Use the toolkit-i18n CLI for repeated Flutter localization workflows with compact outputs, CSV validation, key coverage reporting, generated JSON exports, and safe non-mutating behavior.
toolkit-design-operator
Use the toolkit-design CLI for repeated repository workflows.
toolkit-ci-operator
Use the toolkit-ci CLI for repeated repository workflows.
toolkit-arch-operator
Use the toolkit-arch CLI for repeated repository workflows, including Flutter i18n usage scans, layer checks, and localization coverage.
web-testing
暂无简介
web-state
暂无简介