optimization
暂无简介
- License 许可
- MIT
- 最近更新
- 2026-05-26
- Stars
- ★ 0
- Forks
- 0
03 SKILL.md 正文
karpathy-optimizer
Purpose
Select the best mutation candidate using token efficiency policy and score delta, then promote it to disk or return a REJECT decision with reasoning.
Use when
- You have a set of regression-passing candidate EvalResults and need to decide which (if any) to promote
- The final_score of at least one candidate exceeds the baseline final_score
- You need a machine-readable PromotionDecision dict for downstream reporting
- You are running the last stage of a full optimization cycle before writing the report
Do NOT use when
- Regression checks have not been run yet — call
karpathy-regressionfirst — use a more specific skill instead. - No candidates passed regression — the optimizer requires at least one passing candidate — use a more specific skill instead.
- You want to force-promote a candidate regardless of score — the optimizer enforces the 0.60 floor — use a more specific skill instead.
- You are in a dry-run context where you only want to see the decision without writing to disk — pass
dry_run=Trueto the runner
Inputs required
- The baseline EvalResult (from
karpathy-evaluationrun on the unmodified skill) - A list of candidate EvalResults (from
karpathy-evaluationrun on each passing candidate) - A list of MutationCandidate dicts (matching the candidate_evals by candidate_id)
- The skill_path to write the promoted content (if decision is PROMOTE and not dry-run)
- A dry_run flag (default: False for production runs, True for CI eval-only workflows)
Constraints
- Apply token policy before ranking: reject any candidate where token increase > 35% AND score improvement < 5%
- A candidate must score at least 0.60 (absolute floor) to be eligible for promotion
- A candidate must exceed the baseline score to be eligible for promotion — equal score is a REJECT
- Back up the existing SKILL.md as
SKILL.<timestamp>.bak.mdbefore writing the winner - The PromotionDecision must include reasoning that names the winner_id, mutation_type, score_delta, and token_delta_pct
Step-by-step workflow
- For each candidate EvalResult, compute token_increase_pct = (candidate_tokens - baseline_tokens) / baseline_tokens.
- Apply token policy: if token_increase_pct > 0.35 AND score_improvement < 0.05, mark as token_policy_passed = False and exclude from ranking.
- Sort remaining candidates by final_score descending.
- Select the top-scoring candidate as the winner.
- If winner.final_score <= baseline.final_score, return decision = REJECT with reasoning.
- If winner.final_score < 0.60, return decision = REJECT with reasoning.
- If not dry_run, write backup of existing skill and write winner content to skill_path.
- Return PromotionDecision with: winner_id, decision = PROMOTE, reasoning, score_delta, token_delta_pct, promoted_path.
Output contract
- A PromotionDecision dict with: winner_id (str or null), decision ("PROMOTE" or "REJECT"), reasoning (str), score_delta (float or null), token_delta_pct (float or null), promoted_path (str or null), token_policy_applied (bool), token_policy_rejections (list of str)
- If PROMOTE: the winning SKILL.md is written to skill_path and the previous version is backed up
- The decision is appended to
memory/score_history.jsonvia the Orchestrator
Validation checklist
- Confirm token_policy_rejections contains all candidate_ids where token_increase_pct > 0.35 and score_improvement < 0.05
- Confirm winner_id is null when decision is REJECT
- Confirm the backup file exists at
SKILL.<timestamp>.bak.mdwhen decision is PROMOTE and not dry_run - Run
python -m runners.promotion_runner --dry-run ...and confirm no file is written
Related skills
karpathy-regression— must complete before calling this skillkarpathy-evaluation— provides the EvalResult inputs this skill needskarpathy-guidelines— for understanding why a candidate scored as it did
References
Real example
Historical static snapshot: three candidates pass regression for the flutter deep link skill. Candidate baseline-token_budget is rejected by token policy (+40% tokens, only +2% score). Of the remaining two, baseline-decomposition_steps scores 0.7834 (+0.06 above baseline) and baseline-verification_ordering scores 0.7612 (+0.04). The optimizer promotes baseline-decomposition_steps, writes the backup, and returns reasoning: "Candidate baseline-decomposition_steps improved score by +0.0600 (+8.3%) with token delta +2.1%."
Real file output sample
overlays/agent-karpathy/skills/evaluation/SKILL.20260417T120000.bak.md
overlays/agent-karpathy/skills/evaluation/SKILL.md
reports/latest_report.md
04 同仓库更多 Skills
toolkit-i18n-operator
Use the toolkit-i18n CLI for repeated Flutter localization workflows with compact outputs, CSV validation, key coverage reporting, generated JSON exports, and safe non-mutating behavior.
toolkit-design-operator
Use the toolkit-design CLI for repeated repository workflows.
toolkit-ci-operator
Use the toolkit-ci CLI for repeated repository workflows.
toolkit-arch-operator
Use the toolkit-arch CLI for repeated repository workflows, including Flutter i18n usage scans, layer checks, and localization coverage.
web-testing
暂无简介
web-state
暂无简介