[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6MCHVCCZ2F4J98EC4182QQE":3,"skill-same:01M6MCHVCCZ2F4J98EC4182QQE":107},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":28,"publishedAt":50,"linkOnly":51,"markdownContent":52,"markdownBody":53,"frontMatter":54,"refName":61,"commitSha":62,"skillDirectory":63,"forksCount":64,"repositoryHtmlUrl":65,"repositoryHomepageUrl":66,"ownerHtmlUrl":67,"ownerBlogUrl":68,"ownerDescription":69,"updatedAt":70,"install":71,"resources":84},"01M6MCHVCCZ2F4J98EC4182QQE","ml-experiment-evaluation",null,"Plan evaluation strategies for machine-learning product changes. Use when deciding between offline evaluation, interleaving, online A\u002FB tests, multi-armed bandits, or model filtering for ranking, recommendation, search, personalization, or other ML-powered user experiences.","hashgraph-online\u002Fawesome-codex-plugins","hashgraph-online","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F160371962?v=4","01M6CNB6NRJVNNGN14TR01Q6D3","HOL (Hashgraph Online)","01M6JT1HSKKG9V68KQAQGBA7XP",1575,"MIT","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FLVTD-LLC\u002Fskills\u002Fskills\u002Fml-experiment-evaluation\u002FSKILL.md",[18,23],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",{"categoryNo":24,"categoryName":25,"taxonomyType":26,"taxonomyTypeDesc":27},"01JBSEEDCAT000000000000031","产品经理","PROFESSION","适用职业",[29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49],"ml","ml experiment evaluation","Evaluation","experiment","when","testing","for","ab-testing","experimentation","ranking","ab","strategies","personalization","next","practical","model","plan","offline","search","product","code","2026-11-10 10:12:30",false,"---\nname: ml-experiment-evaluation\ndescription: Plan evaluation strategies for machine-learning product changes. Use when deciding between offline evaluation, interleaving, online A\u002FB tests, multi-armed bandits, or model filtering for ranking, recommendation, search, personalization, or other ML-powered user experiences.\nlicense: MIT\ncompatibility: Codex, Claude Code, and other Agent Skills-compatible clients.\nmetadata:\n  version: \"0.1.0\"\n  displayName: ML Experiment Evaluation\n  category: Product Management\n  tags: practical-ab-testing,next-level-ab-testing,ab-testing,experimentation,machine-learning\n---\n\n# ML Experiment Evaluation\n\nUse this skill to choose how to evaluate machine-learning product changes before\nthey consume live experiment traffic or affect users. It focuses on offline\nevaluation, offline-online correlation, interleaving, model filtering, and when\nclassic A\u002FB testing or adaptive strategies are justified.\n\n## Source Traceability\n\nPrimary source: *Next-Level A\u002FB Testing* by Leemay Nassery. Guidance is\ntransformed and paraphrased from Chapter 4 on offline evaluation, offline-online\ncorrelation, multi-armed bandits, and interleaving for rankers.\n\nRelated skills:\n\n- `experiment-sensitivity-optimization` for reducing live variants and traffic.\n- `adaptive-experimentation-strategy` for bandits and dynamic allocation.\n- `ab-test-design-brief` for standard online A\u002FB test planning.\n\n## Reference Routing\n\n| Need | Read |\n|------|------|\n| ML evaluation concepts | `references\u002Fcore\u002Fknowledge.md` |\n| Selection and validation rules | `references\u002Fcore\u002Frules.md` |\n| Evaluation strategy examples | `references\u002Fcore\u002Fexamples.md` |\n| Step-by-step evaluation plan | `workflows\u002Fchoose-ml-evaluation-strategy.md` |\n\n## Workflow\n\n1. State the model change and product decision.\n2. Identify the user harm or trust risk if a poor model reaches production.\n3. Choose the lowest-cost evaluation that can filter bad candidates.\n4. Check offline metrics and whether they correlate with online outcomes.\n5. Use interleaving when ranker comparison needs high sensitivity with fewer\n   users.\n6. Escalate to online A\u002FB testing or adaptive testing only when live evidence is\n   needed and infrastructure can support it.\n\n## Output Format\n\n```markdown\n# ML Evaluation Strategy\n\n## Model Decision\n[What model or ranking decision must be made.]\n\n## Recommended Evaluation Path\n[Offline only | Offline then A\u002FB | Interleaving | A\u002FB test | Adaptive strategy]\n\n## Why\n- Product risk:\n- Offline signal available:\n- Online evidence needed:\n- Traffic or capacity constraint:\n\n## Metrics\n| Metric | Offline\u002FOnline | Role | Concern |\n|--------|----------------|------|---------|\n\n## Implementation Notes\n- Data needed:\n- Logging needed:\n- Correlation check:\n- Rollout guardrails:\n```\n\n## Quality Bar\n\n- Do not send poor offline candidates to live users just to get online evidence.\n- Do not trust offline metrics until their relationship to online outcomes is\n  understood.\n- Do not use interleaving unless the product has a ranking or choice context\n  where attribution can be logged.\n- Do not recommend adaptive methods without checking data freshness,\n  observability, and operational ownership.\n","# ML Experiment Evaluation\n\nUse this skill to choose how to evaluate machine-learning product changes before\nthey consume live experiment traffic or affect users. It focuses on offline\nevaluation, offline-online correlation, interleaving, model filtering, and when\nclassic A\u002FB testing or adaptive strategies are justified.\n\n## Source Traceability\n\nPrimary source: *Next-Level A\u002FB Testing* by Leemay Nassery. Guidance is\ntransformed and paraphrased from Chapter 4 on offline evaluation, offline-online\ncorrelation, multi-armed bandits, and interleaving for rankers.\n\nRelated skills:\n\n- `experiment-sensitivity-optimization` for reducing live variants and traffic.\n- `adaptive-experimentation-strategy` for bandits and dynamic allocation.\n- `ab-test-design-brief` for standard online A\u002FB test planning.\n\n## Reference Routing\n\n| Need | Read |\n|------|------|\n| ML evaluation concepts | `references\u002Fcore\u002Fknowledge.md` |\n| Selection and validation rules | `references\u002Fcore\u002Frules.md` |\n| Evaluation strategy examples | `references\u002Fcore\u002Fexamples.md` |\n| Step-by-step evaluation plan | `workflows\u002Fchoose-ml-evaluation-strategy.md` |\n\n## Workflow\n\n1. State the model change and product decision.\n2. Identify the user harm or trust risk if a poor model reaches production.\n3. Choose the lowest-cost evaluation that can filter bad candidates.\n4. Check offline metrics and whether they correlate with online outcomes.\n5. Use interleaving when ranker comparison needs high sensitivity with fewer\n   users.\n6. Escalate to online A\u002FB testing or adaptive testing only when live evidence is\n   needed and infrastructure can support it.\n\n## Output Format\n\n```markdown\n# ML Evaluation Strategy\n\n## Model Decision\n[What model or ranking decision must be made.]\n\n## Recommended Evaluation Path\n[Offline only | Offline then A\u002FB | Interleaving | A\u002FB test | Adaptive strategy]\n\n## Why\n- Product risk:\n- Offline signal available:\n- Online evidence needed:\n- Traffic or capacity constraint:\n\n## Metrics\n| Metric | Offline\u002FOnline | Role | Concern |\n|--------|----------------|------|---------|\n\n## Implementation Notes\n- Data needed:\n- Logging needed:\n- Correlation check:\n- Rollout guardrails:\n```\n\n## Quality Bar\n\n- Do not send poor offline candidates to live users just to get online evidence.\n- Do not trust offline metrics until their relationship to online outcomes is\n  understood.\n- Do not use interleaving unless the product has a ranking or choice context\n  where attribution can be logged.\n- Do not recommend adaptive methods without checking data freshness,\n  observability, and operational ownership.\n",{"name":5,"license":15,"metadata":55,"description":7,"compatibility":60},{"tags":56,"version":57,"category":58,"displayName":59},"practical-ab-testing,next-level-ab-testing,ab-testing,experimentation,machine-learning","0.1.0","Product Management","ML Experiment Evaluation","Codex, Claude Code, and other Agent Skills-compatible clients.","main","2b63f856ce19c2a797929cacabd9e0fc936812fa","plugins\u002FLVTD-LLC\u002Fskills\u002Fskills\u002Fml-experiment-evaluation",491,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins","https:\u002F\u002Fhol.org\u002Fplugins\u002Fbest-codex-plugins","https:\u002F\u002Fgithub.com\u002Fhashgraph-online","https:\u002F\u002Fhol.org","Open-source standards and developer tools for interoperable AI agents: discovery, identity, communication, payments, privacy and security.","2026-11-14 14:06:13",{"source":8,"skillName":5,"commands":72,"prompt":82,"zipAvailable":83},[73,76,79],{"tool":74,"command":75},"npx","npx skills add hashgraph-online\u002Fawesome-codex-plugins --skill ml-experiment-evaluation",{"tool":77,"command":78},"bunx","bunx skills add hashgraph-online\u002Fawesome-codex-plugins --skill ml-experiment-evaluation",{"tool":80,"command":81},"pnpm","pnpm dlx skills add hashgraph-online\u002Fawesome-codex-plugins --skill ml-experiment-evaluation","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FLVTD-LLC\u002Fskills\u002Fskills\u002Fml-experiment-evaluation\u002FSKILL.md",true,[85,91,95,99,103],{"path":86,"resourceType":87,"resourceTypeDesc":88,"sizeBytes":89,"sourceHtmlUrl":90},"guidelines.md","REFERENCE","参考文档",1786,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FLVTD-LLC\u002Fskills\u002Fskills\u002Fml-experiment-evaluation\u002Fguidelines.md",{"path":92,"resourceType":87,"resourceTypeDesc":88,"sizeBytes":93,"sourceHtmlUrl":94},"references\u002Fcore\u002Fexamples.md",2389,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FLVTD-LLC\u002Fskills\u002Fskills\u002Fml-experiment-evaluation\u002Freferences\u002Fcore\u002Fexamples.md",{"path":96,"resourceType":87,"resourceTypeDesc":88,"sizeBytes":97,"sourceHtmlUrl":98},"references\u002Fcore\u002Fknowledge.md",3271,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FLVTD-LLC\u002Fskills\u002Fskills\u002Fml-experiment-evaluation\u002Freferences\u002Fcore\u002Fknowledge.md",{"path":100,"resourceType":87,"resourceTypeDesc":88,"sizeBytes":101,"sourceHtmlUrl":102},"references\u002Fcore\u002Frules.md",2470,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FLVTD-LLC\u002Fskills\u002Fskills\u002Fml-experiment-evaluation\u002Freferences\u002Fcore\u002Frules.md",{"path":104,"resourceType":87,"resourceTypeDesc":88,"sizeBytes":105,"sourceHtmlUrl":106},"workflows\u002Fchoose-ml-evaluation-strategy.md",2610,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FLVTD-LLC\u002Fskills\u002Fskills\u002Fml-experiment-evaluation\u002Fworkflows\u002Fchoose-ml-evaluation-strategy.md",[108,117,125,133,141,149],{"skillNo":109,"name":110,"nameZh":6,"description":111,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":112,"sourceHtmlUrl":113,"categories":114,"keywords":115,"publishedAt":116},"01M7DV332YQMGD8WTRJ9DZQXTB","worktable-review-with-annotations","Create, inspect, reply to, update, or resolve situated Worktable annotations. Use when feedback belongs to a Doc section, block, text range, or HTML Doc, or when the user asks to address existing review feedback and coordinate a requested artifact edit.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-review-with-annotations\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-review-with-annotations\u002FSKILL.md",[],[],"2026-11-15 14:19:17",{"skillNo":118,"name":119,"nameZh":6,"description":120,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":121,"sourceHtmlUrl":122,"categories":123,"keywords":124,"publishedAt":116},"01M7DV2YGHCZVBEB1AHYB6CGHE","worktable-find-and-synthesize-context","Find, assess, and synthesize context from Worktable Docs and Records. Use when the user asks what a workspace contains, wants workspace-backed research, needs current facts reconciled across artifacts, or requests a cited synthesis without changing workspace state.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-find-and-synthesize-context\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-find-and-synthesize-context\u002FSKILL.md",[],[],{"skillNo":126,"name":127,"nameZh":6,"description":128,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":129,"sourceHtmlUrl":130,"categories":131,"keywords":132,"publishedAt":116},"01M7DV2TQN1H11ME2SAF9MXN7V","worktable-create-or-update-docs","Create, edit, rename, or maintain narrative Worktable Docs. Use for plans, decisions, briefs, notes, research syntheses, and other durable prose when the user requests a new artifact or a safe update to an existing one.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-update-docs\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-update-docs\u002FSKILL.md",[],[],{"skillNo":134,"name":135,"nameZh":6,"description":136,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":137,"sourceHtmlUrl":138,"categories":139,"keywords":140,"publishedAt":116},"01M7DV2NE452NEA8TXATKSNAJH","worktable-create-or-manage-records","Organize repeated structured information as durable Worktable Records. Use for research datasets, inventories, directories, pipelines, trackers, and other collections whose items need identity, fields, filtering, links, validation, or individual updates.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-records\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-records\u002FSKILL.md",[],[],{"skillNo":142,"name":143,"nameZh":6,"description":144,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":145,"sourceHtmlUrl":146,"categories":147,"keywords":148,"publishedAt":116},"01M7DV2HENRBR2KSCVM064WH0Y","worktable-create-or-manage-interactive-html","Create or improve durable visual and interactive artifacts in Worktable. Use for dashboards, trackers, calculators, mini apps, slide presentations, and other requests that benefit from a polished HTML Doc.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-interactive-html\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-interactive-html\u002FSKILL.md",[],[],{"skillNo":150,"name":151,"nameZh":6,"description":152,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":153,"sourceHtmlUrl":154,"categories":155,"keywords":156,"publishedAt":116},"01M7DV2CADR5R7RQFFYPNAGR9Y","worktable-collaborate-threads","Read, answer, and hand off messages through durable Worktable threads with users or other authenticated participants. Use when the user asks to check addressed messages, reply to a participant, continue a thread, request another agent's help, wait briefly for a response, or watch for comments and messages sent to the agent.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-collaborate-threads\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-collaborate-threads\u002FSKILL.md",[],[]]