[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6MCDFS8FBFSD8SJTM50G7GG":3,"skill-same:01M6MCDFS8FBFSD8SJTM50G7GG":113},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":34,"publishedAt":56,"linkOnly":57,"markdownContent":58,"markdownBody":59,"frontMatter":60,"refName":67,"commitSha":68,"skillDirectory":69,"forksCount":70,"repositoryHtmlUrl":71,"repositoryHomepageUrl":72,"ownerHtmlUrl":73,"ownerBlogUrl":74,"ownerDescription":75,"updatedAt":76,"install":77,"resources":90},"01M6MCDFS8FBFSD8SJTM50G7GG","holdback-experiment-design",null,"Design degradation holdbacks and long-term cumulative holdbacks for product experiments and feature rollouts. Use when a team needs a long-term counterfactual, wants to measure delayed impact after launch, is worried about metric degradation over time, needs to decide holdback size or duration, or must weigh the user\u002Fbusiness cost of withholding a feature.","hashgraph-online\u002Fawesome-codex-plugins","hashgraph-online","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F160371962?v=4","01M6CNB6NRJVNNGN14TR01Q6D3","HOL (Hashgraph Online)","01M6JT1HSKKG9V68KQAQGBA7XP",1575,"MIT","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FLVTD-LLC\u002Fskills\u002Fskills\u002Fholdback-experiment-design\u002FSKILL.md",[18,23,26,31],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000015","视觉设计","SCENARIO","使用场景",{"categoryNo":24,"categoryName":25,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000023","Skill与智能体开发",{"categoryNo":27,"categoryName":28,"taxonomyType":29,"taxonomyTypeDesc":30},"01JBSEEDCAT000000000000031","产品经理","PROFESSION","适用职业",{"categoryNo":32,"categoryName":33,"taxonomyType":29,"taxonomyTypeDesc":30},"01JBSEEDCAT000000000000032","设计师",[35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55],"design","holdback experiment design","experiment design","experiment","when","testing","metric","ab-testing","experimentation","impact","ab","feature","time","practical","rollouts","decide","team","product","code","analytics","experiments","2026-11-10 10:12:30",false,"---\nname: holdback-experiment-design\ndescription: Design degradation holdbacks and long-term cumulative holdbacks for product experiments and feature rollouts. Use when a team needs a long-term counterfactual, wants to measure delayed impact after launch, is worried about metric degradation over time, needs to decide holdback size or duration, or must weigh the user\u002Fbusiness cost of withholding a feature.\nlicense: MIT\ncompatibility: Codex, Claude Code, and other Agent Skills-compatible clients.\nmetadata:\n  version: \"0.1.0\"\n  displayName: Holdback Experiment Design\n  category: Product Management\n  tags: practical-ab-testing,ab-testing,experimentation,holdbacks,product-analytics\n---\n\n# Holdback Experiment Design\n\nUse this skill to plan holdbacks that preserve a comparison group after a\nfeature rollout. Holdbacks are useful when effects may appear later, accumulate,\nor degrade after launch.\n\n## Source Traceability\n\nPrimary source: Practical A\u002FB Testing by Leemay Nassery. Guidance is\ntransformed and paraphrased from chapter 3 lines 2487-2836. Experiment-type\ncontext comes from chapter 3 lines 2013-2486.\n\n## Related Advanced Skills\n\n- `long-term-impact-evaluation`: use before detailed holdback design when the\n  team should compare holdbacks against post-period analysis, continuous\n  monitoring, CLV models, or hybrid methods.\n- `trustworthy-experiment-insights`: use when deciding whether long-term or\n  holdback evidence is credible enough to drive a product decision.\n\n## Reference Routing\n\n| Need | Read |\n|------|------|\n| Holdback concepts | `references\u002Fcore\u002Fknowledge.md` |\n| Design and cost rules | `references\u002Fcore\u002Frules.md` |\n| Scenario examples | `references\u002Fcore\u002Fexamples.md` |\n| Step-by-step planning | `workflows\u002Fdesign-holdback.md` |\n\n## Workflow\n\n1. State why short-term A\u002FB evidence is insufficient.\n2. Choose degradation or long-term cumulative holdback.\n3. Define who remains withheld, for how long, and from what experience.\n4. Select long-term metrics and guardrails.\n5. Estimate the user, business, and ethical cost of withholding.\n6. Define monitoring cadence, exit criteria, and communication plan.\n\n## Output Format\n\n```markdown\n# Holdback Plan\n\n## Purpose\n[What long-term question this holdback answers.]\n\n## Holdback Type\n[Degradation | Long-term cumulative]\n\n## Population And Duration\n- Holdback population:\n- Rollout population:\n- Duration:\n- Removal criteria:\n\n## Metrics\n| Metric | Role | Readout Cadence | Concern |\n|--------|------|-----------------|---------|\n\n## Cost Of Withholding\n- User cost:\n- Business cost:\n- Ethical or trust concern:\n\n## Decision Rules\n- Continue holdback if:\n- End holdback if:\n- Escalate if:\n```\n\n## Quality Bar\n\n- Do not create a holdback without a specific long-term question.\n- Do not withhold a clearly valuable feature longer than the question requires.\n- Do not ignore the opportunity cost to held-back users.\n- Monitor guardrails while the holdback is active.\n","# Holdback Experiment Design\n\nUse this skill to plan holdbacks that preserve a comparison group after a\nfeature rollout. Holdbacks are useful when effects may appear later, accumulate,\nor degrade after launch.\n\n## Source Traceability\n\nPrimary source: Practical A\u002FB Testing by Leemay Nassery. Guidance is\ntransformed and paraphrased from chapter 3 lines 2487-2836. Experiment-type\ncontext comes from chapter 3 lines 2013-2486.\n\n## Related Advanced Skills\n\n- `long-term-impact-evaluation`: use before detailed holdback design when the\n  team should compare holdbacks against post-period analysis, continuous\n  monitoring, CLV models, or hybrid methods.\n- `trustworthy-experiment-insights`: use when deciding whether long-term or\n  holdback evidence is credible enough to drive a product decision.\n\n## Reference Routing\n\n| Need | Read |\n|------|------|\n| Holdback concepts | `references\u002Fcore\u002Fknowledge.md` |\n| Design and cost rules | `references\u002Fcore\u002Frules.md` |\n| Scenario examples | `references\u002Fcore\u002Fexamples.md` |\n| Step-by-step planning | `workflows\u002Fdesign-holdback.md` |\n\n## Workflow\n\n1. State why short-term A\u002FB evidence is insufficient.\n2. Choose degradation or long-term cumulative holdback.\n3. Define who remains withheld, for how long, and from what experience.\n4. Select long-term metrics and guardrails.\n5. Estimate the user, business, and ethical cost of withholding.\n6. Define monitoring cadence, exit criteria, and communication plan.\n\n## Output Format\n\n```markdown\n# Holdback Plan\n\n## Purpose\n[What long-term question this holdback answers.]\n\n## Holdback Type\n[Degradation | Long-term cumulative]\n\n## Population And Duration\n- Holdback population:\n- Rollout population:\n- Duration:\n- Removal criteria:\n\n## Metrics\n| Metric | Role | Readout Cadence | Concern |\n|--------|------|-----------------|---------|\n\n## Cost Of Withholding\n- User cost:\n- Business cost:\n- Ethical or trust concern:\n\n## Decision Rules\n- Continue holdback if:\n- End holdback if:\n- Escalate if:\n```\n\n## Quality Bar\n\n- Do not create a holdback without a specific long-term question.\n- Do not withhold a clearly valuable feature longer than the question requires.\n- Do not ignore the opportunity cost to held-back users.\n- Monitor guardrails while the holdback is active.\n",{"name":5,"license":15,"metadata":61,"description":7,"compatibility":66},{"tags":62,"version":63,"category":64,"displayName":65},"practical-ab-testing,ab-testing,experimentation,holdbacks,product-analytics","0.1.0","Product Management","Holdback Experiment Design","Codex, Claude Code, and other Agent Skills-compatible clients.","main","2b63f856ce19c2a797929cacabd9e0fc936812fa","plugins\u002FLVTD-LLC\u002Fskills\u002Fskills\u002Fholdback-experiment-design",491,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins","https:\u002F\u002Fhol.org\u002Fplugins\u002Fbest-codex-plugins","https:\u002F\u002Fgithub.com\u002Fhashgraph-online","https:\u002F\u002Fhol.org","Open-source standards and developer tools for interoperable AI agents: discovery, identity, communication, payments, privacy and security.","2026-11-14 14:06:13",{"source":8,"skillName":5,"commands":78,"prompt":88,"zipAvailable":89},[79,82,85],{"tool":80,"command":81},"npx","npx skills add hashgraph-online\u002Fawesome-codex-plugins --skill holdback-experiment-design",{"tool":83,"command":84},"bunx","bunx skills add hashgraph-online\u002Fawesome-codex-plugins --skill holdback-experiment-design",{"tool":86,"command":87},"pnpm","pnpm dlx skills add hashgraph-online\u002Fawesome-codex-plugins --skill holdback-experiment-design","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FLVTD-LLC\u002Fskills\u002Fskills\u002Fholdback-experiment-design\u002FSKILL.md",true,[91,97,101,105,109],{"path":92,"resourceType":93,"resourceTypeDesc":94,"sizeBytes":95,"sourceHtmlUrl":96},"guidelines.md","REFERENCE","参考文档",1363,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FLVTD-LLC\u002Fskills\u002Fskills\u002Fholdback-experiment-design\u002Fguidelines.md",{"path":98,"resourceType":93,"resourceTypeDesc":94,"sizeBytes":99,"sourceHtmlUrl":100},"references\u002Fcore\u002Fexamples.md",1467,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FLVTD-LLC\u002Fskills\u002Fskills\u002Fholdback-experiment-design\u002Freferences\u002Fcore\u002Fexamples.md",{"path":102,"resourceType":93,"resourceTypeDesc":94,"sizeBytes":103,"sourceHtmlUrl":104},"references\u002Fcore\u002Fknowledge.md",1803,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FLVTD-LLC\u002Fskills\u002Fskills\u002Fholdback-experiment-design\u002Freferences\u002Fcore\u002Fknowledge.md",{"path":106,"resourceType":93,"resourceTypeDesc":94,"sizeBytes":107,"sourceHtmlUrl":108},"references\u002Fcore\u002Frules.md",1632,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FLVTD-LLC\u002Fskills\u002Fskills\u002Fholdback-experiment-design\u002Freferences\u002Fcore\u002Frules.md",{"path":110,"resourceType":93,"resourceTypeDesc":94,"sizeBytes":111,"sourceHtmlUrl":112},"workflows\u002Fdesign-holdback.md",1877,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FLVTD-LLC\u002Fskills\u002Fskills\u002Fholdback-experiment-design\u002Fworkflows\u002Fdesign-holdback.md",[114,123,131,139,147,155],{"skillNo":115,"name":116,"nameZh":6,"description":117,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":118,"sourceHtmlUrl":119,"categories":120,"keywords":121,"publishedAt":122},"01M7DV332YQMGD8WTRJ9DZQXTB","worktable-review-with-annotations","Create, inspect, reply to, update, or resolve situated Worktable annotations. Use when feedback belongs to a Doc section, block, text range, or HTML Doc, or when the user asks to address existing review feedback and coordinate a requested artifact edit.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-review-with-annotations\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-review-with-annotations\u002FSKILL.md",[],[],"2026-11-15 14:19:17",{"skillNo":124,"name":125,"nameZh":6,"description":126,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":127,"sourceHtmlUrl":128,"categories":129,"keywords":130,"publishedAt":122},"01M7DV2YGHCZVBEB1AHYB6CGHE","worktable-find-and-synthesize-context","Find, assess, and synthesize context from Worktable Docs and Records. Use when the user asks what a workspace contains, wants workspace-backed research, needs current facts reconciled across artifacts, or requests a cited synthesis without changing workspace state.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-find-and-synthesize-context\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-find-and-synthesize-context\u002FSKILL.md",[],[],{"skillNo":132,"name":133,"nameZh":6,"description":134,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":135,"sourceHtmlUrl":136,"categories":137,"keywords":138,"publishedAt":122},"01M7DV2TQN1H11ME2SAF9MXN7V","worktable-create-or-update-docs","Create, edit, rename, or maintain narrative Worktable Docs. Use for plans, decisions, briefs, notes, research syntheses, and other durable prose when the user requests a new artifact or a safe update to an existing one.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-update-docs\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-update-docs\u002FSKILL.md",[],[],{"skillNo":140,"name":141,"nameZh":6,"description":142,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":143,"sourceHtmlUrl":144,"categories":145,"keywords":146,"publishedAt":122},"01M7DV2NE452NEA8TXATKSNAJH","worktable-create-or-manage-records","Organize repeated structured information as durable Worktable Records. Use for research datasets, inventories, directories, pipelines, trackers, and other collections whose items need identity, fields, filtering, links, validation, or individual updates.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-records\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-records\u002FSKILL.md",[],[],{"skillNo":148,"name":149,"nameZh":6,"description":150,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":151,"sourceHtmlUrl":152,"categories":153,"keywords":154,"publishedAt":122},"01M7DV2HENRBR2KSCVM064WH0Y","worktable-create-or-manage-interactive-html","Create or improve durable visual and interactive artifacts in Worktable. Use for dashboards, trackers, calculators, mini apps, slide presentations, and other requests that benefit from a polished HTML Doc.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-interactive-html\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-create-or-manage-interactive-html\u002FSKILL.md",[],[],{"skillNo":156,"name":157,"nameZh":6,"description":158,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":159,"sourceHtmlUrl":160,"categories":161,"keywords":162,"publishedAt":122},"01M7DV2CADR5R7RQFFYPNAGR9Y","worktable-collaborate-threads","Read, answer, and hand off messages through durable Worktable threads with users or other authenticated participants. Use when the user asks to check addressed messages, reply to a participant, continue a thread, request another agent's help, wait briefly for a response, or watch for comments and messages sent to the agent.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-collaborate-threads\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fworktable\u002Fworktable-plugin\u002Fskills\u002Fworktable-collaborate-threads\u002FSKILL.md",[],[]]