AGENT SKILL

holdback-experiment-design

Design degradation holdbacks and long-term cumulative holdbacks for product experiments and feature rollouts. Use when a team needs a long-term counterfactual, wants to measure delayed impact after launch, is worried about metric degradation over time, needs to decide holdback size or duration, or must weigh the user/business cost of withholding a feature.

License 许可
MIT
最近更新
2026-11-14
Stars
★ 1.6k
Forks
491

01 前言 Front Matter

name
holdback-experiment-design
license
MIT
metadata
{ "tags": "practical-ab-testing,ab-testing,experimentation,holdbacks,product-analytics", "version": "0.1.0", "category": "Product Management", "displayName": "Holdback Experiment Design" }
description
Design degradation holdbacks and long-term cumulative holdbacks for product experiments and feature rollouts. Use when a team needs a long-term counterfactual, wants to measure delayed impact after launch, is worried about metric degradation over time, needs to decide holdback size or duration, or must weigh the user/business cost of withholding a feature.
compatibility
Codex, Claude Code, and other Agent Skills-compatible clients.

02 资源文件 5

03 SKILL.md 正文

Holdback Experiment Design

Use this skill to plan holdbacks that preserve a comparison group after a feature rollout. Holdbacks are useful when effects may appear later, accumulate, or degrade after launch.

Source Traceability

Primary source: Practical A/B Testing by Leemay Nassery. Guidance is transformed and paraphrased from chapter 3 lines 2487-2836. Experiment-type context comes from chapter 3 lines 2013-2486.

Related Advanced Skills

  • long-term-impact-evaluation: use before detailed holdback design when the team should compare holdbacks against post-period analysis, continuous monitoring, CLV models, or hybrid methods.
  • trustworthy-experiment-insights: use when deciding whether long-term or holdback evidence is credible enough to drive a product decision.

Reference Routing

Need Read
Holdback concepts references/core/knowledge.md
Design and cost rules references/core/rules.md
Scenario examples references/core/examples.md
Step-by-step planning workflows/design-holdback.md

Workflow

  1. State why short-term A/B evidence is insufficient.
  2. Choose degradation or long-term cumulative holdback.
  3. Define who remains withheld, for how long, and from what experience.
  4. Select long-term metrics and guardrails.
  5. Estimate the user, business, and ethical cost of withholding.
  6. Define monitoring cadence, exit criteria, and communication plan.

Output Format

# Holdback Plan

## Purpose
[What long-term question this holdback answers.]

## Holdback Type
[Degradation | Long-term cumulative]

## Population And Duration
- Holdback population:
- Rollout population:
- Duration:
- Removal criteria:

## Metrics
| Metric | Role | Readout Cadence | Concern |
|--------|------|-----------------|---------|

## Cost Of Withholding
- User cost:
- Business cost:
- Ethical or trust concern:

## Decision Rules
- Continue holdback if:
- End holdback if:
- Escalate if:

Quality Bar

  • Do not create a holdback without a specific long-term question.
  • Do not withhold a clearly valuable feature longer than the question requires.
  • Do not ignore the opportunity cost to held-back users.
  • Monitor guardrails while the holdback is active.

04 同仓库更多 Skills

hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-review-with-annotations

Create, inspect, reply to, update, or resolve situated Worktable annotations. Use when feedback belongs to a Doc section, block, text range, or HTML Doc, or when the user asks to address existing review feedback and coordinate a requested artifact edit.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-find-and-synthesize-context

Find, assess, and synthesize context from Worktable Docs and Records. Use when the user asks what a workspace contains, wants workspace-backed research, needs current facts reconciled across artifacts, or requests a cited synthesis without changing workspace state.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-create-or-update-docs

Create, edit, rename, or maintain narrative Worktable Docs. Use for plans, decisions, briefs, notes, research syntheses, and other durable prose when the user requests a new artifact or a safe update to an existing one.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-create-or-manage-records

Organize repeated structured information as durable Worktable Records. Use for research datasets, inventories, directories, pipelines, trackers, and other collections whose items need identity, fields, filtering, links, validation, or individual updates.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-create-or-manage-interactive-html

Create or improve durable visual and interactive artifacts in Worktable. Use for dashboards, trackers, calculators, mini apps, slide presentations, and other requests that benefit from a polished HTML Doc.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-collaborate-threads

Read, answer, and hand off messages through durable Worktable threads with users or other authenticated participants. Use when the user asks to check addressed messages, reply to a participant, continue a thread, request another agent's help, wait briefly for a response, or watch for comments and messages sent to the agent.

★ 1.6k