trustworthy-experiment-insights
Assess whether experiment results are credible enough to influence product decisions. Use when checking false positive or false negative risk, underpowered metrics, suspiciously large lifts, replication needs, meta-analysis, stratified sampling, covariate adjustment, or whether A/B test insights should be trusted.
- License 许可
- MIT
- 最近更新
- 2026-11-14
- Stars
- ★ 1.6k
- Forks
- 491
01 前言 Front Matter
- name
- trustworthy-experiment-insights
- license
- MIT
- metadata
- { "tags": "practical-ab-testing,next-level-ab-testing,ab-testing,experimentation,analytics", "version": "0.1.0", "category": "Product Management", "displayName": "Trustworthy Experiment Insights" }
- description
- Assess whether experiment results are credible enough to influence product decisions. Use when checking false positive or false negative risk, underpowered metrics, suspiciously large lifts, replication needs, meta-analysis, stratified sampling, covariate adjustment, or whether A/B test insights should be trusted.
- compatibility
- Codex, Claude Code, and other Agent Skills-compatible clients.
02 资源文件 5
- 参考文档guidelines.md1.8 KB
- 参考文档references/core/examples.md2.5 KB
- 参考文档references/core/knowledge.md3.1 KB
- 参考文档references/core/rules.md2.4 KB
- 参考文档workflows/review-experiment-credibility.md2.5 KB
03 SKILL.md 正文
Trustworthy Experiment Insights
Use this skill to decide whether an experiment result is believable enough to shape a product or engineering decision. It focuses on false positives, false negatives, power, replication, meta-analysis, stratified sampling, covariate adjustment, and suspicious result review.
Source Traceability
Primary source: Next-Level A/B Testing by Leemay Nassery. Guidance is transformed and paraphrased from Chapter 6 on false positives and negatives, meta-analysis, metric sensitivity, stratified random sampling, covariate adjustments, replication, longer runs, and statistical power.
Related skills:
ab-test-results-readoutfor standard experiment reporting.experiment-sensitivity-optimizationfor improving precision before or during experiment design.experiment-verification-monitoringfor operational validity checks.
Reference Routing
| Need | Read |
|---|---|
| Insight-quality concepts | references/core/knowledge.md |
| Credibility and follow-up rules | references/core/rules.md |
| Result-review scenarios | references/core/examples.md |
| Step-by-step credibility review | workflows/review-experiment-credibility.md |
Workflow
- Confirm the experiment was operationally valid enough to interpret.
- Check power, practical significance, and whether metrics were underpowered.
- Look for false positive risk: suspicious lift, many comparisons, early stop, weak prior, or contradiction with prior experiments.
- Look for false negative risk: noisy metrics, small sample, low sensitivity, or over-broad metric choice.
- Compare with similar experiments or run meta-analysis when available.
- Recommend launch, replicate, extend, investigate, or reject the result.
Output Format
# Experiment Insight Credibility Review
## Result Under Review
[Experiment, metric, observed result, and proposed decision.]
## Credibility Assessment
[Trust | Trust with caveats | Replicate | Extend | Investigate | Do not trust]
## Evidence
| Check | Finding | Risk |
|-------|---------|------|
## Follow-Up
- Replication needed:
- Longer run needed:
- Meta-analysis/comparison:
- Variance reduction opportunity:
## Decision Guidance
[What decision can be made now, and what should wait.]
Quality Bar
- Do not celebrate a result before checking whether it could be a false positive.
- Do not dismiss a flat result before checking power and sensitivity.
- Do not compare against prior experiments without noting differences in population, metric, design, and timing.
- Do not use statistical checks to hide operational failures; verify experiment health first.
04 同仓库更多 Skills
worktable-review-with-annotations
Create, inspect, reply to, update, or resolve situated Worktable annotations. Use when feedback belongs to a Doc section, block, text range, or HTML Doc, or when the user asks to address existing review feedback and coordinate a requested artifact edit.
worktable-find-and-synthesize-context
Find, assess, and synthesize context from Worktable Docs and Records. Use when the user asks what a workspace contains, wants workspace-backed research, needs current facts reconciled across artifacts, or requests a cited synthesis without changing workspace state.
worktable-create-or-update-docs
Create, edit, rename, or maintain narrative Worktable Docs. Use for plans, decisions, briefs, notes, research syntheses, and other durable prose when the user requests a new artifact or a safe update to an existing one.
worktable-create-or-manage-records
Organize repeated structured information as durable Worktable Records. Use for research datasets, inventories, directories, pipelines, trackers, and other collections whose items need identity, fields, filtering, links, validation, or individual updates.
worktable-create-or-manage-interactive-html
Create or improve durable visual and interactive artifacts in Worktable. Use for dashboards, trackers, calculators, mini apps, slide presentations, and other requests that benefit from a polished HTML Doc.
worktable-collaborate-threads
Read, answer, and hand off messages through durable Worktable threads with users or other authenticated participants. Use when the user asks to check addressed messages, reply to a participant, continue a thread, request another agent's help, wait briefly for a response, or watch for comments and messages sent to the agent.