ab-test-analysis
Analyze A/B tests end-to-end — data quality, SRM, metrics, segments, and decision recommendations with uncertainty. Use when reading experiment results and advising ship/iterate/kill decisions.
- License 许可
- MIT
- 最近更新
- 2026-07-22
- Stars
- ★ 7
- Forks
- 1
01 前言 Front Matter
- name
- ab-test-analysis
- description
- Analyze A/B tests end-to-end — data quality, SRM, metrics, segments, and decision recommendations with uncertainty. Use when reading experiment results and advising ship/iterate/kill decisions.
03 SKILL.md 正文
A/B Test Analysis
Overview
A/B test analysis turns experimental data into a decision. It requires quality checks first, pre-specified metrics, and resistance to story-seeking in segments.
When to Use
- Experiment readouts and decision meetings
- Diagnosing surprising or null results
- Comparing treatment effects with guardrails
- Post-hoc investigation (labeled as exploratory)
Core Practices
- Verify assignment integrity and SRM
- Check data completeness and exposure definitions
- Analyze primary metric with planned method
- Review guardrails for harm
- Treat segment cuts as exploratory unless pre-specified
- Recommend ship / iterate / abandon with effect size + uncertainty
Principles
- Never skip quality checks because the result “looks significant”
- Peeking and optional stopping distort inference
- Practical significance drives decisions, not p-values alone
- Null results are informative when powered adequately
Verification
- [ ] SRM and data quality passed
- [ ] Primary metric analysis matches the design
- [ ] Decision accounts for effect size and uncertainty
04 同仓库更多 Skills
compliance-monitoring
Design compliance monitoring and testing programs that verify controls work in practice and surface issues before regulators do. Use when building monitoring plans, compliance testing, or continuous control monitoring.
complaints-handling
Operate complaints-handling processes that meet regulatory standards — logging, timely resolution, root-cause analysis, and fair outcomes. Use when designing or improving regulated complaints frameworks or analyzing complaint themes for conduct risk.
aml-kyc-basics
Apply AML/KYC fundamentals — customer due diligence, monitoring, suspicious activity escalation, and recordkeeping — for financial crime compliance. Use when designing onboarding controls, transaction monitoring concepts, or AML program components in financial services.
ai-model-compliance
Apply compliance controls to AI and automated decision systems — transparency, fairness, human oversight, documentation, and regulatory alignment. Use when deploying AI in regulated processes (credit, fraud, customer decisions) or responding to AI governance expectations.
data-quality-management
Manage data quality end-to-end — dimensions, rules, remediation, and prevention — beyond one-off cleaning scripts. Use when building sustained data quality programs or fixing systemic accuracy, completeness, or consistency problems.
data-lineage
Capture and use data lineage — where data comes from, how it transforms, and what depends on it — for impact analysis, debugging, and trust. Use when tracing metric origins, assessing change impact, or diagnosing broken pipelines.