⚡AgentSkills
📊 Data Science & Analytics · Critical Consumption

Read other people's A/B tests critically

Validity checklist catching underpowered tests, peeking, and metric tricks in published results.

intermediate~25 minData ScientistsAnalystsResearch Scientists

Steps

  1. 1Check sample size math against claimed effect size first
  2. 2Look for stopping rules: when did they decide it was over?
  3. 3Verify primary metric matches headline claim exactly
  4. 4Inspect guardrails: what degraded while the winner won?
  5. 5Segment sanity: does the effect survive obvious subgroups?
  6. 6Estimate business impact yourself from raw numbers provided

Common Pitfalls

  • ▲Headline metrics switched post-hoc from preregistered ones
  • ▲Relative percentages hiding absolute effects near zero

Commands

Install with skills CLI
$ npx skills add aniruddhaadak80/skills --skill critical-consumption-ab-test-reading-checklist
Install globally
$ npx skills add aniruddhaadak80/skills --skill critical-consumption-ab-test-reading-checklist -g

Tags

#experimentation#critical-thinking#statistics#data-science#critical-consumption