A/B testing mistakes I learned the hard way - PostHog
posthog.com · 1,818 words · saved by 1 readers
Running experiments is equal parts powerful and terrifying. Powerful because you can validate changes that will transform your product for the better…
A/B testing mistakes I learned the hard way Lior Neu-ner Aug 06, 2024 Product engineers , Experiments Contents 1. Including unaffected users in your experiment 2. Only viewing results in aggregate (aka Simpson's paradox) 3. Conducting an experiment without a predetermined duration 4. Running an experiment without testing it first 5. Neglecting counter metrics 6. Not accounting for seasonality 7. Testing an unclear hypothesis 8. Relying too much on A/B tests for decision-making Conclusion Running experiments is equal parts powerful and terrifying. Powerful because you can validate changes that
related reading
- 5 Mistakes To Avoid When Running A/B Tests - Issue 87dataanalysis.substack.com
- The top 3 mistakes that make your A/B test results invalidconversion.com
- P-Hacking in Startups | Hacker Newsnews.ycombinator.com
- Mediocre Success Is Worse Than Outright Failureevery.to
- Detecting interference: An A/B test of A/B testslinkedin.com
- The Perils and Benefits of AB Testingcaseyaccidental.com
- How to improve product sense: A practical guide | Austin Yangaustinyang.co
- 🧠 106 Cognitive Biases & Principles That Affect Your UX | Growth.Designgrowth.design
- How do product analytics work?technically.substack.com
- Fake Door Testing - How it Works, Benefits & Risks | Chameleonchameleon.io
- Be good-argument-driven, not data-driven - Richard Marmorsteintwitchard.github.io
- What The Heck is... A Business Experiment?linkedin.com