You Are Not Measuring What You Think You Are Measuring — LessWrong
Two laws of experiment design: First, you are not measuring what you think you are measuring. Second, if you measure enough different stuff, you migh…
x You Are Not Measuring What You Think You Are Measuring — LessWrong Best of LessWrong 2022 Practice & Philosophy of Science World Modeling Curated 443 You Are Not Measuring What You Think You Are Measuring by johnswentworth 20th Sep 2022 10 min read 45 443 Eight years ago, I worked as a data scientist at a startup, and we wanted to optimize our sign-up flow. We A/B tested lots of different changes, and occasionally found something which would boost (or reduce) click-through rates by 10% or so. Then one week I was puzzling over a discrepancy in the variance of our daily signups. Eventually I s
related reading
- How much do you believe your results? — LessWronglesswrong.com
- Study Guide — LessWronglesswrong.com
- Humans are not automatically strategic — LessWronglesswrong.com
- Your intuitions are not magic — LessWronglesswrong.com
- What Money Cannot Buy — LessWronglesswrong.com
- How to Measure Anything — LessWronglesswrong.com
- Science in a High-Dimensional World — LessWronglesswrong.com
- How To Write Quickly While Maintaining Epistemic Rigor — LessWronglesswrong.com
- The Best of LessWrong — LessWronglesswrong.com
- The correct response to uncertainty is *not* half-speed — LessWronglesswrong.com
- Bad Problems Don't Stop Being Bad Because Somebody's Wrong About Fault Analysis — LessWronglesswrong.com
- Humans are not automatically strategic — LessWronglesswrong.com