You Are Not Measuring What You Think You Are Measuring — LessWrong
Two laws of experiment design: First, you are not measuring what you think you are measuring. Second, if you measure enough different stuff, you migh…
x You Are Not Measuring What You Think You Are Measuring — LessWrong Best of LessWrong 2022 Practice & Philosophy of Science World Modeling Curated 443 You Are Not Measuring What You Think You Are Measuring by johnswentworth 20th Sep 2022 10 min read 45 443 Eight years ago, I worked as a data scientist at a startup, and we wanted to optimize our sign-up flow. We A/B tested lots of different changes, and occasionally found something which would boost (or reduce) click-through rates by 10% or so. Then one week I was puzzling over a discrepancy in the variance of our daily signups. Eventually I s
related reading
- How to Measure Anything — LessWronglesswrong.com
- 2. A brief introduction to research design — Learning Statistics with Pythonethanweed.github.io
- Research Designs | Nobanobaproject.com
- Cargo Cult Sciencecalteches.library.caltech.edu
- Getting Eulered | Slate Star Codexslatestarcodex.com
- Goodhart's Law and Why Measurement is Hard — Ribbonfarmribbonfarm.com
- How much do you believe your results? — LessWronglesswrong.com
- Self-experiment risk-taking interview, by Elizabeth van Nostrand, Gwern · Gwern.netgwern.net
- Beware The Man Of One Studyslatestarcodex.com
- Science in a High-Dimensional World — LessWronglesswrong.com
- 1 Introduction – <span style='font-weight: 700'>Causal Inference</span><br/> <i style='color: #00b7ff'>The Mixtape</i>mixtape.scunning.com
- 2 Theories – Experimentologyexperimentology.io