flâneur

P-Hacking in Startups | Hacker News

news.ycombinator.com · 10,236 words · saved by 1 readers

No, it means "I’m willing to ship something that if it was not better than the alternative it would have had only a 5% chance of looking as good as it did.” reply reply Consider this example - we don't change the treatment at all, we just update its name. We split into two groups and run the same treatment on both, but under one of the two names at random. We get a p value of 0.2 that the new one is better. Is it reasonable to say that there's a >= 80% chance it really was better, knowing that it was literally the same treatment? reply Parent: "5% chance of looking as good as it did, if it were truly no better than the alternative." This accepts the premise that the product quality is a fact, and only uses probability to describe the (noisy / probabilistic) measurements, i.e. "5% chance of looking as good". Parent is right to pick up on this, if we're talking about a single product (or, in medicine, if we're talking about a single study evaluating a new treatment). But if we're talking

On the one hand, this is a very nicely presented explanation of how to run statistically significant A/B style tests. It's worth emphasizing though that if your startup hasn't achieved product market fit yet this kind of thing is a huge waste of time! Build features, see if people use them. “This kind of thing” being running AB tests at all. There’s no reason to run AB / MVT tests at all if you’re not doing them properly. Proper way to make decisions is Bayesian. Take into account every bit of evidence, all the time. Statistical hypothesis testing is a simplification for people who…

saved by

related reading