flâneur

The top 3 mistakes that make your A/B test results invalid | Conversion

conversion.com · 2,741 words · saved by 1 readers

How elite subscription brands like Netflix, Spotify, and HelloFresh are making UX a competitive advantage By comparing Spotify, Netflix, HelloFresh, Headspace, and Kiwico, we used our pioneering benchmarking technique to gain valuable critical insights into how these leading brands power their UX. Groomed for success: Building an experimentation program for a high growth business Increasing sales during a pandemic using behavioural science Immediate results: How we generated £2.8 million for Clarks with our first experiment alone Sizing recommendations drive a 7% revenue increase for T.M. Lewin Back to Basics… Disrupting the ‘offline’ shopping experience for Tommy Hilfiger. How testing SaaS pricing and the user journey helped Wistia double sales Generating double digit conversion rate lift and Executive-level buy-in for experimentation Optimizing the SaaS acquisition experience with a rigorous experimentation program Heifer International leverages donor insights and experimentation to

A Fortune 500 company asked that I review their A/B testing strategy. The results were good, the hypotheses strong, everything seemed to be in order… until I looked at the log of changes in their experimentation tool. I noticed several blunders: In some experiments, they had adjusted the traffic allocation for the variations mid-experiment; some variations had been paused for a few days, then resumed; and experiments were stopped as soon as statistical significance was reached. When it comes to testing, too many companies worry about the “what”, or the design of their variations, and not…

related reading