The Math of Split Testing Part 3: The Chance of being Similar
tl;dr
Sometimes we want to verify that a new design will convert at nearly the same rate as an old design. Split tests of this type are not intended to find conversion rate wins, but rather to ensure that the new design is not “too much worse” than the old design. Here, we demonstrate how to analyze a split test of this sort and present an approximate formula to quickly calculate the chance that there is an acceptable difference between the conversion rates of the new and old designs.
Let’s say that we think that the design of our website is starting to look a little outdated and needs to be redone. In this scenario, we’re not really looking to increase conversion rate. We’d be happy to keep our current conversion rate and just make the website look a little more hip. To verify that our new design (branch B) converts at a similar rate as our old design (branch A) we should run a...