A/B/n Split Test Confidence Calculator with Graphing
TweetCalculating the confidence of a split test helps you make a decision to choose a winning variation based on strong enough data. Here are some of the potential applications:
- PPC ad split tests
- Landing page split tests
- Email marketing tests
- … and virtually any other experiments you need to calculate statistical significance for
Hint: Bookmark this page (Ctrl+D) or share it with your colleagues.
VisitorsConversionsTreatment 1:
Interpreting the results
Essentially the table will highlight any winners in green. Meanwhile, on the graph, you will see the range where the conversion rate is estimated to sit at 80% confidence (the large block) and 95% confidence (the thin line).
Ensuring statistical significance and validity
Significance and validity help make sure your test results can be generalisable. Here are a few general rules you may like to apply for your tests:
- Run your test for at least two weeks (weekends can really affect test results)
- Avoid testing during unusual periods (Christmas, stock market crash etc)
- Get at least 100 conversions per variation
- If using gradual ramp-up, beware of Simpson’s Paradox
- Use a split testing tool that cookies visitors so they only see one variation
- Only compare data from the same data source (i.e. AdWords and Analytics don’t mix)
- Beware of robots and how they influence your split testing tools’ data (GA and GWO are pretty safe from robots)
About Rob
I work as a Senior Analyst (Analytics) at Experian Hitwise to help clients turn their stats into results. I also like to run, drink, eat and be merry. My girlfriend would say I play too many games and use my Android too much.
