A/B/n Split Test Confidence Calculator with Graphing

Calculating the confidence of a split test helps you make a decision to choose a winning variation based on strong enough data. Here are some of the potential applications:

  • PPC ad split tests
  • Landing page split tests
  • Email marketing tests
  • … and virtually any other experiments you need to calculate statistical significance for

Hint: Bookmark this page (Ctrl+D) or share it with your colleagues.

 VisitorsConversions

Control:

Treatment 1:




 Calculate Add row Remove row

Interpreting the results

You will know you have a winner when a message dsiplays saying you have a statistically significant result. On the graph, you will see the range where the conversion rate is estimated to sit at 80% confidence (the large block) and 95% confidence (the thin lines). If there is no overlap between the large blocks, and chance to win for your treatment is >95%, you will have a winner.

Notes on statistical significance and validity

Significance and validity help make sure your test results can be generalisable. Here are a few general rules you may like to apply for your tests:

  • Run your test for at least two weeks (weekends can really affect test results)
  • Avoid testing during unusual periods (Christmas, stock market crash etc)
  • Get at least 25 conversions per variation
  • … and at least 100 visitors per variation
  • If using gradual ramp-up, beware of Simpson’s Paradox
  • Use a split testing tool that cookies visitors so they only see one variation
  • Only compare data from the same data source (i.e. AdWords and Analytics don’t mix)
  • Beware of robots and how they influence your split testing tools’ data (GA and GWO are pretty safe from robots)