Bayesian A/B Test Duration
& Sample Size Calculator

Powered by Dynamic Yield's Bayesian Stats engine, this free A/B test duration and sample size calculator will show you how long will you have to run your experiments for, to get statistically significant results. This calculator will help you avoid false positives and increase the validity of your A/B testing.
Baseline conversion rate
Expected uplift in conversion rate
Number of variations in test
Average sample size per day
In order to identify a winning variation with a 95% confidence level, you need a sample size of 105,000, which will take approximately 2 weeks.
Frequently Asked Questions
How many variations can you test against the control?
There is no hard limit. Instead, it depends on your patience and the amount of traffic. You need to take into account the more variations you test, the longer it will take the experiment for a winner to be declared. An important factor to consider is the difference between variations. The more substantial the difference in appearance, the faster you can detect a statistically significant difference in performance. Additionally, a control variation is not a must. You can compare variations to one another using a Bayesian stats calculator which determines a 'probability to be best' for each variation, allowing you to declare a winner faster than ever before.
What factors determine how many variations you put against the control?
It comes down to how bold you are and how quickly you want results. Your resources traffic and creativity and you need to use them wisely. The explore-exploit tradeoff means you need to balance your desire to exploit the knowledge you have, giving users the best experience you know of so far, while also risking serving sub-optimal experience as you try to discover an even better experience.

There is no silver bullet. Do your best to come up with a diverse set of variations and risk exploring them in the short term to improve performance in the long run. Don't create a new variation just for the sake of testing more ― do it only if you have a reasonable belief that it will be better than anything you have tried so far.
Does it matter whether the change is big or small?
Working with small changes takes a lot of time and only allows you to "fine tune" the performance of your page. On the other hand, if you test major differences, you can make leaps in performance. Having said that, major differences are risky during an experiment and may also cause a drop in performance for the duration of the experiment. An optional solution is to make major changes and run the experiment on a small portion of the population.
The Importance of Statistical Significance in A/B Tests The Power of Bayesian A/B Testing Machine Learning Driven Optimization Conversion Attribution in A/B Tests A Contextual-Bandit Approach to Website Optimization