Bayesian A/B Testing Calculator

Use this free bayesian A/B testing calculator to find out if your test results are statistically significant. For each variation you tested, input the total sample size, and the number of conversions.
Samples
The number of users, sessions or impressions depending on your KPI
Conversions
The number of clicks or goal completions
A
B
Calculate
Add Variation
You've reached the maximum of 10 variations.
Winner significance level
Variations that exceed this threshold are declared the winner of the test
Samples must be greater or equal to Conversions
Variation A is the winner!
Based on 95% significance level
Probability to be Best
Each variation's long-term probability to out-perform all other live variations, given collected data since the creation or change of any variation included in the test
96.79%
3.21%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Samples
Conversion
Conversion Rate
Probability to be Best
Each variation's long-term probability to out-perform all other live variations, given collected data since the creation or change of any variation included in the test
Expected Loss
Assuming I declare the variation as a winner, and I am wrong, how much am I expected to lose in the long term, in term of % vs the variation which is actually the best
A
8,500
1,500
17.647%
96.79%
0.9%
B
8,500
1,410
16.588%
3.21%
6.22%
Posterior simulation of difference
The distribution of conversion rates given the sample size collected so far
120%
110%
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%

About this tool

You can use this Bayesian A/B testing calculator to run any standard hypothesis Bayesian equation (up to a limit of 10 variations). To do so, specify the number of samples per variation (users, sessions, or impressions depending on your KPI) and the number of conversions (representing the number of clicks or goal completions). Click the Calculate button to compute probabilities.

Please note: This tool does not intend to represent, nor replace Dynamic Yield's product calculations. Calculations in this tool are based only on binary models, while Dynamic Yield's product calculations use a different formula for non-binary, revenue-based experiments as well as for handling probabilities for unique conversions.

About DY Labs

DY Lab products are still in development but show a lot of promise, so we want to give you a chance to try them out. These products haven't been subjected to the same level of reliability, scalability, and security measures as other Dynamic Yield products, and are not being officially supported.

Explore DY Labs →
Recommended Reading
The Power of Bayesian A/B Testing → Bayesian A/B Test Duration & Sample Size Calculator → Frequentism and Bayesianism: A Practical Introduction → The Importance of Statistical Significance in A/B Tests → How Not To Run An A/B Test → Definition of Probability to Be Best in A/B Testing →