Use this free bayesian A/B testing calculator to find out if your test results are statistically significant. For each variation you tested, input the total sample size, and the number of conversions.

Samples

The number of users, sessions or impressions depending on your KPI

Conversions

The number of clicks or goal completions

A

B

Add Variation

You've reached the maximum of 10 variations.

Winner significance level

Variations that exceed this threshold are declared the winner of the test

Samples must be greater or equal to Conversions

Variation A is the winner!

Based on 95% significance level

Probability to be Best

Each variation's long-term probability to out-perform all other live variations, given collected data since the creation or change of any variation included in the test

96.79%

3.21%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Samples

Conversion

Conversion Rate

Probability to be Best

Each variation's long-term probability to out-perform all other live variations, given collected data since the creation or change of any variation included in the test

Expected Loss

Assuming I declare the variation as a winner, and I am wrong, how much am I expected to lose in the long term, in term of % vs the variation which is actually the best

A

8,500

1,500

17.647%

96.79%

0.9%

B

8,500

1,410

16.588%

3.21%

6.22%

Posterior simulation of difference

The distribution of conversion rates given the sample size collected so far

120%

110%

100%

90%

80%

70%

60%

50%

40%

30%

20%

10%

0%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

You can use this Bayesian A/B testing calculator to run any standard hypothesis Bayesian equation (up to a limit of 10 variations). To do so, specify the number of samples per variation (users, sessions, or impressions depending on your KPI) and the number of conversions (representing the number of clicks or goal completions). Click the Calculate button to compute probabilities.

Please note: This tool does not intend to represent, nor replace Dynamic Yield's product calculations. Calculations in this tool are based only on binary models, while Dynamic Yield's product calculations use a different formula for non-binary, revenue-based experiments as well as for handling probabilities for unique conversions.

DY Lab products are still in development but show a lot of promise, so we want to give you a chance to try them out. These products haven't been subjected to the same level of reliability, scalability, and security measures as other Dynamic Yield products, and are not being officially supported.

Explore DY Labs →Recommended Reading

The Power of Bayesian A/B Testing →
Bayesian A/B Test Duration & Sample Size Calculator →
Frequentism and Bayesianism: A Practical Introduction →
The Importance of Statistical Significance in A/B Tests →
How Not To Run An A/B Test →
Definition of Probability to Be Best in A/B Testing →