Introducing our Bayesian A/B Testing Calculators

A while back, we explored a less restrictive and more reliable approach to A/B testing in the form of a newer, Bayesian testing method.

With its simplicity, reliability, and intuitiveness, the Bayesian framework is a superior A/B testing methodology which will provide marketers a quicker and more robust statistical engine.

Therefore, we’ve been working long and hard to make it easier for marketers and conversion rate optimizers to utilize this new approach. And for us, that’s meant taking the complex math out of the equation.

So, today, we’re excited to announce the official release of two free and useful tools…

Bayesian A/B Testing Calculator

Our Bayesian-powered A/B testing calculator will help you find out if your test results are statistically significant.

For each variation you test, all you have to do is input the total sample size and number of conversions. Then, based on statistical significance, the statistical engine will declare a winning variation.

Here’s a quick breakdown of the terms and metrics we run:

• Sample Size – The number of users, sessions, or impressions depending on your KPI.
• Conversion – The number of clicks, even purchases or goal completions (e.g. purchases or video views).
• Conversion Rate – The number of completed actions (i.e. conversions) divided by the sample size.
• Probability to be Best – Each variation’s long-term probability to outperform all other live variations (given collected data since the creation or change of any variation included in the test).
• Expected Loss – The percent you are expected to lose in the long term if you declare the wrong variation as a winner versus the variation which is actually the best.
• Posterior Simulation of Difference – The distribution of conversion rates given the sample size collected so far.

To better understand your results, consider the winning variation above. Across the board, Variation A dominates in terms of conversion, conversion rate, Probability to Be The Best, as well as Expected Loss. Remember, the higher the PTBB and lower the EL, the better. This indicates confidence in long-term performance as well as lower expected loss if the variation declared a winner ends up being wrong (which it doesn’t look like it will be).

Learn more about the importance of statistical significance in A/B tests.

Bayesian A/B Test Duration & Sample Size Calculator

Not sure how long you will have to run your experiments in order to get statistically significant results?

No worries. As the first of its kind, our free online Bayesian-powered A/B test duration and sample size calculator will help you avoid false positives and increase the validity of your A/B testing. There’s no hard limit on how many variations you can test against the control and we offer a few suggestions for how to approach your testing.

The calculated output provides range estimations of the time required to run the test in order to get statistically significant results, and the minimum required sample size to support that. While we know there are many other variables involved, these calculated estimations can be used for planning experiments in advance, as well as for analyzing ongoing tests.

Here’s a quick breakdown of the terms and metrics we run:

• Baseline conversion rate – The current conversion rate (number of successful actions divided by the number of visitors, sessions, or impressions) for the experience you’re testing.
• Expected uplift in conversion rate – The X% change in conversion rate you are aiming for from your baseline rate. In the example below, if the baseline conversion rate is 2.5%, a 5% expected uplift would result in a new conversion rate of 2.625%.
• Number of variations – The number of variations compared in a single test.
• Average sample size per day – The number of visitors to be served in the experiment over the course of one day.

Now that you have everything you need to get started, we know you’ll be deploying Bayesian A/B tests at higher confidence intervals. And with a higher degree of self-confidence.

It’s important to note that while these tools support binary objectives (user converted or not) when measuring conversion, the real statistical engine behind the Dynamic Yield platform behaves differently, and supports more advanced, non-binary calculations, such as maximizing completions number, or revenue. To learn more about our statistical engine and automated optimization approach, request a product demo.