The Difference Between A/B Testing, Multivariate and Split URL Testing

May 19, 2014

While all experts agree that conducting controlled experiments for the benefit of conversion rate optimization and performance improvement is an important aspect of every website optimization procedure, occasionally, there is a bit of confusion revolving around the difference between various testing methods.

In this article, I will explain in detail the different methodologies of each test, helping you understand the best practices and benefits of each and every one. I will also provide a glimpse into our very own, unique approach to continuous and dynamic content testing.

Free Guide: Take your testing to the next level.
Delve deeper into the mechanics of A/B testing and discover 10 rules for running impactful tests.

Let us start by explaining traditional A/B and multivariate testing methods:

A/B Testing vs. Multivariate Testing

A/B Tests –

Do I need to have two separate URLs when A/B testing?

In a simple A/B test, you’re splitting traffic between two variations of content. One is considered the control and contains the original content and design. The other functions as a new version of the controlled variation. The variation may be different in many aspects. For example, we could test a variation with different headline text, call-to-action buttons, a new layout or design and so on.

Generally speaking, you don’t necessarily need two different URLs in order to run a proper A/B test. Most A/B testing solutions will let you create variations dynamically by modifying the original URL. If you’re looking for a solution to test performance of two different URLs, you should probably consider using a split URL test.

And what if I have two separate URLs I want to test?

Split URL testing, sometimes referred to as “multi-page” or “multi-URL” testing, is a similar method to a standard A/B test, which allows you to conduct experiments based on separate URLs of each variation.

With this method, you can conduct tests between two existing URLs, which is especially useful when serving dynamic content. Run a split URL test when you already have two existing pages and you want to test which one of them performs better.

What if I need to test more than two content variations?

In that case, use an A/B/n testing solution. When testing large content changes, A/B/n tests allow you to measure performance of three or more variations of a page instead of testing only one variation against a control page. High-traffic sites can use this testing method to evaluate performance of a much broader set of variations and to maximize test time with faster results.

Are there any best practices I should follow?

Although it is useful for any sort of testing, from minor to dramatic changes, I recommend not making too many changes between the control and variation. Try making just a few important and dominant changes, in order to try to understand the possible causal reasons for the results of the experiment.

Multivariate Tests –

Multivariate tests, sometimes referred to as “multi-variant” tests, allow you to test changes to multiple sections on a single page. As an example, run a multivariate test on one of your landing pages and change it with two new elements. On the first version, add a contact form instead of the main image. On the second version, add a video item. The system will now generate another possible combination based on your changes, which includes both the video and the contact form:

Total test versions: 2 x 2 = 4

V1Control variation (no contact form and no video item)
V2Contact form version
V3Video item version
V4Contact form + video item version

Are there any best practices I should follow?

Since multivariate tests generate all possible combinations of your changes (as described above), it is not recommended to create a large amount of variations unless you’re running the test on a high-traffic site.

On the other hand, running multivariate tests on low-traffic sites will provide poor results and insufficient data to draw any significant conclusions. Be sure to have at least a few thousand monthly visitors to your site before choosing to run a multivariate test.

When should I use multivariate tests instead of A/B tests?

A/B tests will help you answer questions such as:

  • Which of the two versions of my page perform better in terms of the visitors’ response to it?

Multivariate tests will answer questions like:

  • Do visitors respond better to a video item next to a contact form?
  • Or to a webpage with just a contact form and no video item?
  • Or to a webpage with a video item but no contact form?

And so on…

A/B Testing vs. Multivariate Testing

Introducing: The Dynamic Yield Approach to A/B/n Tests

In traditional A/B or multivariate testing, you need to wait for a declaration of one winning variation with a clear statistical significance – usually with a threshold of at least 95-percent confidence level. However, while others rely on one winning variation, we understand that in many cases, there cannot be one possible variation that fits all users.

Instead of optimizing experiences for the average user, our algorithms dynamically serve the most relevant variations to each user. This is possible thanks to our bayesian a/b testing engine, and contextual bandit algorithms.

Read more about traffic allocation in A/B tests.

The result of running continuous optimizations with Dynamic Yield are an ongoing and fully automated testing procedure showcasing higher conversions, better user experience, and faster results.

Find Out if Your Test Results are Statistically Significant
Use our free bayesian testing calculator and discover a less restrictive and more reliable approach to A/B testing

Want to learn more about our experimentation approach? Drop us a line and we’ll show you how to improve you conversion rates and overall revenue.

The Difference Between A/B Testing, Multivariate and Split URL Testing
5 (100%) 4 votes

Menu Title
Contact Us