The Main KPI of Your A/B Test Is Wrong – This Is How to Do It Right

Oct 11, 2017

For data-driven marketers looking to improve their site’s user experience, A/B testing is one pillar of optimization used to help increase conversion rates. Relied on as a way to experiment with large and small changes before going “all in,” it is an important tool for deciding what experiences get applied to all users.

Therefore, not analyzing the results properly could mean the difference between potentially massive uplifts and diminishing returns.

By addressing some common A/B testing pitfalls and introducing more reliable approaches to measuring the efficacy of your experiences, you can be sure to leave here knowing how to optimize for success.

Look at Conversions Instead of Unique Conversions

Marketers have gotten very used to setting Conversion Rate as a main KPI, but when it comes to testing, it simply does not tell the whole story. While Conversion Rate refers to the number of unique users that converted, your business goals should be set up for more conversions, not necessarily more convertors.

Look at the table below, and ask yourself, which variation performs better?

table2_test

It is safe to say that Variation A performed better, resulting in more conversions and an uplift of 25%. So, even though it might be easier to think about percentages than small fractions, is it worth losing out on the meaningful data?

Don’t opt for the less optimized path.

To conclude: If your objective is an action that can only be done once, like registering to a site or becoming a newsletter subscriber, Conversion Rate and Conversions per User should show the same result (one with percentages and the other with fractions). However, if the objective is to add an item to a cart or purchase, Conversion Rate is an inferior metric, as it assumes you’re looking for convertors and not conversions — and multiple conversions by a user do not weigh more than a single conversion.

Average by Users, not Sessions

Google Analytics taught us to analyze behavior through the prism of Sessions. For instance, an eCommerce brand measuring revenue would look to RPV (revenue per session), and a media site, average number of pageviews per session.

However, if you’re choosing the variation once for each user, you should average results by number of users. Averaging by sessions will many times result in a bias in favor of the worst variation.

As an example, here is the result of a test where the primary metric is clicking the Add to Cart button. After two users entered the test, one was served with the Control and converted four times, not returning to the site. The other was served with Variation A and converted five times, returning for another session, in which he converted twice.

table3_test

When dividing conversions by sessions, the Control won, with 4 conversions per user against Variation A’s 3.5. Though, Conversions per User paints a different story, with Variation A winning against the Control 7 – 4.

Would it make sense to punish Variation A, which ultimately produced more conversions because it led to an additional session that had a small number of conversions?

To conclude: If you’re comparing two alternatives, averaging results by users, and not sessions will allow you to optimize for a better user experience on your site.

Choose the Right Primary Metric

When performing a test, you usually have one metric in mind that is the primary — the winning variation will be determined by the success of each variation for this metric. This can include any user activity, such as clicking a specific button, adding items to the cart, completing a purchase, etc.

A marketer for an eCommerce site once asked:

Why do you allow so many metrics to choose from? I only need one: Revenue.”

As with any business relying on revenue, it makes sense as an approach that everything contributing to this goal would be good, and everything negatively impacting it, bad. However, conversion funnel methodology contradicts this model for thinking as each step of the funnel is aimed towards advancing the user to the next step in the funnel, meaning results should be optimized accordingly.

Here’s a simplified version of an eCommerce site conversion funnel.

eCommerce Conversion Funnel -- Dynamic Yield

After analyzing tests run earlier in the conversion funnel on a new homepage navigation element of an eCommerce site, after seven days, this became quite obvious:

table1_test

There is no clear winner yet for the primary metric, as the number of purchases was too small. Though, when looking at clicks, enough data was collected to determine Variation A as a winner. And so we see the mathematical reason to select a closer, less delayed conversion.

Furthermore, selecting a conversion farther down the funnel will cause significant results to arrive a lot later because many of the users that advanced in the funnel will now experience the same site. It is not unlikely the rest of the experience and conversion rate will be the same.

Hence, the difference between their performance will be small.

This is what we refer to as Immediate Reward and Delayed Reward.

  • Clicks on a variation is immediate
  • Purchases on a test in the checkout page are immediate reward
  • Purchases on a test in the homepage are delayed reward

To conclude: The conversion funnel methodology allows marketers to more quickly improve conversion rates. By setting objectives that are achieved more immediately, results are easier to track, and key metrics easier to improve.

Dynamic Yield allows for the reporting of any secondary metric so data is not lost by the selection of a more immediate primary metric.

The Main KPI of Your A/B Test Is Wrong – This Is How to Do It Right
4.33 (86.67%) 6 votes

Menu Title
Contact Us
×