Choosing the Right Conversion Optimization Objective

Conversion optimization can be a very powerful process, but it’s easy to get carried away by the methodology. Despite the best intentions, many conversion optimization initiatives fail to produce valuable results, leading to misleading and unsubstantiated insights. Fortunately, the biggest blunders can be prevented by working methodically.

A conversion optimization process can fail for a variety of reasons. One of the biggest considerations when running experiments revolves around choosing the right optimization objective. Generally speaking, conversions are measured when a visitor executes actions that are defined as valuable to your business. Conversion can be measured as a click on an element, completion of a goal, or any online action which has a direct impact on revenue. Whether you’re trying to improve display ad clicks using CTR-based optimization, generate more leads using goal-based optimization or increase sales using revenue-based optimization, choosing the right objective for your optimization initiatives can make a huge difference between their failure or success.

Take your testing to the next level. Delve deeper into the mechanics of A/B testing and discover 10 rules for running impactful tests.

Having a specific, measurable objective in mind, however, is not enough, and that’s the first thing you need to realize before planning a test. Many other variables are involved, such as the type of experiment you’re running, the audience sample size, experiment length, conversion attribution, the number of content variations that are being tested, and how prominent the changes being tested are. It’s not just about measuring the results – it’s about optimizing the data as effectively as possible using the right optimization method.

Below is a table that illustrates this point. Use it to get a better idea of what the optimization objective of your experiments should be, based on sample size and estimated length of the experiment. If you don’t know what the sample size is, use a sample size calculator like the one Evan Miller offers. Of course, there is no silver bullet, and this table can only serve to guide you towards making the right decision.

How to Choose the Right Optimization Objective:

Low
Sample Size
Medium
Sample Size
High
Sample Size
Short
Experiment
Length
No OptimizationGoal Completion
Optimization
Goal Completion
Optimization
Medium
Experiment
Length

CTR Based
Optimization
Goal Completion
Optimization
Revenue-based
Optimization
Long
Experiment
Length

CTR Based
Optimization
Revenue-based
Optimization
Revenue-based
Optimization

Time is of great concern, and it’s one of the main factors influencing the decision of choosing the right objective for your experiments. If you’re running an eCommerce site and intending to get fast results for an experiment you’re just about to launch, remember this: Revenue-based optimization requires greater purchase cycles than CTR or goal-based optimization. With revenue-based optimization, it’s far more complicated to attribute the actual conversion to a single experiment variation. It takes more time for a random visitor exposed to an experiment to complete a purchase than it takes to complete a click on an element. Some customers compare prices, some are just slow buyers. In fact, many things can happen between the first exposure to a live experiment and the final destination of completing a purchase. So as a general rule of thumb, if you’re looking for fast results, forget about optimizing for revenue and stick with shorter and simpler conversion cycles, such as click-based optimization or goal completion optimization.

Drawing conclusions too quickly is also a major issue. It can be difficult to keep your cool and ignore an (allegedly) obvious winner a few days into a test. While results vary across industries and sample sizes, waiting for the optimization process to accumulate enough data to achieve statistical significance is absolutely crucial. Even if your testing tool forecasts incredible uplift, keep the test running for at least a few weeks or a few purchase cycles before starting to analyze results. While some experiments reveal major uplifts early in the test process, results can vary dramatically over time, causing many marketers to draw the wrong conclusions and even lose money over time if they implement changes based on the initial results.

Every test is fallible. If you can’t send enough traffic and generate conversions to all variations, your efforts will go unrewarded. Just because you can test a page element, that doesn’t mean you should. In fact, the larger the sample size, the more data becomes available for the system to optimize. Make sure you test noticeable changes that are big enough to have a real impact on the bottom-line business. Without sufficient data (e.g. traffic or conversions), your experiments will never reach the statistical significance minimum of 95% and thus will be bound to fail.

Every test requires considerable resources (budget, time, and people), so stick to the elements you know make a difference, namely headlines, hero shot, and call-to-action. Start testing minor features and you’ll find yourself lost in data you can’t use. There’s no point in running short-term optimization initiatives for low-traffic sites. If the sample size is low, the system will struggle to optimize and gain real uplifts. It would take a lot of time and may eventually even fail.

As with any tool or methodology, a single experiment will not revolutionize your business, so don’t expect the test to produce a “Silver Bullet”. However, integrating testing over time and rolling out insights into your website will introduce incremental improvements that can (and do) reach a tipping point. Take a granular approach, choose the right objective and run your optimization initiatives intelligently and methodically. Plan for small wins, and they’ll add up to a big difference in the long run.

Choosing the Right Conversion Optimization Objective
5 (100%) 2 votes

Menu Title
Contact Us
×