Machine Learning Based Optimization vs. A/B Testing

Read the full transcript

As a data-driven company, we are obsessed about A/B testing. We A/B test every single thing we do. However, it is important to acknowledge the limitations of A/B testing. First of all, if you’re just A/B testing for all your audience at the same time, you’re basically optimizing for the quote-on-quote average user, but what is the average user? You have your frequent shoppers. You have first time visitors. You have people who came from a Google campaign. You have people who came through your app. So we tie A/B testing technology very closely to segmentation’s strategy. But there is a limit to how many segments you can manage as a human being before it becomes to complicated and this why we introduced automated optimization. Which basically means that we’re using ad serving like techniques for changing the on site experience. So what our customers do today is, instead of just doing an A/B test of five different banners or five different call-to-actions, they just create all these variations and they upload them to Dynamic Yield and we make a real-time machine learning based decision on what variation to show each individual user based on all the data we have on that individual whether it’s first-party data or third-party data. The other big advantage of optimization versus A/B testing is the duration of the test. When it comes to running, experimenting with real-time events for a holiday or you have like back-to-school. Instead of doing an A/B test, they go for automated optimization. And this is where the machine learning algorithms kick in and they start predicting for each individual user what we should show them in order to maximize revenue. And then we keep a control group and after the holiday is over, you can see, oh my optimization mechanism has generated 10% more than my control group which was the variation I had before I started the test. So, the idea of using machine learning and real-time optimization versus A/B testing is very important for anything that is short lived.

A company can be data-driven and still acknowledge the limitations of A/B testing. In the face of human limitation, machine learning may be necessary if you want to scale your experimentation efforts.