Congratulations! You’ve found an aspect of your website that, upon changing, real differences appeared in a test! When you hear that you have a conclusive result, that means that the observed difference is significant enough to draw a conclusion and declare a winning variation.
Although some debate exists as to what percentage of probability is needed to declare a winner, something above 95 should be considered and the closer it gets to 100 – the more likely you have a true winner. If you have an extremely high number of conversions, consider running your test for an extended time to ensure other variables aren’t at play. For example, if your #1 competitor went out of business the week before and you didn’t know – this could account for atypical traffic that had nothing to do with your proposed website changes.
Also, be careful not to become overly confident or apply these changes broadly without additional tests. Just because a blue button works better on your landing page doesn’t mean that it will work better on your product pages with different color backgrounds. In this case, a change improved things – but this doesn’t mean you should go back and redo everything. Customers also like consistency, so changing too much may hurt your brand identity and your overall consumer-facing image.
