Three reasons to stop A/B testing

Print

Posted 11 December 2012 12:04pm by Maurits Kaptein with 4 comments

• Tweet

• Tweet

Successful (online) companies are organised to experiment continuously. This is preaching to the choir, something we all agree on.

But what exactly is the best way to experiment as a company? We don't necessarily agree on the answer to that question.

This article deals with one of the most used types of online experiments, A/B tests.

I would like to explain why you should stop running them.

1. Individual visitors are not the same

We optimize for individuals. Opposed to other A/B or MVS testing applications we keep track of individual customers, and will model them individually.

This means that although version A might work best for the majority of your customers (and thus is the first choice for a new customer) we adapt if a specific – recurring – customer never responds to A.

We will try B for him or her, and stick with it if it works.

2. Test the Why instead of the What

We are not interested in button color or size of the font on your page (the What?). We optimize the influence strategies (the Why?) that are used on a site.

Thus, should you present customer reviews, or discounts, or expert endorsements, or limited time offers… etc. This enables us to create a profile of individual customers based on these influence strategies which can be used for future pitches - pitches that are different implementations of the same strategies.

The What's? stay dumb and the Why's change the game.

3. Don't exploit when it's time to explore

We adopt a bayesian paradigm to optimize the explore-exploit tradeoff. Thus, we don't tell people to use version A after A and B have been visited by a 1,000 visitors. We will keep pitching B every now and then if the uncertainty in our estimates is too high.

This enables us to explore whether a version becomes popular at a later point in time, or for a specific market segment. We thus  decide in real time which version to show to a new customer. We believe we are the first to provide this option commercially.

Image credit: mil8 via Flickr. 

Maurits Kaptein is Chief Science Officer at Science Rockstars and a guest blogger on Econsultancy.