How to Build a Company Culture of Experimentation

Charlie Graham is founder and CEO of Shop It To Me, an e-commerce company focused on being the most personalized way to shop. Follow him @imcharliegraham.

If you are a consumer internet company you should constantly be experimenting with changes to your site or product. That's because no matter how smart or experienced you are at building things, not all of your ideas will be successful.

The benefits of A/B testing — testing two or more variations in a controlled random setting — are well documented, with numerous websites devoted to A/B testing case studies. Still, many companies only briefly dabble in it. Why? Probably because creating a company culture that's open to experimentation is tough to do. Plus, integrated A/B testing sounds like a lot of extra work when you already have limited resources.

The truth is, to succeed on a long-term basis, you have to build a culture of experimentation where testing is part of everything you do. Here's how to get started.

Let Everyone See They Can Be Wrong

People generally believe they will get the product right the first time. As a leader in your company, you need to show everyone why testing is an important investment.

Start small. Choose three to five simple things that employees think they know to be true about the company website, and test each. For example, set an A/B test to change the color or the wording of one page on the site. Have everyone vote on expected results (the percentage lift or decline each change will have). Then, put the team responses in a spreadsheet and send them all a copy. Run the test, tabulate results, and announce them.

On average, even the best people are wrong about half the time. It's humbling, eye-opening, and the first step to get people to realize how much opportunity for growth exists in testing.

If you need some inspiration for designing a test, check out Which Test Won?. You'll always find new tests as well as the results.

Make it Easy to Test Ideas

Convenience is critical. The easier it is to do a test, the more tests you'll be able to do. Consider investing the time up front to build an A/B testing infrastructure as well as an experiments "dashboard" to monitor how all experiments are doing. You can also use existing third party-systems such as Visual Website Optimizer or Optimizely.

Make Brainstorming Part of Your Process

Your experiment results are only going to be as good as the ideas that go into them. Brainstorm at least once a week, and include team members from across the organization. Keep a list of different experiments you want to running at any time. You'll not only find good ideas but discover that people will get more excited when they realize how many new improvements they could make. And not all of these have to be big changes. According to Visual Website Optimizer, Wikijob, a job site in the U.K., increased sales by 34% by changing its user testimonials.

Make it a Numbers Game

The reality is, when you start testing, only one out of every five experiments will generate positive change. Most will make no difference or have a negative effect. As your website starts improving, your odds get worse. After you're in the swing of testing, you will have to put a lot of experiments out there just to get a few positive boosts.

Get a Statistics Champion

Your answers must be statistically significant and someone at your company really needs to understand this. You may find that one result outperforms another on day one by 50%, but it might not be statistically significant and you'll head down a wrong path. Consider setting up your dashboard to automatically calculate and highlight statistically significant shifts. Better yet, get a statistics guru.

Let it Change Your Culture

Proper experimentation can be beneficial for the company but can also be empowering and fun for employees. It gives everyone a chance to make a difference. It's also a great way to handle disputes. Two people have two strong but different opinions? We no longer argue about it in our offices. We just test it.

Image courtesy of iStockphoto, AnsonLu