The Importance of A/B Testing

A/B testing is an analytical method that has received a great deal of buzz recently due to its increasing popularity in web development. A/B testing, however, is much more than just a buzz term; it is a tried and true marketing concept that was in use long before that advent of the Internet. Previously, it was most commonly associated with direct mail marketing, and many of those direct mail concepts and strategies have translated seamlessly to doing business on the World Wide Web.

A/B Testing

A/B Testing Defined

A/B testing is a method for comparing a control sample with one or more other samples. The goal is to prove the assertion that the control sample is superior. In other words, the objective is to prove that A is greater than B. This is a basic scientific concept applied to marketing. To understand why this is useful, consider a direct mail marketing campaign that aims to secure donations for a charity. This campaign has two possible calls to action for use in the mailed letter. Let’s call them A and B. Which call to action should the campaign use? The marketers can answer that question scientifically by using an A/B test.

The Terminology

In this context, a sample is a subset of the whole. In the above example, the marketers may use one percent of the population to test A and one percent to test B. The population is the group to which they will mail the donation invitation. A/B testing is also referred to as bucket testing or split testing. Multivariate, also called multi-variable, testing is a marketing method often confused with A/B testing. The goal of A/B testing is comparing two variables in order to determine which is superior. The goal of multivariate testing, on the other hand, is to challenge hypotheses that are more complex.

How A/B Testing Is Used

Let’s continue with the previously established scenario. Our marketers must send mailings to a population of 10,000 people. The goal is to convert three percent of those mailings into donations. The marketers have developed two calls of action. They now have several options: They can choose one to use intuitively. They can send each call to action to 5,000 people. Or they can A/B test. With A/B testing, the marketers only send the less effective call to action to 1,000 people rather than 5,000.

The Role of A/B Testing in Web Development

Goals on the Web are often quite similar to the objectives of a direct mail campaign. Websites want to convert a sale, entice us to register, persuade us to consume particular content and so forth. Web developers can use A/B testing to assess the effectiveness of the phrasing, color, size or placement of a call to action. They can also us A/B testing to analyze a layout, headline, description, field format, font, price, promotional offer, image, video, landing page, so on and so forth. In other words, a webmaster can use A/B testing to measure the effectiveness of almost every aspect of the website.

Types of A/B Testing Tools for the Web

Web-oriented A/B testing tools can be categorized into three groups: code-free, light code and heavy code. Code refers to source code, which can be as simple as HTML or as complex as a compiled application. Code-free tools are very popular because they are easy and fast to use and aren’t resource intensive. On the con side, code-free tools tend to be inflexible. Heavily coded solutions, on the other hand, are infinitely versatile, since you can simply adapt them to the specific scenario. Heavily coded solutions, however, are complex and resource intensive. Solutions that require light coding seek to find a middle ground between the accessibility of code-free and the power of heavily coded.

Popular A/B Testing Tools for the Web

Two of the most popular code-free A/B testing tools are Optimizely and Performable. Optimizely gained fame for its use by the Obama campaign. Optimizely is Web-based and subscription-based, and it boasts a powerful visual interface. Performable is one of Optimizely’s chief competitors, and it provides additional robustness at the expense of some simplicity.

With both Optimizely and Performable, the developer is limited to common Web goals. That list is expansive, but it does come up short at times. Two light-code alternatives are Google Website Optimizer and Visual Website Optimizer. Google’s tool requires you to provide two unique webpages for comparison, but it is powerful, versatile and completely free. Visual Website Optimizer is not free, but the testing process is more streamlined than it is with the Google version.

Two of the most popular heavy code options are Genetify and Vanity. Genetify can run directly on the web server, and it includes an extensive JavaScript library for testing. What sets Genetify apart from other tools is that it actually has a learning mechanism that allows it to train and improve tests over time. Vanity is open source, completely free and a full-blown development framework for A/B testing. The initial investment with Vanity can be quite high, but it pays great dividends over time.

Written 2012-09-10 (Updated 2016-10-10)
david for reviews7

Written by David Walsh

David Walsh is the editor in chief here at Web Hosting Search. Having been in the industry for many years now he knows pretty much everything about everything. At least that's what he keeps telling everyone at the office. So, don't hesitate to drop him a line  if you've got a question - david(a)

Share your thoughts

David,  27 September, 2012

Hi Anuradha,

It's never wrong to A/B test your site on a regular basis. But, you need a certain amount of traffic to get useful data. No traffic = no data to analyze. A lot of traffic = a lot of data to analyze.

Good luck!


Anuradha,  26 September, 2012

When should a webmaster initialize A/B testing? Only when the conversion rate is low or is it a continuous process to improve the usability of a website?

Dennis van der Heijden,  21 September, 2012

Hi David, your so right A/B testing is important. A side note. Performable as a separate product does not exist anymore. It's been integrated in Hubspot 3. But if you want to compare to any other product, use ours A/B testing. I'll send you a login to try it out.


Show all related articles..

Best Value Hosting 2016

Why wait? Get today's best deals now!