A/B Testing Defined
A/B testing is a method for comparing a control sample with one or more other samples. The goal is to prove the assertion that the control sample is superior. In other words, the objective is to prove that A is greater than B. This is a basic scientific concept applied to marketing. To understand why this is useful, consider a direct mail marketing campaign that aims to secure donations for a charity. This campaign has two possible calls to action for use in the mailed letter. Let’s call them A and B. Which call to action should the campaign use? The marketers can answer that question scientifically by using an A/B test.
The Terminology
In this context, a sample is a subset of the whole. In the above example, the marketers may use one percent of the population to test A and one percent to test B. The population is the group to which they will mail the donation invitation. A/B testing is also referred to as bucket testing or split testing. Multivariate, also called multi-variable, testing is a marketing method often confused with A/B testing. The goal of A/B testing is comparing two variables in order to determine which is superior. The goal of multivariate testing, on the other hand, is to challenge hypotheses that are more complex.
How A/B Testing Is Used
Let’s continue with the previously established scenario. Our marketers must send mailings to a population of 10,000 people. The goal is to convert three percent of those mailings into donations. The marketers have developed two calls of action. They now have several options: They can choose one to use intuitively. They can send each call to action to 5,000 people. Or they can A/B test. With A/B testing, the marketers only send the less effective call to action to 1,000 people rather than 5,000.
The Role of A/B Testing in Web Development
Goals on the Web are often quite similar to the objectives of a direct mail campaign. Websites want to convert a sale, entice us to register, persuade us to consume particular content and so forth. Web developers can use A/B testing to assess the effectiveness of the phrasing, color, size or placement of a call to action. They can also us A/B testing to analyze a layout, headline, description, field format, font, price, promotional offer, image, video, landing page, so on and so forth. In other words, a webmaster can use A/B testing to measure the effectiveness of almost every aspect of the website.
Types of A/B Testing Tools for the Web
Web-oriented A/B testing tools can be categorized into three groups: code-free, light code and heavy code. Code refers to source code, which can be as simple as HTML or as complex as a compiled application. Code-free tools are very popular because they are easy and fast to use and aren’t resource intensive. On the con side, code-free tools tend to be inflexible. Heavily coded solutions, on the other hand, are infinitely versatile, since you can simply adapt them to the specific scenario. Heavily coded solutions, however, are complex and resource intensive. Solutions that require light coding seek to find a middle ground between the accessibility of code-free and the power of heavily coded.
Popular A/B Testing Tools for the Web
Two of the most popular code-free A/B testing tools are Optimizely and Performable. Optimizely gained fame for its use by the Obama campaign. Optimizely is Web-based and subscription-based, and it boasts a powerful visual interface. Performable is one of Optimizely’s chief competitors, and it provides additional robustness at the expense of some simplicity.
With both Optimizely and Performable, the developer is limited to common Web goals. That list is expansive, but it does come up short at times. Two light-code alternatives are Google Website Optimizer and Visual Website Optimizer. Google’s tool requires you to provide two unique webpages for comparison, but it is powerful, versatile and completely free. Visual Website Optimizer is not free, but the testing process is more streamlined than it is with the Google version.
Two of the most popular heavy code options are Genetify and Vanity. Genetify can run directly on the web server, and it includes an extensive JavaScript library for testing. What sets Genetify apart from other tools is that it actually has a learning mechanism that allows it to train and improve tests over time. Vanity is open source, completely free and a full-blown development framework for A/B testing. The initial investment with Vanity can be quite high, but it pays great dividends over time.
David, 27 September, 2012
Hi Anuradha,
It's never wrong to A/B test your site on a regular basis. But, you need a certain amount of traffic to get useful data. No traffic = no data to analyze. A lot of traffic = a lot of data to analyze.
Good luck!
/David
Anuradha, 26 September, 2012
When should a webmaster initialize A/B testing? Only when the conversion rate is low or is it a continuous process to improve the usability of a website?
Dennis van der Heijden, 21 September, 2012
Hi David, your so right A/B testing is important. A side note. Performable as a separate product does not exist anymore. It's been integrated in Hubspot 3. But if you want to compare to any other product, use ours Convert.com A/B testing. I'll send you a login to try it out.
Dennis