A/B Testing Defined
A/B testing is a method for comparing a control sample with one or more other samples. The goal is to prove the assertion that the control sample is superior. In other words, the objective is to prove that A is greater than B. This is a basic scientific concept applied to marketing. To understand why this is useful, consider a direct mail marketing campaign that aims to secure donations for a charity. This campaign has two possible calls to action for use in the mailed letter. Let’s call them A and B. Which call to action should the campaign use? The marketers can answer that question scientifically by using an A/B test.
In this context, a sample is a subset of the whole. In the above example, the marketers may use one percent of the population to test A and one percent to test B. The population is the group to which they will mail the donation invitation. A/B testing is also referred to as bucket testing or split testing. Multivariate, also called multi-variable, testing is a marketing method often confused with A/B testing. The goal of A/B testing is comparing two variables in order to determine which is superior. The goal of multivariate testing, on the other hand, is to challenge hypotheses that are more complex.
How A/B Testing Is Used
Let’s continue with the previously established scenario. Our marketers must send mailings to a population of 10,000 people. The goal is to convert three percent of those mailings into donations. The marketers have developed two calls of action. They now have several options: They can choose one to use intuitively. They can send each call to action to 5,000 people. Or they can A/B test. With A/B testing, the marketers only send the less effective call to action to 1,000 people rather than 5,000.
The Role of A/B Testing in Web Development
Goals on the Web are often quite similar to the objectives of a direct mail campaign. Websites want to convert a sale, entice us to register, persuade us to consume particular content and so forth. Web developers can use A/B testing to assess the effectiveness of the phrasing, color, size or placement of a call to action. They can also us A/B testing to analyze a layout, headline, description, field format, font, price, promotional offer, image, video, landing page, so on and so forth. In other words, a webmaster can use A/B testing to measure the effectiveness of almost every aspect of the website.
Types of A/B Testing Tools for the Web
Web-oriented A/B testing tools can be categorized into three groups: code-free, light code and heavy code. Code refers to source code, which can be as simple as HTML or as complex as a compiled application. Code-free tools are very popular because they are easy and fast to use and aren’t resource intensive. On the con side, code-free tools tend to be inflexible. Heavily coded solutions, on the other hand, are infinitely versatile, since you can simply adapt them to the specific scenario. Heavily coded solutions, however, are complex and resource intensive. Solutions that require light coding seek to find a middle ground between the accessibility of code-free and the power of heavily coded.
Popular A/B Testing Tools for the Web
Two of the most popular code-free A/B testing tools are Optimizely and Performable. Optimizely gained fame for its use by the Obama campaign. Optimizely is Web-based and subscription-based, and it boasts a powerful visual interface. Performable is one of Optimizely’s chief competitors, and it provides additional robustness at the expense of some simplicity.
With both Optimizely and Performable, the developer is limited to common Web goals. That list is expansive, but it does come up short at times. Two light-code alternatives are Google Website Optimizer and Visual Website Optimizer. Google’s tool requires you to provide two unique webpages for comparison, but it is powerful, versatile and completely free. Visual Website Optimizer is not free, but the testing process is more streamlined than it is with the Google version.