Expert Commentary

A/B and Multivariate In-Market Testing: What's the Difference?

Marketers can learn a great deal about which messages, promotions or ads are effective through in-market testing.

By Paul Markowitz

First published on julio 13, 2016
min read


In marketing, the best way to learn what works and what doesn’t work is with in-market testing. Consumers or business customers receive and react to stimuli through the normal course of their activities. In most cases, recipients aren’t even aware that a test is going on. They just see an email, direct mail offer, a website, a price, a display or an ad. But as marketers, we can learn a great deal about which messages, prices, promotions or ads are effective by observing the results of these tests. Conducting the tests with companies in such industries as telecommunications, retail, financial services and consumer products, we have covered a range of situations: Pricing Promotion (coupons, rebates, buy-one-get-one) Marketing mix Ad copy Direct mail Email Banner ads Search marketing Website design The most popular method for these in-market tests is the A/B test, which uses one test cell and one control cell. The winner is the one that performs better in market. You can easily administer A/B tests and run many at the same time. For example, Progressive Insurance’s research conducts thousands of A/B tests each year. With that platform in place, Progressive can quickly gather data-driven insight into many business questions that arise. Read More Advanced Analytics Expert Commentary Success with advanced analytics requires both technical know-how and a thoughtful approach. In this series, Bain's experts offer practical advice on some of the most common data issues. An alternative method is a multivariate test, designed to systematically test several attributes or variables at once. Several test cells contain combinations of the attributes. By running a statistical model at the end, you can predict what would have happened for any combination of attributes, whether they were in a test cell or not. Here is a comparison of the two methods: Note that test design can be complex. You have to make trade-offs among sample size, sample frame, the overall number of cells, number of attributes or levels, and restrictions. Another key issue, not always apparent, is the interactions among attributes, which you must plan for in advance and account for in the model. We have conducted dozens of these tests. Occasionally, when we need a quick win, we will use the A/B test. Most often, though, we recommend multivariate tests. We typically want to test multiple attributes at one time, assign success to each of them and optimize for the best combination. The multivariate test gives us the flexibility we need to test several attributes simultaneously at the lowest cost, with the smallest sample. With a well-designed in-market test, we have seen companies achieve remarkable results (see the Bain Brief “Go Big and Get Smart: Boosting Your Digital Marketing Effectiveness” for more detail). Paul Markowitz is a principal in Bain & Company’s Advanced Analytics practice. He is based in Boston.

In marketing, the best way to learn what works and what doesn’t work is with in-market testing. Consumers or business customers receive and react to stimuli through the normal course of their activities. In most cases, recipients aren’t even aware that a test is going on. They just see an email, direct mail offer, a website, a price, a display or an ad.

But as marketers, we can learn a great deal about which messages, prices, promotions or ads are effective by observing the results of these tests. Conducting the tests with companies in such industries as telecommunications, retail, financial services and consumer products, we have covered a range of situations:

Pricing
Promotion (coupons, rebates, buy-one-get-one)
Marketing mix
Ad copy
Direct mail
Email
Banner ads
Search marketing
Website design

The most popular method for these in-market tests is the A/B test, which uses one test cell and one control cell. The winner is the one that performs better in market. You can easily administer A/B tests and run many at the same time. For example, Progressive Insurance’s research conducts thousands of A/B tests each year. With that platform in place, Progressive can quickly gather data-driven insight into many business questions that arise.

Advanced Analytics Expert Commentary

Success with advanced analytics requires both technical know-how and a thoughtful approach. In this series, Bain's experts offer practical advice on some of the most common data issues.

An alternative method is a multivariate test, designed to systematically test several attributes or variables at once. Several test cells contain combinations of the attributes. By running a statistical model at the end, you can predict what would have happened for any combination of attributes, whether they were in a test cell or not. Here is a comparison of the two methods:

ab-multivariate-test-table-01_embed

Note that test design can be complex. You have to make trade-offs among sample size, sample frame, the overall number of cells, number of attributes or levels, and restrictions. Another key issue, not always apparent, is the interactions among attributes, which you must plan for in advance and account for in the model.

We have conducted dozens of these tests. Occasionally, when we need a quick win, we will use the A/B test. Most often, though, we recommend multivariate tests. We typically want to test multiple attributes at one time, assign success to each of them and optimize for the best combination. The multivariate test gives us the flexibility we need to test several attributes simultaneously at the lowest cost, with the smallest sample. With a well-designed in-market test, we have seen companies achieve remarkable results (see the Bain Brief “Go Big and Get Smart: Boosting Your Digital Marketing Effectiveness” for more detail).

Paul Markowitz is a principal in Bain & Company’s Advanced Analytics practice. He is based in Boston.