Brief

Can Marketing Experimentation Become Your Superpower?

A unified, companywide approach to experimentation can generate sustained business growth.

By Kelly Edwards, Philip Dowling, Will Warren, Oliver Gertz, and Manel Oliva-Trastoy

First published on fevereiro 07, 2023
min read


Executive Summary Done right, experimentation can guide the most difficult marketing questions, such as how to allocate marketing investments, balance the master brand and sub-brands, and participate in the metaverse. At several global companies, experimentation has helped spur business growth with ROI increases of 20% or more. Every company experiments, but the experiments tend to be siloed, use mainly marketing metrics, and fall short of full potential. Best-in-class experimentation programs employ a consistent methodology that supports a broader business objective, connects results to sales, and empowers decision making. There are good reasons that Netflix runs over 1,000 marketing tests per year and Amazon more than 2,000, while Booking.com runs over 1,000 experiments at any given moment, dynamically changing the layout and structure of web pages and offerings. These companies have harnessed the discipline of experimentation to improve their business on several fronts, in response to marketplace realities. Marketing has become increasingly complex as consumers seek unique and tailored brand experiences, yet marketing budgets have come under intense scrutiny, especially with the current macroeconomic turmoil. Moreover, the proliferation of consumer data and digital technologies gives marketers more tools to determine which segments to target with what product and content, via which channels, but the sheer volume of options can be overwhelming. Experimentation done right can address these challenges, sorting the gold from the dross. For example, we’ve seen a food company use experimentation to accelerate growth by up to 5%, and a beverage company increase its digital marketing return on investment (ROI) by more than 15%. Coca-Cola’s CEO recently talked about using experimentation to optimize the firm’s marketing. Leaders in marketing experimentation often can prove or disprove a hypothesis within weeks. About EDB Singapore The Singapore Economic Development Board (EDB), a government agency under the Ministry of Trade and Industry, is responsible for strategies that enhance Singapore’s position as a global centre for business, innovation, and talent. We undertake investment promotion and industry development, and work with international businesses, both foreign and local, by providing information, connection to partners and access to government incentives for their investments. Our mission is to create sustainable economic growth, with vibrant business and good job opportunities for Singapore and Singaporeans. www.edb.gov.sg Although virtually all large consumer companies do some sort of experimentation, few consistently experiment on a large enough scale to make meaningful business decisions. Where experimentation exists, it tends to be narrowed to brand, category, or geographic silos, with limited application companywide. Inside the experiments, many companies fail to measure the full impact throughout the marketing funnel, or consistently use the optimal mix of marketing metrics and sales results. They also tend to underestimate the cultural shift required to foster collaboration across functions, more rapid decision making and action, and openness to challenging past practices or assumptions. Anchor in a business objective Firms that excel in marketing experimentation have overcome these hurdles in their own way. But they do share some characteristics that any company can learn from. For example, they anchor the experimentation program in a broader business objective. It’s not sufficient to have a hypothesis that motivates a test. One also needs to set a clear business goal—say, to raise marketing effectiveness by 20%, or generate double-digit growth for a brand—in order to galvanize the organization toward a common outcome. Experimentation becomes central to the strategy, not a side project. Kelly Edwards sits down with Coca-Cola's Romain Mallard to discuss how marketing experimentation is transforming the company. Measure across the full funnel Once the business objective is clear, accomplished experimenters assess results in marketing metrics and sales, and pressure-test commonly assumed truths with data-backed insights. We often see organizations view specific channels in isolation or default to media metrics, from click-through rates to view-through or traffic metrics, which don’t reliably predict tangible business results. To promote business growth, experiments should boost measurable performance across the full funnel, using consistent brand metrics through to offline and online sales. A full-funnel approach, which rests on a strong data foundation, is feasible across almost any industry, even where companies don’t have direct relationships with consumers and consumer sales data is more difficult to gather. This shift in approach has also benefited other industries and companies such as Allianz, a major insurance and asset-management company. A few years ago, measuring the return on marketing investments was difficult. Marketing teams had access to top and midfunnel metrics, such as unaided brand awareness and online leads, but limited access to sales data such as the number of new signed policies, so they couldn’t gauge how marketing investments affected new business growth. Allianz decided to establish a marketing experimentation capability, complemented with a return on marketing investment artificial intelligence model, to uncover the full funnel effect of marketing investments in online and offline sales channels. In one country, for example, the marketing group wanted to understand the value of investing more in paid social media channels. The team identified four local regions, used one region as control, and sharply increased social media spending in the others. They measured how greater spending influenced top-of-the-funnel metrics of reach and brand awareness, midfunnel metrics of quotes and leads, and lower-funnel metrics including the number of new policies. The results included a 25% lift in marketing effectiveness. Allianz could identify how paid social media affected each part of the funnel, and it could calculate an incremental ROI to inform budgeting and marketing-mix decisions. For instance, the experimentation revealed that digital marketing had a large effect on tied agents’ sales, since customers saw an online ad, skipped filling out the lead form, and went straight to their agent. Scale up while staying local Companies that have mastered experimentation take advantage of their organization’s scale. Experimentation doesn’t sit in pockets around the enterprise or consist of a series of isolated anecdotes. The companies orchestrate experiments across the brand portfolio and in most, if not all, markets. They allow employees to innovate within a consistently applied approach—a consistent methodology, tool kit, set of metrics, and data foundation (see Figure 1). Figure 1 Effective experimentation balances a globally consistent, scaled approach with the intimacy of experiments run in local markets Experimenting at scale allows a company to test the same hypothesis in multiple markets or brands, obtain proof points more rapidly, learn from markets that are ahead of others, and ultimately develop insights that can be deployed everywhere. The challenge is reconciling a natural tension: how to bridge the desire to experiment on locally relevant questions with the need to develop a global strategy and avoid diluting efforts or spreading resources too thin. Cocreating the learning agenda across business units and geographies helps to collectively define the best use cases and better understand how to achieve the business objective (see Figure 2). Figure 2 A learning agenda includes a range of higher- and lower-value use cases Of course, local empowerment still matters greatly, given that experiments happen at the front lines, closest to consumers. Effective scaling involves a standardized, consistent process and measurement approach. This circumvents debate about setup and aids comparison of results across functions and geographies. A global experimentation database and forums for sharing also help disseminate learnings. A hypothesis on audience targeting will play out differently in various local markets, so linking the results of each local targeting experiment will give managers confidence making a global adjustment to the audience strategy. Local teams should be equipped with an experimentation tool kit to quickly set up experiments, analyze results, and document and share the lessons. They’re supported by a small orchestration team to scale up the capability building and the insights. Consider how a year ago, beverage giant Coca-Cola set out to scale up its experimentation using a global marketing team and nine operating units with unique geographic needs and nuances. They started by focusing on digital media mix and audience targeting, using a small, central orchestration team to train local teams in each operating unit. As just one example, Coca-Cola wanted to increase spending on digital platforms, but needed to better understand whether social media, gaming, or other platforms were most effective. Using the firm’s standardized six-step approach, one country team ran an experiment on connected television (CTV), the fastest-growing video ad platform. They believed that increasing the CTV share within online video would lift sales and generate a higher return, but they also wanted to identify the maximum they could spend before hitting diminishing returns. An experiment showed that raising the share of CTV in the digital video mix beyond roughly 25% proved to be more effective until the share exceeded roughly 75%. For the first time, Coca-Cola had connected CTV ad investment to sales and not just media impressions. Once they shared the results with teams in other countries, those teams ran the same experiment with surprisingly similar results. Coca-Cola now runs 50 to 100 experiments per quarter across all the operating units, contributing to the broader business objective of 20% improvement in marketing effectiveness globally. Don’t forget culture Effective experimentation requires a cultural change throughout the organization so that the benefits persist over time. Like many a cultural challenge, this one requires a commitment from senior leaders to change their own behavior. For example, it pays to celebrate insights whether the hypothesis is proven right or wrong. Running safe experiments that confirm what you know already brings no value. Having senior leaders send clear signals to give teams autonomy and support will help to instill a discipline of progress over perfection and remove the fear of failure. Besides relying on support from the top, behavioral change will be more likely to stick when reinforced by the right incentives. These could include rewards for the number of experiments and scalable lessons, coupled with meeting progress toward the business objective. Cultural change marked Haleon, a leading consumer health company, which rolled out a sophisticated experimentation capability in the US. Initial experiments uncovered a range of opportunities to execute marketing differently, which in some instances challenged popular beliefs in the organization. To encourage the experimentation team to rapidly test and share what they learned, especially if it might disprove the logic of historical marketing decisions, leadership had to establish the right culture early on, through: regular meetings with the team to discuss both successes and failures equally; championing local marketers who challenged the status quo, through awards and opportunities to present lessons in other markets; reinforcement of the organizational commitment to learning, through regular internal communications and external publications; and ensuring that agencies and media vendors aligned with the experimentation method and invested in uncovering the best insights for each brand. Through its experimentation capability coupled with the right culture, one of Haleon’s brands in the US realized over 30% of its targeted annual revenue increase in the first three months of the program’s launch, thanks to optimized media and creative spending. This increase stemmed from experiments that increased sales 10% to 15% over previous campaign tactics. Unlocking ROI under the microscope Think of marketing experimentation as a microscope that reveals new data and insights at a deep, specific level. It complements the telescope of traditional marketing research, which is generally broader and can help inform which hypotheses to test. Once established at a large scale, a robust experimentation capability can enhance marketing-mix or econometric modeling and bring teams closer to their customers. It’s a recipe for both higher marketing ROI and stronger sales growth—it's a way of life in marketing. The authors thank Bain & Company Senior Manager David Fall for his contribution to this article.

Executive Summary

Done right, experimentation can guide the most difficult marketing questions, such as how to allocate marketing investments, balance the master brand and sub-brands, and participate in the metaverse.
At several global companies, experimentation has helped spur business growth with ROI increases of 20% or more.
Every company experiments, but the experiments tend to be siloed, use mainly marketing metrics, and fall short of full potential.
Best-in-class experimentation programs employ a consistent methodology that supports a broader business objective, connects results to sales, and empowers decision making.

There are good reasons that Netflix runs over 1,000 marketing tests per year and Amazon more than 2,000, while Booking.com runs over 1,000 experiments at any given moment, dynamically changing the layout and structure of web pages and offerings.

These companies have harnessed the discipline of experimentation to improve their business on several fronts, in response to marketplace realities. Marketing has become increasingly complex as consumers seek unique and tailored brand experiences, yet marketing budgets have come under intense scrutiny, especially with the current macroeconomic turmoil. Moreover, the proliferation of consumer data and digital technologies gives marketers more tools to determine which segments to target with what product and content, via which channels, but the sheer volume of options can be overwhelming.

Experimentation done right can address these challenges, sorting the gold from the dross. For example, we’ve seen a food company use experimentation to accelerate growth by up to 5%, and a beverage company increase its digital marketing return on investment (ROI) by more than 15%. Coca-Cola’s CEO recently talked about using experimentation to optimize the firm’s marketing. Leaders in marketing experimentation often can prove or disprove a hypothesis within weeks.

About EDB Singapore

The Singapore Economic Development Board (EDB), a government agency under the Ministry of Trade and Industry, is responsible for strategies that enhance Singapore’s position as a global centre for business, innovation, and talent. We undertake investment promotion and industry development, and work with international businesses, both foreign and local, by providing information, connection to partners and access to government incentives for their investments. Our mission is to create sustainable economic growth, with vibrant business and good job opportunities for Singapore and Singaporeans. www.edb.gov.sg

Although virtually all large consumer companies do some sort of experimentation, few consistently experiment on a large enough scale to make meaningful business decisions. Where experimentation exists, it tends to be narrowed to brand, category, or geographic silos, with limited application companywide. Inside the experiments, many companies fail to measure the full impact throughout the marketing funnel, or consistently use the optimal mix of marketing metrics and sales results. They also tend to underestimate the cultural shift required to foster collaboration across functions, more rapid decision making and action, and openness to challenging past practices or assumptions.

Anchor in a business objective

Firms that excel in marketing experimentation have overcome these hurdles in their own way. But they do share some characteristics that any company can learn from. For example, they anchor the experimentation program in a broader business objective. It’s not sufficient to have a hypothesis that motivates a test. One also needs to set a clear business goal—say, to raise marketing effectiveness by 20%, or generate double-digit growth for a brand—in order to galvanize the organization toward a common outcome. Experimentation becomes central to the strategy, not a side project.

Kelly Edwards sits down with Coca-Cola's Romain Mallard to discuss how marketing experimentation is transforming the company.

Measure across the full funnel

Once the business objective is clear, accomplished experimenters assess results in marketing metrics and sales, and pressure-test commonly assumed truths with data-backed insights.

We often see organizations view specific channels in isolation or default to media metrics, from click-through rates to view-through or traffic metrics, which don’t reliably predict tangible business results. To promote business growth, experiments should boost measurable performance across the full funnel, using consistent brand metrics through to offline and online sales.

A full-funnel approach, which rests on a strong data foundation, is feasible across almost any industry, even where companies don’t have direct relationships with consumers and consumer sales data is more difficult to gather.

This shift in approach has also benefited other industries and companies such as Allianz, a major insurance and asset-management company. A few years ago, measuring the return on marketing investments was difficult. Marketing teams had access to top and midfunnel metrics, such as unaided brand awareness and online leads, but limited access to sales data such as the number of new signed policies, so they couldn’t gauge how marketing investments affected new business growth.

Allianz decided to establish a marketing experimentation capability, complemented with a return on marketing investment artificial intelligence model, to uncover the full funnel effect of marketing investments in online and offline sales channels. In one country, for example, the marketing group wanted to understand the value of investing more in paid social media channels. The team identified four local regions, used one region as control, and sharply increased social media spending in the others. They measured how greater spending influenced top-of-the-funnel metrics of reach and brand awareness, midfunnel metrics of quotes and leads, and lower-funnel metrics including the number of new policies.

The results included a 25% lift in marketing effectiveness. Allianz could identify how paid social media affected each part of the funnel, and it could calculate an incremental ROI to inform budgeting and marketing-mix decisions. For instance, the experimentation revealed that digital marketing had a large effect on tied agents’ sales, since customers saw an online ad, skipped filling out the lead form, and went straight to their agent.

Scale up while staying local

Companies that have mastered experimentation take advantage of their organization’s scale. Experimentation doesn’t sit in pockets around the enterprise or consist of a series of isolated anecdotes. The companies orchestrate experiments across the brand portfolio and in most, if not all, markets. They allow employees to innovate within a consistently applied approach—a consistent methodology, tool kit, set of metrics, and data foundation (see Figure 1).

Effective experimentation balances a globally consistent, scaled approach with the intimacy of experiments run in local markets

Experimenting at scale allows a company to test the same hypothesis in multiple markets or brands, obtain proof points more rapidly, learn from markets that are ahead of others, and ultimately develop insights that can be deployed everywhere.

The challenge is reconciling a natural tension: how to bridge the desire to experiment on locally relevant questions with the need to develop a global strategy and avoid diluting efforts or spreading resources too thin. Cocreating the learning agenda across business units and geographies helps to collectively define the best use cases and better understand how to achieve the business objective (see Figure 2).

A learning agenda includes a range of higher- and lower-value use cases

Of course, local empowerment still matters greatly, given that experiments happen at the front lines, closest to consumers. Effective scaling involves a standardized, consistent process and measurement approach. This circumvents debate about setup and aids comparison of results across functions and geographies. A global experimentation database and forums for sharing also help disseminate learnings. A hypothesis on audience targeting will play out differently in various local markets, so linking the results of each local targeting experiment will give managers confidence making a global adjustment to the audience strategy.

Local teams should be equipped with an experimentation tool kit to quickly set up experiments, analyze results, and document and share the lessons. They’re supported by a small orchestration team to scale up the capability building and the insights.

Consider how a year ago, beverage giant Coca-Cola set out to scale up its experimentation using a global marketing team and nine operating units with unique geographic needs and nuances. They started by focusing on digital media mix and audience targeting, using a small, central orchestration team to train local teams in each operating unit.

As just one example, Coca-Cola wanted to increase spending on digital platforms, but needed to better understand whether social media, gaming, or other platforms were most effective. Using the firm’s standardized six-step approach, one country team ran an experiment on connected television (CTV), the fastest-growing video ad platform. They believed that increasing the CTV share within online video would lift sales and generate a higher return, but they also wanted to identify the maximum they could spend before hitting diminishing returns. An experiment showed that raising the share of CTV in the digital video mix beyond roughly 25% proved to be more effective until the share exceeded roughly 75%. For the first time, Coca-Cola had connected CTV ad investment to sales and not just media impressions. Once they shared the results with teams in other countries, those teams ran the same experiment with surprisingly similar results.

Coca-Cola now runs 50 to 100 experiments per quarter across all the operating units, contributing to the broader business objective of 20% improvement in marketing effectiveness globally.

Don’t forget culture

Effective experimentation requires a cultural change throughout the organization so that the benefits persist over time. Like many a cultural challenge, this one requires a commitment from senior leaders to change their own behavior.

For example, it pays to celebrate insights whether the hypothesis is proven right or wrong. Running safe experiments that confirm what you know already brings no value. Having senior leaders send clear signals to give teams autonomy and support will help to instill a discipline of progress over perfection and remove the fear of failure.

Besides relying on support from the top, behavioral change will be more likely to stick when reinforced by the right incentives. These could include rewards for the number of experiments and scalable lessons, coupled with meeting progress toward the business objective.

Cultural change marked Haleon, a leading consumer health company, which rolled out a sophisticated experimentation capability in the US. Initial experiments uncovered a range of opportunities to execute marketing differently, which in some instances challenged popular beliefs in the organization.

To encourage the experimentation team to rapidly test and share what they learned, especially if it might disprove the logic of historical marketing decisions, leadership had to establish the right culture early on, through:

regular meetings with the team to discuss both successes and failures equally;
championing local marketers who challenged the status quo, through awards and opportunities to present lessons in other markets;
reinforcement of the organizational commitment to learning, through regular internal communications and external publications; and
ensuring that agencies and media vendors aligned with the experimentation method and invested in uncovering the best insights for each brand.

Through its experimentation capability coupled with the right culture, one of Haleon’s brands in the US realized over 30% of its targeted annual revenue increase in the first three months of the program’s launch, thanks to optimized media and creative spending. This increase stemmed from experiments that increased sales 10% to 15% over previous campaign tactics.

Unlocking ROI under the microscope

Think of marketing experimentation as a microscope that reveals new data and insights at a deep, specific level. It complements the telescope of traditional marketing research, which is generally broader and can help inform which hypotheses to test. Once established at a large scale, a robust experimentation capability can enhance marketing-mix or econometric modeling and bring teams closer to their customers. It’s a recipe for both higher marketing ROI and stronger sales growth—it's a way of life in marketing.

The authors thank Bain & Company Senior Manager David Fall for his contribution to this article.