A recent conversation on A/B testing with a client revealed an interesting perspective about messaging and positioning. The client, extolling their company’s rigorous A/B testing approach, failed to recognize a simple but scary fact: it is easy to compare multiple versions of a sub-optimal message. In the end, you end up with a “less-worse” version of an already weak message. This is not equivalent to building brand value over time — and building a moat around your brand’s essence.

What were they thinking?


The client had overlooked the obvious by ignoring underlying reasons to buy. Instead of testing which alternative was more persuasive based on price, the more important question they should have been asking was: what is the underlying motivation behind purchase? What segments, personas, or buyer types fall into our wheelhouse? Why should our brand be considered in this crowded category? This client, and so many others, seem to miss a simple tenant of marketing: why give away your marketing advantage so early in the game?


This client’s products have significant performance advantages over others in the category, yet they were A/B testing multiple executions built around being a lower-cost, value alternative. If they had taken just a little time to understand buyer behavior, they would have realized that price can be a relatively small factor in the buying decision when the brand looms large. 

In this case, A/B testing was fueling a race to the bottom. By choosing the “less worse” option, the client had already decided that they would primarily compete on price, pushing them deeper into a commodity mindset for the customer. 

When misused, A/B testing behaves like a cost-reduction test. There are many instructive lessons here: a well-known case is Maxwell House coffee. Over the course of a many years, the company increased its use of lower quality beans in the blend to cut its COGS. It conducted taste tests to make sure that consumers did not detect a difference when compared to the previous blend. But market share began to fall. Why? Because they never tested it against the original formula. What if there were thousands of Maxwell Houses across the globe instead of Starbucks? In the same way, test between meaningful options, rather than confine your evaluation to a narrow set of sub-optimal choices.


Be smart. A/B testing works best when the strategy is well-defined and plays to your advantage. First figure out what that is. Focus on highly persuasive messages that support your brand, rather than identify the best way to discount your business into oblivion. Don’t give away the store when you don’t have to.

Surveys & Forecasts, LLC