If LadyLike beer were available, how likely would you be to choose it over white wine?
If you could choose between LadyLike and a regular light beer, which one would you choose?
Did you prefer the peach-flavored or the cantaloupe-flavored LadyLike?
Once again the results of the focus group are compiled:
47% of participants who normally choose white wine said that they would order LadyLike if it were available.
54% said that they’d choose LadyLike over regular light-beer brands.
82% preferred the peach flavor to the cantaloupe flavor.
Compelling results. The ABC executives’ eyes light up as they imagine a double-digit growth in market share. They give LadyLike the go-ahead with a multimillion-dollar marketing budget to launch the product—and start thinking about how they are going to use their bonus money.
Nine months later, accompanied by a multimillion-dollar multimedia marketing blitz, LadyLike beer hits bars and store shelves. But a few months after that, the majority of that initial shipment is still on those shelves. The few six-packs that managed to make it into household refrigerators are still there—with just one bottle missing. Despite all that market research and publicity, few women tried LadyLike, and even fewer came back for seconds. LadyLike’s slogan should have been: “One sip and that’s it.” The Law of Market Failure strikes again.
The market focus group turned out to be a hocus-pocus group—a magic trick that made months of work and millions of dollars disappear in a puff of smoke.
If I seem particularly harsh in my depiction of focus groups and similar Thoughtland-based market research and market-survey methods, it’s only because I’ve heard too many stories from people who have been badly burned by such studies. They hired the best research firms, spent huge sums of money with them, and a few months later got impressive-looking reports—which pointed them in the wrong direction. I have fallen into the Thoughtland-based market-research trap a couple of times myself, and each time it cost me and my investors years of wasted effort and millions of dollars.
Even if Thoughtland-based market research could produce more dependable results—and that’s a gargantuan if—it still would not be my first choice, because there are, as you shall see, faster, cheaper, and far better ways to get the data we need. You can use the heel of an expensive Italian shoe to drive in a nail, but why abuse a Ferragamo when you have a proper hammer at your disposal?
If your personal experience using this kind of market research (or using shoes as hammers) is more positive than mine, good for you—don’t stop on my account. But augment it with the tools and techniques I am about to show you. In other words, run two parallel market-research efforts using both methods. If the results from the two approaches don’t agree, it means that one of them is lying to you. When people need to make a major medical decision, they usually seek a second and even a third or fourth opinion. I suggest you do the same with your product decisions.
As you might have guessed by now, I am rarely satisfied by just knowing facts; I want to understand the reasons and the mechanics behind those facts—the root causes. Especially when those facts run counter to common wisdom and practices. In this case, I wanted to understand why this Thoughtland-based market research—which is so common and seems so sensible on the surface—can produce such unreliable and untrustworthy results. So, once again, I went in search of answers.
The Four Trolls of Thoughtland
Thoughtland-based market-research tools such as focus groups and market surveys are a big business. It’s not unheard of for companies to spend hundreds of thousands of dollars—even millions—on this type of research for a single new product. And, to be fair, well-planned and -executed market research of this kind can occasionally provide you with some interesting insights. But be very careful of how much weight you put on those insights, because these are Thoughtland-based tools and, because of that, they are subject to a variety of mental traps.
The Beast of Failure does not work alone; it benefits from the mischievous doings of some devoted little helpers. These impish troll-like creatures dwell in Thoughtland where, through their trickery, they create all sorts of havoc with our ideas. Four of the most common problems caused by these mind-dwelling mini-monsters are:
The Lost-in-Translation Problem
The Prediction Problem
The No-Skin-in-the-Game Problem
The Confirmation-Bias Problem
Let’s look at each of them in some detail.
The Lost-in-Translation Problem
The first problem we face in Thoughtland is one of communication. Until it’s made concrete or tangible in some form, your idea for a new product or service is just an abstraction. It’s something that you imagine or picture in your head in your own unique way. The moment you try to communicate what you see in your mind’s eye to someone else, you run into a challenging translation problem—especially if your idea is new and different from anything else they’ve seen.
This problem stems from the fact that the way you imagine the new product and its uses may be completely different from the way other people will imagine it after you describe it to them. Their interpretation of your idea is going to be distorted by their own mental trolls: their personal beliefs, preferences, and prejudices. Not only is their understanding of the idea itself likely to be different from yours, but they will judge the idea within the context of their unique mental model of the world.
When I first heard about the car service Uber, for example, I was seriously skeptical about its chances for success. This is how I saw and judged the idea in my head:
You mean, strangers get into another stranger’s car? Not a taxi with a licensed and professional driver, but just any car with any driver? Who’s going to go for that? “Don’t get into a stranger’s car” is the first thing that my mother taught me! This is a crazy idea. It’ll never work, and I’d never use it.
In my own mind, Uber was about as unsafe as hitchhiking (for both the drivers and the passengers). Even as it gained in popularity, I thought that it would become at best a niche option for a small market—definitely not a threat to taxis, limos, or public transportation. A few months later, a friend convinced me to give Uber a try on my way back from the airport: “I bet you that you’ll never set foot in a taxi again!”
I downloaded the Uber app, and a few minutes later I was sitting in a black Toyota Prius driven by a friendly, chatty twentysomething who offered me candy and bottled water and got me home safely for half the cost of a taxi. Since then, Uber has become my default choice when I need a ride. A few years after I first taught my daughter never to get into a stranger’s car, I told her that she should try Uber. Her reply? “Dad, I’ve been using Uber for months.” So much for parental advice and preconceived notions.
The Prediction Problem
Even if you manage to successfully communicate your idea without major distortions caused by the translation troll, you will run into another serious issue. People are notoriously bad at predicting whether they would actually want or like something they have not yet experienced—as well as how or how often they would actually use it.
I was a teenager living in Italy when I first heard about sushi. A friend had returned from a trip to Japan and described dishes of raw tuna, salmon, eel, and even shrimp. I thought he was pulling my leg; uncooked fish sounded like a disgusting idea. But now I love sushi, and I try to eat it at least once a week.
Let’s go back to Uber for another example of the Prediction Problem. Even though I had come to terms with the whole idea of jumping in a car driven by a stranger who is neither a taxi nor a limo driver, I initially thought I’d be using the service exactly the same way I was using taxis and limos—once or twice every few weeks. Wrong prediction—way wrong. I find myself using Uber three to four times more often than I’ve ever used taxis.
In another outcome I had not predicted, my daughter thought that using Uber all the time might prove easier and b
e more cost-effective than having her own car with all the traffic and parking hassles of San Francisco. To validate this Thoughtland-based scenario, she decided to run a six-month test to see if and how much she would miss having her own car and how the cost of using Uber would compare to the cost (insurance, maintenance, gas) of having her own car. She parked her Toyota in our driveway, dropped off her keys, and took an Uber back to her apartment. Six months later, she had the data she needed to make an informed decision. Much to her surprise and ours, she sold the car and, to this day, she has no plans to buy another one.
Bottom line: as a species, we suck at predicting if, how, and how often we might use a new product or service.
The Skin-in-the-Game Problem
The concept of “skin in the game” is central to this book; you will see it come up several times. In case you are not familiar with the expression, skin in the game refers to the idea of having a vested interest in an outcome—something to lose or gain. For example, it’s one thing to encourage your entrepreneur friend to quit her cushy job to start a new company because you think she will be hugely successful; it’s a very different thing if you back your encouragement by offering to invest $10,000 of your own money in her new company. With $10,000 of skin in the game, you will lose your investment if your friend’s business fails.
People love to give their opinions and advice; most of us do that without much thought if we have no skin in the game—because we’ve got nothing to lose or gain either way. Going back to the ABC LadyLike beer focus-group example, one of the main problems with this type of market research is that the participants don’t have any stake in the outcome. If a focus-group participant gives enthusiastic responses to the survey questions and ABC fails with LadyLike beer, it’s no foam off her nose. I’ll have a lot more to say about skin in the game later on in the book.
The Confirmation-Bias Problem
The first three problems challenge the validity of the information we collect. This last one is a problem with how we interpret that information.
The term confirmation bias refers to our (very human) tendency to seek evidence that confirms our existing beliefs or theories and to avoid or dismiss anything that runs counter to them. In other words, not only do we fail to look for objective ways to gather information, but we also fail to look objectively at the information we do get. We pick and choose and give more weight to bits of data that confirm our beliefs and dismiss those that run against them. That’s why in the US, for example, most conservatives follow conservative news channels and most liberals follow liberal news channels.
Most people don’t like to have their beliefs challenged, let alone be proven totally wrong. Confirmation bias can and does affect the way we design our experiments, interpret the results, and come to conclusions. As cognitive and mathematical psychologist Amos Tverski put it: “Once we have adopted a particular hypothesis or interpretation, we grossly exaggerate the likelihood of that hypothesis, and find it very difficult to see things any other way.”*
When the Trolls Team Up
Every one of these four basic problems can by itself lead us to reach a wrong conclusion, and here’s what happens when they are combined:
First, the original idea is distorted in translation.
Then the distorted version of the idea is viewed and judged through each person’s unique experiences and biases.
Next, a no-skin-in-the-game opinion is issued.
Finally, those no-risk opinions based on biased judgment of a distorted version of the idea are carefully selected and interpreted to confirm what we wanted to believe all along.
Bottom line: instead of dependable, objective, and actionable data, Thoughtland coughs up fur balls of subjective, biased, misguided, and misleading opinions.
It’s true that sometime these fur balls of opinion align and reflect the reality of the market—even a broken watch displays the correct time twice a day. But, more often than not, Thoughtland produces false positives and false negatives, that is, results suggesting the presence of a market that is not actually there or results suggesting the absence of a market that is actually there.
Thoughtland and False Positives
Thoughtland produces a false positive when your idea for a new product collects enough positive opinions and predictions to convince you that the idea is worth pursuing—perhaps even worth pursuing quickly and going all-in before someone else beats you to it. Armed with enthusiasm and a can’t-fail attitude, you make a big investment to develop the idea, and a few months (or years) later you launch a beautiful and competently executed new product. And then . . . nothing. All those glowing opinions and golden predictions, all those promises of “If you build it, I will come (or I will buy it, use it, adopt it)” never materialize—at least not in the numbers you were expecting.
How common are false positives caused by Thoughtland? Based on the hard fact that most new products fail in the market, I’d say they are as common as roaches in New York City. Whenever you read about the surprising failure of an exciting and promising new product, chances are that you are reading about such a scenario. And, like roaches, for every such story that you read about, you can bet that there are hundreds more that you never see.
Dramatic examples of false positives can be found in every industry and business, but I’ve chosen a particularly juicy one, an oldie but goodie, to illustrate my point: a not so little startup called Webvan.
In the late 1990s, around the time when Amazon (a great example of The Right It built right) was beginning to seriously disrupt book and CD retailers with its new online store, a group of smart, successful, and experienced people were convinced that the grocery business was ripe for an Amazon-like disruption. It sounded like a sure bet. After all, most households spend considerably more on groceries than they do on books or CDs. Furthermore, since shopping for cauliflower and cheddar cheese at a supermarket is far less fun and exciting than shopping for books or CDs, it was logical to expect an even faster and deeper market adoption.
It makes perfect sense. I love hanging around and browsing at bookstores, and I look forward to the experience—not so with supermarkets. Overall, the online grocery business appeared to be a once-in-a-lifetime opportunity, with a market that was considerably bigger and even more compelling than the market Amazon was in. Or so Webvan thought . . . in Thoughtland.
Based on that, Webvan’s founders decided to create a new company that would make it easy for people to order groceries online and have them delivered to their home at a specified time by a fleet of vans. Almost everyone who heard about the idea—business analysts, grocery-business consultants, internet pundits—agreed that this was a huge market opportunity. More important, most of the potential customers interviewed had the same enthusiastic response: “That would be awesome. I hate shopping for groceries, waiting in line, schlepping them to my car, and so on.”
In other words, lots of excitement, promises, and hot air, but no skin in the game. By the way, as a potential customer, I had exactly the same reaction as most other people. I was already addicted to Amazon, and I couldn’t wait for Webvan to launch. I had predicted not only that Webvan would be a huge success, but that our household would make the shift and do most of our grocery shopping online.
As far as Thoughtland was concerned, there wasn’t a single cloud on the horizon. All Webvan had to do was execute well—but also fast, before someone else jumped on this can’t-lose opportunity. After securing an initial round of funding of over $100 million from some of the best VCs in the industry, Webvan went on a turbocharged hiring, shopping, and building spree. It recruited hundreds of people, signed up dozens of partners, bought or built huge refrigerated warehouses, and of course purchased an impressive fleet of vans and trucks, which were painted beige with a huge company logo on the side. Eventually Webvan raised—and spent—over $800 million.
You can guess where this is going. Webvan launched with great fanfare and publicity. But somehow all those Thoughtland promis
es of masses of consumers forgoing checkout lines to do their regular grocery shopping online never materialized—at least in nowhere near the numbers predicted by their Thoughtland-based research. For whatever reason, even though the internet proved great for selling books and CDs, it did not, at least at the time, prove to be a great channel for grocery shopping.
In 2001, roughly two years after it started operations, Webvan filed for bankruptcy. Some of the company’s beige vans, which were sold at bankruptcy auctions, can still be seen driving around Silicon Valley. You can even make out a faint outline of the Webvan logo—a ghostly reminder of what can happen when you put too much trust and spend too much time in Thoughtland.
Thoughtland and False Negatives
We have seen how false positives can convince you to overinvest in a new product idea that turns out to be The Wrong It. False negatives have the opposite result; they convince you to abandon an idea that would have turned out to be The Right It.
Here’s how the false-negative scenario typically unfolds. You have what you think is a great idea: a new way to solve a common problem, a new market opportunity, a great plot for a thriller. Barely able to contain your excitement, you take your idea on a road trip through Thoughtland. You share it with family, friends, possible partners or investors, potential users—anyone who will listen. You explain your idea with the enthusiasm of a freshman cheerleader who has had one Frappuccino too many. But instead of sharing your vision and excitement, most people don’t get your idea: “Why would anyone want to use that?” “That’s a stupid idea.” “Keep your day job.” And so on.
At first, you manage to absorb a few blows. You remember that line from Rudyard Kipling’s poem “If”: “If you can trust yourself when all men doubt you.” So you pick yourself up and keep going. But after a few more poundings, you remember the next line from the poem: “But make allowance for their doubting too.” You allow some doubt to creep in and, soon after, you decide to drop your idea and wonder what on earth made you think that something like that would even work.
The Right It Page 4