Unfortunately, this approach, conceived of and carried out in Thoughtland, produces opinions, not data. And people’s opinions, including so-called expert opinions, are not reliable predictors of success, because our thought process and conclusions are invariably distorted by a covey of cognitive errors and biases. In Thoughtland, many ideas that are The Wrong It get an enthusiastic thumbs-up from the target market (a false positive), while many ideas that are The Right It get shot down as lame or ludicrous (a false negative).
So when it comes to validating ideas for new products, we cannot depend on what people think, say, or promise. Instead, we must escape the treacherous trolls of Thoughtland by recognizing and acting upon our fourth hard fact:
Data beats opinions.
But not just any old data, especially not Other People’s Data (OPD). OPD is at best unreliable, because what happened to other people, with other products, at other times does not necessarily apply to our idea for a new product.
So the next hard fact specifies what kind of data we need:
You need to collect Your Own DAta (YODA).
If “data beats opinions” and YODA beats OPD, then does that mean that YODA beats every other possible source or form of data about the likelihood of success of your idea? Yes! Most emphatically YES! YODA beats them all—provided that YODA is collected, filtered, and analyzed rigorously and objectively.
The final hard fact assures the quality of your data:
To count as proper YODA, your market data must come with some skin in the game.
You cannot just ask the market if they are interested in and likely to buy your new product and take their word for it. You need something of value to back up their statements and promises, ideally money—the most universal and quantifiable form of skin in the game. But how do you solve the chicken-and-egg challenge of collecting YODA to determine if your idea is The Right It unless you have already built It? You use the tools in Part II.
A Review of the Sharp Tools
Finally, after a barrage of hard facts, the first bit of good news: not only is YODA the most reliable and relevant indicator of potential market success, but collecting YODA is faster, cheaper, and more fun than collecting worthless opinions and stale OPD—provided you have the right tools.
Thinking Tools
The first step in our quest for reliable YODA is to rid our idea of the mental fuzz (vague descriptions, unspoken assumptions, etc.) acquired in Thoughtland and articulate it as precisely and as clearly as possible. This step is particularly valuable for teams, because it helps identify and reconcile conceptual differences among team members.
Behind every idea for a new product is a Market Engagement Hypothesis (MEH). The MEH is a high-level description of how we assume (or hope) the market will engage with our product. In the case of Second-Day Sushi, for example, the MEH was:
Many people will buy less-than-fresh sushi if we make it cheap enough.
The Market Engagement Hypothesis is a necessary starting point, but it’s usually too vague to be usable. We need to say it with numbers, turning the fuzzy MEH into an unambiguous XYZ Hypothesis using the format: “At least X% of Y will Z.”
At least 20% of packaged-sushi eaters will try Second-Day Sushi if it’s half the price of regular packaged sushi.
Finally, we need to go from a broad XYZ Hypothesis to a set of smaller xyz hypotheses that we can test quickly and cheaply. Here’s one example:
At least 20% of students buying packaged sushi at Coupa Café today at lunch will choose Second-Day Sushi if it’s half the price of regular packaged sushi.
In three quick steps, we went from a vague high-level idea to a clearly stated, ready-to-be-tested hypothesis. Now, armed with our xyz hypothesis, we are ready to test our new product idea on our target market. Even though we don’t have the product to test our hypotheses with yet, pretotyping will enable us to forge ahead without it.
Pretotyping Tools
Pretotyping can play a key role in validating our idea. Pretotyping differs from traditional prototyping in that prototypes are usually designed, built, and used to verify that an idea can be built, to explore the best ways to build it, and to test that it will work as expected, and pretotypes are built for a single—but singularly important—purpose: to validate our Market Engagement Hypothesis.
Prototypes help us answer the key question: Can we build it? Pretotypes, in contrast, answer a very different question: Should we build it? With pretotyping, we can answer this question quickly and inexpensively. Whereas traditional prototypes can take weeks, months, or years to develop and cost millions, pretotyping experiments can produce data in a matter of hours or days and cost little.
Some pretotyping techniques include the Mechanical Turk, the Pinocchio, the Fake Door, the Facade, the YouTube, the One-Night Stand, the Infiltrator, the Relabel, but these are just a small sample of the many great effective, efficient, and fun ways to pretotype an idea. For any new product idea there is—at least—one great way to pretotype it and to get some YODA in a matter of hours. I encourage you to modify, adapt, and combine these basic techniques to best fit your idea. Better yet, invent and experiment with your own technique and give it a name (e.g., Mike’s Magic Carpet pretotype).
Analysis Tools
After running a few well-crafted pretotyping experiments you will finally possess the most valuable, relevant, and reliable type of data: fresh YODA straight from your target market. But having fresh YODA is not enough; you need to calibrate it and interpret it before you can use it to reach a conclusion on which to base a decision. A pair of analysis tools, the Skin-in-the-Game Caliper and the TRI (The Right It) Meter, will help you weigh, analyze, and interpret the YODA you collected with rigor and objectivity.
The Skin-in-the-Game Caliper allows you to assign the proper value to the data you collect according to how much skin in the game accompanies it. A $250 preorder counts more than, say, a $50 deposit to be put on the waiting list, which counts more than a submitted email address. Opinions, of course, are worth zero points; as are thumbs-ups, likes, and comments.
The TRI Meter is a graphic tool designed to help you take the result from each pretotyping experiment, put it against your XYZ and xyz hypotheses, and judge the degree to which the YODA supports the likelihood that your idea is The Right It. The TRI Meter uses a five-step scale that ranges from Very Unlikely to Very Likely, and, as a reminder that most new ideas will fail in the market, at the bottom, pointing to Very Unlikely, is the big black Law of Market Failure arrow. That arrow reminds us that we need more than just a couple of encouraging experiments to balance out those initial odds.
Since most new ideas will fail, if you’ve been objective in designing the experiments and fair in evaluating the results, you should not be surprised if your first few TRI Meters yield Unlikely or Very Unlikely scores. But if you keep at it, unless you are really unlucky, you will eventually find yourself staring at a TRI Meter with Idea arrows pointing to Likely or Very Likely. There are no guarantees, of course, but—assuming that you’ve been careful, objective, and fair in applying the tools and judging the data—a positive TRI Meter should allow you to move on to the next stage in the development of your idea: building It right.
A Review of the Plastic Tactics
Some tactics to help you use our tools more efficiently and effectively include “Think globally, test locally,” “Testing now beats testing later,” and “Think cheap, cheaper, cheapest.” Those tactics suggest that you design your experiments to minimize Distance to Data, Hours to Data, and Dollars to Data. A final tactic, “Tweak it and flip it before you quit it,” teaches the importance of tweaking your idea before abandoning it if the YODA from your initial set of experiments is discouraging. The first version of your idea may be The Wrong It, but you may be just a few tweaks away from The Right It: test, tweak, repeat.
A final comprehensive example, BusU, teaching classes on commuter buses, showed how the many tools and tactics can be applied, combined, and
put into action. We went from idea, to hypotheses, to experiments, to decisions. The example was realistic and peppered with unexpected challenges and opportunities. These demonstrated that it’s perfectly okay to tweak your original idea and change your plans along the way based on what you learn about the market as you run your experiments. In fact, it’s more than okay—it’s exactly what you should do.
There Are No Guarantees
Phew, we’ve covered a lot of ground, haven’t we? It may seem like a lot to keep track of. If I could have made it shorter or simpler, I would have, as in the quote usually attributed to Albert Einstein, “Everything should be made as simple as possible, but not simpler.” However, each of the steps fits into a logical sequence that should make it easier to remember and that, with a bit of practice, will become second nature and automatic.
Start with an idea.
Identify the Market Engagement Hypothesis.
Turn the MEH into a say-it-with-numbers XYZ Hypothesis.
Hypozoom into a set of smaller, easier to test xyx hypotheses.
Use pretotyping techniques to run experiments and collect YODA.
Use the TRI Meter with the Skin-in-the-Game Caliper to help you analyze the YODA.
Decide on the next step: Go for it! You can’t be 100% sure that your idea is The Right It, but the YODA looks promising enough for you to take a calculated risk.
Drop it. As much as you’d like your idea to succeed, the YODA stubbornly indicates otherwise.
Tweak it. In the process of testing your idea, you have learned invaluable things about your target market and its willingness (or unwillingness) to engage with your product. Don’t hesitate to tweak your original idea (or your hypotheses) to reflect what you’ve learned along the way. You may find that although there is not enough market interest in your original idea, there may be great interest in something related to it. Or try flipping the idea on its head and see what you come up with.
Even if competently executed, these steps cannot guarantee that an idea will succeed in the market. It’s impossible to predict or take steps to prevent all of the possible external factors that can turn even the most promising and well-tested idea into a market failure. But if you diligently follow and apply what I’ve shared with you in this book, I feel confident enough to make you three promises.
Promise 1: You will dramatically reduce your probability of failure. How dramatically? If you run enough well-designed pretotyping experiments, and their results provide strong and consistent validation for your hypotheses, then you have probably found an idea that is The Right It. Armed with The Right It, you should be able to flip the odds in your favor: from an 80ish percent chance of failure to an 80ish percent chance of success. Provided, of course, that you execute the idea competently (i.e., you build It right). Unforeseeable and unpreventable external events and bad luck may still cause you to ultimately fail. But if that happens I can make the next promise to you.
Promise 2: Even if you fail, you will not feel like a fool. If you crash and burn in the market because you haven’t adequately tested your idea, you should feel like a fool. Especially now, because after reading this book you should know better. But if you fail despite all your diligent and objective market testing, then you might feel disappointed and discouraged, but you should not feel foolish. If a poker player holding four aces (1 in more than 4,000 odds) bets big and loses to another player’s royal flush (1 in more than 70,000 odds), that’s not being foolish; that’s being extremely unlucky.
Promise 3: If you stick with it, you will eventually succeed. Throughout the book, I’ve talked a lot about the Beast of Failure and the Law of Market Failure. Failure is the villain of this book and, like all good villains, it grabs our interest and demands attention. But we should not forget that although market success happens much less frequently, it is not that rare. It’s not a one-in-a-million shot. It’s not like winning the lottery. It’s much, much better than that. If 80% to 90% of new products fail in the market, that means that some 10% to 20% succeed. Those are odds that we can work with.
That means that, with some luck, if you apply what you’ve learned in this book, you may have to go through and test five or ten ideas (or variations on the original idea), but you will eventually hit on an idea that is The Right It—provided, of course, that each successive idea (or idea variation) integrates and takes into account what you’ve learned from your previous market experiments. Even if a pretotyping experiment fails to validate your hypotheses, you will discover something new about the market: new facts, new opportunities you did not know about, new resources. And if you are smart, you will use that new information and those newly discovered resources to guide your next steps—which will help you find The Right It sooner.
It’s a bit like card counting in blackjack: if you keep track of the cards that have already been dealt and bet accordingly (bet less when the remaining deck is unfavorable, more when it’s favorable), you can turn the odds of winning in your favor. That’s why most casinos ban card counters. The real world is, of course, infinitely more complicated and far less predictable than a casino game. In the market, the risks, rewards, and odds of success for any given idea change all the time (that’s why it’s so important to have fresh firsthand data—YODA). But the fact remains that the more data you collect by experimenting in a given market, the better you will understand some of its behaviors, and the more likely you are to converge on an idea that is The Right It.
All right, we are almost done. But there is one final exceedingly important topic that we need to cover.
What to Build?
As I hope you will learn firsthand by applying them, the tools and tactics I’ve shared with you are powerful stuff. They have the power, if properly used, to help you avoid The Wrong It, find The Right It, and thus flip the odds for success in your favor. And that’s no small thing. In fact, it’s a huge advantage!
But, as they say, with power comes responsibility. How will you use that power? Or more precisely, what kind of new idea would you create and bring to market if you were reasonably confident (say 80ish percent) that it would succeed?
That’s a big question. An existential question. One that deserves a lot of thought, because even if you are sure that you have The Right It, it will still take a lot of time, investment, effort, sacrifice, and commitment to build It right and to keep It going. A full and proper answer to this question is beyond the scope of this book and probably deserves a book of its own. But I will briefly address two key aspects of this question:
Is this idea The Right It for you?
Is this idea The Right It for the world?
Make Sure It’s The Right It for You
The best ideas—the ones to which the market responds with the greatest interest and enthusiasm—are often the ones that require the most effort and commitment. Before you know it, you may find yourself with thousands of eager customers knocking on your (fake) door. It’s exciting, but it can also be overwhelming. And unless you are fully committed to your idea and prepared for all the work ahead, that kind of success can create more pain and headaches than a failure.
Early in my career, one of my managers had an expression for that kind of situation; he called it a success disaster. He was fond of saying, “If you think failure is hard, you haven’t experienced real success.” I didn’t fully understand what he meant until I found myself in a success-disaster scenario myself. He was right! Failure is a beast, but success can be a beast too. The experience of finding and latching on to an idea that is The Right It can often be compared to having the proverbial tiger by the tail. Just ask Tesla’s Elon Musk.
As I am writing this, Musk is experiencing just such a success disaster. In 2016 Tesla announced the long-awaited Tesla Model 3—the company’s most affordable car, about half the price of the previous models. Following the proven Tesla playbook, the company required people to put down a $1,000 deposit (skin in the game) to get in line for the car. Preorders poured in by the hundreds
of thousands (talk about YODA!). The first year of production for the Model 3 sold out in a matter of days, and the orders kept pouring in. Eventually Tesla found itself with approximately 500,000 preorders—half a billion dollars of skin in the game, an unprecedented and unqualified success by most automotive-industry standards. Now all that Tesla had to do was to build and deliver a few hundred thousand Model 3s. That’s a lot of cars for a fledgling and relatively small auto company.
Fast-forward a year or so and, not too surprisingly, a series of production issues resulted in significant delays and uncertainty in delivery dates. Many of the people who had put their $1,000 deposits in Tesla’s bank account were getting upset and wanted to know if and when they would get their cars. Elon Musk responded (via Twitter) as follows to one of those people:
You’ll know as soon as we do.
We are deep in production hell.
On another occasion, Musk tweeted the following about the situation: “The reality is great highs, terrible lows, and unrelenting stress. Don’t think people want to hear about the last two.”
But people did want to hear more about the “last two”—and how Musk handles terrible lows and unrelenting stress. He replied as follows (emphasis mine): “I’m sure there are better answers than what I do, which is just take the pain and make sure that you really care about what you’re doing.”
Thank you, Elon, for providing us with the inspiration, and even the wording, for the key message of this section—an important qualification to our main message:
Make sure you are building The Right It
and make sure that you really care about It
before you build It right.
As we’ve learned, finding The Right It is not easy; it takes a lot of work, creativity, and persistence. Unless you are very lucky, it will take several rounds of testing, failing, and going back to the drawing board before you land on an idea that survives our market validation process. So when you finally discover that winning idea for a new product, be prepared to do it justice—and be prepared for the hard work ahead. Finding The Right It is the end of one quest and the beginning of another, longer and harder quest: building It right, and then marketing It right, selling It right, servicing It right, and so on, all the while fighting the competitors that inevitably pop up when a new idea that is The Right It is discovered.
The Right It Page 19