by Gneezy, Uri
He greets the three marketing guys from Westinghouse like best friends while the hostess, outfitted in stilettos and a bottom-hugging pencil skirt, nods and smiles coyly at him. She guides the party to a linen-and-crystal-adorned table beneath a gorgeous, multicolored Tiffany-glass ceiling. He and his guests sit down and scan the menu while the waiter takes the first round of drink orders.
“Hello, Roger,” he says familiarly. “I’ll have the usual. Dry martini with triple olives. What’s your soup today?”
“Our soup is a freshly made, creamy Maine lobster bisque, sir. Delicious!”
“No, I’ll pass,” he says. “I had the lobster yesterday. I’ll take the pâté en croute to start, followed by the wild boar, and the lemon custard pie with coffee for dessert.”
Just one lunch in the day of the life of another Madison Avenue businessman, albeit nearly fifty years ago. What does the rest of his workweek look like? A short week, thanks to all those three-martini lunches that last for hours. More client meetings in the fanciest restaurants and clubs in New York City. More bourbon-laced office meetings. And, of course, the hanky-panky.
This is the kind of weird world depicted in the TV series Mad Men (and, no, it isn’t a huge exaggeration of the then-reality). The setting makes for great drama, hence the thirteen (and counting) Emmy awards. But from the perspective of the curmudgeonly economist, the series begs the question: What on earth were the businesses that hired fancy-schmancy, Don Draperesque advertising guys thinking? They were certainly creative, but how did they know that what they suggested worked?
Today, top executives might not always make big, important decisions about products, prices, and ad campaigns at booze-filled lunches, but too often they make such decisions based on little more than hunches. We think that businesses that don’t experiment—and that fail to show, through hard data, that their ideas can actually work before the company takes action—are wasting their money. Not only that, but these executives are clearly placing themselves on an endangered species list.
Netflix: A Case in Point
Netflix, the movie delivery service, is the poster child for the need for experimentation in business. Because its product and client base is second to none, it was able to avoid bankruptcy following a series of breathtakingly, entirely avoidable, bad moves it made in 2011.
Netflix was founded in 1997 on the basis of a great question: Would people pay a monthly subscription to have DVDs delivered to their doors (without late fees) instead of having to tromp down to the local video store (which made a lot of money on late fees)? The market responded with a resounding “yes.” The feisty little Silicon Valley company delivered movies people wanted, delivered them fast, and in general did a fantastic job of playing David to the Goliath of video chains such as Blockbuster.
Later, Netflix also started offering streaming online videos, though the selection was far more limited, so that customers could watch films two different ways. In so doing, it effectively disrupted brick-and-mortar video rental outfits, including the giant Blockbuster, which was forced to shutter many of its stores. With twenty-five million happy subscribers, Netflix was a darling of the stock market, too: it was trading at nearly $300 a share by July 2011.
But then, the company did something strange: it told its customers in a lengthy, rather confusing email that it was breaking up its bundled mail and streaming service into two separate services. Customers already paid $9.99, $12.99 or $14.99 a month to rent one, two, or three DVDs at a time, respectively, depending on their plan, and a limited number of streaming videos. But now, the company said, it would charge all customers $7.99 a month for one-at-a-time movies by mail and another $7.99 a month for streaming service, effectively raising its previous prices by 60 percent.
Customers vociferously disapproved of the move, likening it to a “brain fart” on the part of management. Posting a comment on the Netflix site, one fellow named Greg (signing himself as an “ex-customer”) wrote the following:
Dear Netflix,
To say the least, I am shocked and appalled at your recent behavior. It seems like yesterday we were the best of friends. You informed me with your poignant documentaries; I always laughed at your corny B horror flicks. For four years you’ve been the gracious receptacle of my hard earned money, but alas, your current actions have forced me to reevaluate our relationship. Your nominal price increase, while unexpected, does not deter my loyalty. However, your mouthpiece Jessie Becker’s presentation of this upcharge—as an added choice for my own benefit—insults my intelligence and reveals the breadth of your arrogance. Had I been treated like an adult and informed of these changes in a straightforward, honest manner, perhaps we could rekindle our spark. Unfortunately, this course of action is no longer available; your condescending and manipulative tone has irreparably ruined our relationship.1
Netflix got so many complaints that it had to hire extra customer service employees. The company’s stock plunged 51 percent. Then, in September of 2011, CEO Reed Hastings apologized to customers and announced that Netflix was going to try to correct the situation. How? By splitting the company into two operations: one, called Qwikster, would be the mail-delivery business, run by a new CEO; the other would be the online streaming service, called Netflix.
This announcement made customers even angrier. Now, subscribers to both streaming video and DVDs would see two separate charges on their credit card statements and have to log on to two different websites. The stock dropped another 7.4 percent.
Realizing they’d made things even worse, “The Netflix Team” sent out the following email to customers in October 2011:
Dear [Customer’s Name Here],
It is clear that for many of our members two websites would make things more difficult, so we are going to keep Netflix as one place to go for streaming and DVDs. This means no change: one website, one account, one password . . . in other words, no Qwikster.
Netflix thought that some customers would leave; but they were shocked that close to a million customers would drop Netflix. By this time, Netflix was being universally slammed as a badly managed company. Even Saturday Night Live wound up making fun of it.2
To see just how costly it was not to experiment, take a look at Netflix’s stock price, pre– and post–mess-up in 2011:
We tell you this story because the company could have avoided the loss of billions of dollars and the damage to its brand if it had run some simple field experiments. Rather than coming up with a national scheme to thrust upon customers, and instead of relying on rough-hewn ideas (based on the intuition of some very smart people on the board, maybe a few focus groups, or some expensive consulting firms), all Netflix had to do was run a pilot of their grand plan in a small portion of the country—say, San Diego—and then study its customers’ reactions. The small-scale experiment could have saved the company lots of money without cutting its value. Netflix might have lost a few customers in San Diego, but it would have had a chance to improve the plan (or maybe cancel it altogether) and remain the market leader. Even if this experiment had stirred up some negative attention, Netflix executives could have explained it was a local snag. The damage would have been much smaller and the experiment worth its weight in gold. Netflix has since recovered, and we expect that, given its product base and strong customer profile, the company will continue to do well, especially if it improves its performance by conducting field experiments.
When we discuss experimentation with business leaders, they usually reply by saying, “Tests are expensive to run.” After we point out that they are not, we turn the tables on them by showing how expensive it is not to experiment, as the Netflix example shows. We politely explain that every day that they set suboptimal prices, place ads that do not work, or use ineffective incentive schemes for their workforce, they effectively leave millions of dollars on the table.
Of course, many businesses do experiment, and often. Businesses always tinker with the machine and try new things. For instance, Apple’s St
eve Jobs was constantly experimenting with design and with new ways to sell products. The problem is that businesses rarely conduct experiments that allow a comparison between a treatment group and a control group. Jobs’s launch of the iPod and of the iTunes music store revolutionized an industry. But for years, Jobs insisted that recording artists and record companies charge exactly 99 cents per song on iTunes. Defending any justification Apple can offer for this policy is difficult, however. The company never compared the impact of iTunes prices on its sales of songs and iPods. And in the absence of solid evidence, Apple executives turned to their intuitions. They did well with this strategy, but could they have moved “from good to great,” as author Jim Collins puts it, through experimentation?
Put differently, say that you have a serious illness. You go to your doctor, and she prescribes a new treatment regimen for you. When you ask what evidence she has for trusting this treatment, she says, “It’s my intuition.” In such a case, you’d probably leave and never come back, because you prefer to entrust your life to someone whose medical decisions are based on scientific evidence.
How does making the right business decisions differ from choosing the right medical treatment? You might say lives aren’t at stake, but executives who are paid millions of dollars a year to sign off on decisions can cost people their jobs and the economy billions. Business experiments are research investigations that give companies the opportunity to get fast and accurate data regarding important decisions. By manipulating various factors in the environment, companies can better understand the causal relationship between a change in strategy and a response in consumers’, competitors’, employees’, or other stakeholders’ behavior.
Field experiments in business are also different from other research efforts—say, focus groups—because participants make real-life decisions often without even knowing that they are part of a study. When designed properly, business field experiments can provide invaluable insights and reveal surprising results, which the company can then implement on a larger scale. In this chapter, we tell the story of two great executives who have guided their companies’ futures with field experiments. Along the way, we mix in experiments that we have conducted with these and other firms.
Innovation at Intuit
Intuit, the Silicon Valley–based firm famous for its QuickBooks and TurboTax software, has spent years building experimentation into the core of its being. “We used to make decisions through managerial analysis and opinion, and from the top-down,” says founder and chairman Scott Cook. “Now we let our small, rapid-fire experiments make the decisions for us.”
In the old days, Intuit was run like most large organizations. Product-development folks would come up with ideas. Managers of the business units would pull together data from focus groups and other research, run some analysis, stuff their findings into PowerPoints, disseminate the information to the rest of the company, and their higher-ups would decide whether to fund the project or not. But Cook began to understand that this way of doing work was like walking in concrete shoes. “I started to be convinced that experimentation was the solution to two problems,” Cook says. “The first was how to get a large, successful company to be agile and innovative, because the larger and more successful a company it is, the less innovative and entrepreneurial it can become. The second problem was that the decisions made in the old-fashioned way were often wrong.”
Intuit trained people in “design thinking,” a methodology for investigating problems (especially fuzzy, vague ones), gathering information, and coming up with creative solutions. Design thinkers use a holistic approach, bring creativity to their work, and then innovate new approaches to problems. A small group of design thinkers and executives trained 100 leaders in the organization to run experiments that tested assumptions and hypotheses; they gathered data and came up with solutions. And these leaders taught people who worked for them to do the same. In addition, 150 “innovation catalysts” throughout the organization work in all the firm’s departments to drive this culture of experimentation. Today, everyone is encouraged to toy with new ideas using the same, Galileo-like, scientific experimental methods that we use in our work.
In the old days, people in the Turbotax.com division ran seven experiments a year. Today they are running 141 rapid, low-cost experiments during tax season on a weekly cycle, beginning on Thursdays. They test the idea, run the experiment, read the data, tweak the experiment, and the following Thursday they test again. The rapid experimentation cycle “uncorks innovation and entrepreneurship,” Cook says.
As a company, Intuit frees its employees to spend 10 percent of their time working on projects of their own invention. Today, Intuit experiments whenever possible on a small, cheap basis as the core of the discovery process. Employees who come up with innovative ideas must prove that the concepts work by tracking results from real customers; the most promising ideas rise like cream to the top. In this way, Intuit developed SnapTax (which prepares taxes on a camera or a mobile phone); SnapPayroll (which enables employers to pay their employees via mobile phone); and an Intuit Health Debit Card, which offers health coverage to small businesses that cannot afford health insurance for their employees; and more.
Very often, such experiments result in new product features. For example, the development team used specific experimental questions about their tax situations; based on the answers, the software could recommend either the standard deduction or the itemized deduction. Testing showed that the feature could reduce the time it took to finish tax forms by 75 percent, so all subsequent versions of the product incorporated the new feature, called “Fast Path,” into its free “Federal edition” of the TurboTax software.
Intuit’s development teams also created an “Audit Support Center” that helped guide all customers through the audit experience, just as if they received an audit letter from the IRS. Testing confirmed that more customers started and completed their TurboTax filings when the feature was presented on the website. “Our customer conversion rate—the number of people who purchase the product after shopping around on the Internet—is up 50 percent in six years,” says Cook.
Employees are also encouraged to come up with solutions to serious social problems. In one instance, a team in India developed a service for Indian farmers called “FASAL” (“harvest” in Hindi). The team members had observed that farm families—comprising half of Indian society—were so poor that they didn’t have access to some of the most basic necessities. How, the engineers wondered, could they make these farmers’ lives better?
The team from Intuit conducted its own study, observing the poor farmers both in the fields and when they went to market. Most farmers had access to just one or two markets at a distance from each other, and they had to work through one middleman at each market to get a price for their produce. The middleman sat under a cloth and signaled the price through hand gestures. There was no transparent pricing, and the system worked against the farmers. But the farmers had one big thing going for them: they had cell phones.
So the engineers conceived of a cell phone texting application that let farmers know what the middlemen from a variety of markets were offering. In just weeks, the engineers tested the concept with a quick-and-dirty experiment, hand-typing text messages to 120 farmers that told them which markets could secure them better prices for their crops. The test worked, and farmers began adopting the application. Today, the FASAL service is helping 1.2 million farmers out of poverty.
“FASAL is not a charity. We run it as a business, so we can attack head-on one of the most pernicious problems of the developing world, rural poverty,” says Cook. “We go out and look for the biggest problems we can solve, and a number of them are social problems. We attack these by running low-fidelity, rapid experiments.”
Working with Intuit, we now have dozens of field experiments under way that promise to shed light on what works and why. We suspect many will help move Intuit’s bottom line. Intuit is a great company because the f
ield experiment gene is built into their DNA.
Interventions at Humana
Another company that likes running field experiments is Humana, the giant health benefits firm that started out as a chain of nursing homes and hospitals. “I like to know what makes things hum,” says Mike McCallister, Humana’s affable, mustachioed chairman and CEO. Indeed, McCallister is one of those guys who is constantly thinking about better ways of doing things. In fact, he thinks a lot more like an entrepreneur—or even a field economist—than a CEO. Whereas others may trust their intuitions, he trusts his counter intuitions. “I try to find what is doable,” he says. “People assume that things are not doable, but who is to say they aren’t? Let’s find out!”
For example, in the old days before Humana was a health benefits provider, it owned hospitals and medical buildings, and McCallister was then in charge of the medical offices. The medical offices were money losers; but hospital pharmacies were money-makers. McCallister’s bright idea: attach some pharmacies to medical offices and see how they performed financially when compared to medical offices that didn’t have attached pharmacies. Lo and behold, the medical offices with the pharmacies proved more profitable. Evidence in hand, Humana expanded the pairing across its medical offices and made money. Nobody had ever tried this kind of thing before. It just wasn’t “done” at Humana, or elsewhere in the healthcare industry for that matter. Breaking the mold takes guts, we argue, and evidence from a field experiment that gives you confidence your idea is actually correct.
Once Humana switched to being a benefits provider and McCallister became CEO, he began experimenting with other policies. As an employer, Humana found that its own healthcare costs were out of control, in part because employees weren’t taking care of their own health. McCallister is a big believer in personal responsibility, so he told his employees they weren’t going to be told what to do. Employees had to work on the problem together. One approach was to run little incentivized experiments. Humana offered a weight-loss program that began and ended with a BMI (body mass index) measurement. Those who lost some of their girth had their names entered in a lottery for a hefty check for $10,000. Not surprisingly, this incentive created quite a bit of buzz around the firm—and, yes, some people lost weight.