by Jacob Taylor
“It turns out, the simple model proved to be extremely good at predicting benign versus malignant. And the experts? The doctors’ diagnoses were all over the map. First, the experts didn’t agree with each other. That’s not surprising--experts have conflicting opinions all the time. Here’s the crazy part: when presented slides of the same ulcer in duplicate, every doctor ended up contradicting themselves. The experts couldn’t even agree with themselves! The stupid little model outperformed not only the average doctor in the group, it outperformed the single best doctor’s score! If your life was on the line, you’d want the dumb model, not this handsome fellow behind me making your diagnosis.”
I saw a few people shaking their heads in disbelief. I pushed on.
“How many of you have high school-aged kids?” A smattering of hands went up. “You might appreciate this one.” Behind me the slide flashed to a hopeful youth in cap and gown.
“Some college admissions require a panel interview. Students prep witty anecdotes and and rattle off lists of their extracurriculars. But there’s a problem: repeat studies have found interview scoring has no correlation with graduation rates. If the purpose of the interview was to predict who could cut the mustard in college, the results aren’t good. The interviews are completely worthless. A simple model ranking the students’ test scores has greater predictive value. So what’s the point of the expert interviews?”
“Million-dollar baseball contracts, cancer rates, and college admission are all life-changing events. Let’s get into something a little more fun. Specifically, the pricing of Bordeaux wines. I’ve been to the Big Rock Gala and I know we appreciate wine around here.” This elicited a few grins. Behind me, the slide showed a verdant winery bathed in Italian sunset.
“Researchers developed a simple four-factor equation to predict the price of Bordeaux wines. They used the age of the vintage, the average temperature during the growing season, and two separate rainfall measurements that are key during the grapes’ growth cycle. That’s it. And yet, that model explained 83% of the variation in pricing. The model routinely beat the snobby experts’ predictions.”
“Back to the serious. Hopefully something none of us have to face: parole.” Behind me a picture of a man in an orange jumpsuit stood before a panel of frowning judges.
“The justice system tries to prevent parolees from committing another crime and winding up back in prison. This failure goes by the fancy name ‘criminal recidivism.’ A study in the Pennsylvania corrections system created a simple model based on just three factors. One, the type of offense. Two, the number of past convictions. Three, the number of violations of prison rules. Very obvious measurements of rehabilitation. This “dumb” three-factor model ended up being nearly four times as accurate at predicting recidivism than the parole board filled with expert judges. Having the parolee go before a panel served no purpose.”
“Fun side note: other studies have found that you are anywhere between two- and six-times as likely to be released if you’re one of the first three cases considered during the day when compared to the last three. The reason? The judge is likely hungry or tired. At least the simple model doesn’t get hangry. If you’re ever on trial for something, ask for the earliest possible slot.” Knowing the scruples of Big Rock, it wasn’t impossible this would be relevant information for some of them.
“Back to medicine. Researchers developed a simple rules-based test that assessed intellectual deficits due to brain damage. The model correctly identified eighty-three percent of new out-of-sample cases. Pretty solid hit rate. How did the experts do? Unfortunately, experienced professionals working from the same data only scored with fifty-eight percent accuracy, dramatically underperforming the model. It gets worse for our beloved experts. Groups of inexperienced professionals given the same data scored sixty-three percent, beating the experienced cohort. The researchers attributed this bizarre finding to the experienced professionals overconfidence in their own judgment. Good thing we don’t have too many experts in this room, right?,” I said. The group laughed nervously, probably assuming I wasn’t referring to them personally. We’re always the exception.
“There’s another wrinkle to this story. A follow-up medical study gave the doctors the results of the model as part of their data to aid decision-making. The humans knew the model had scored better than they did. They took a similar test with this new information. Access to the model did improve their scores, but the humans still underperformed the simple model on its own. They still thought they could add something helpful to the model’s results. Instead, they subtracted. Let me repeat that: even when given the model’s results, the experts still underperformed.”
Heads were shaking in disbelief.
“As painful as it is to admit, the evidence is damning. A simple, quantitative model represents a ceiling in performance that us humans subtract from, and not a floor onto which we can build. The simple model just keeps beating us again and again.”
I gave them a few quiet beats to contemplate the implications.
“Perhaps you think I’m cherry-picking studies where models are the clear winners? Sadly, I’m not. Researchers conducted a meta-analysis of 136 studies where experts squared off against simple models. I’m glad you’re all sitting down before I tell you the results. In sixty-four of the studies, the simple model was the clear winner. In another sixty-four studies, the model and the experts were basically tied. That leaves just eight studies where the experts beat the simple model. Eight out of 136.”
A new slide appeared behind me. Even though I knew it was a cardinal sin of presenting, I read the quote on the slide directly to the group.
“There is no controversy in social science which shows such a large body of qualitatively diverse studies coming out so uniformly in the same direction as this one… predicting everything from the outcomes of football games to the diagnosis of liver disease--when you can hardly come up with a half a dozen studies showing even a weak tendency in favor of the clinician, it is time to draw a practical conclusion.”
- Paul Meehl, Professor of Psychology at the University of Minnesota
“I hope I’ve belabored the point enough to prove that simple models perform better than experts across a range of domains. So… what does this mean for Big Rock? What should we do with this information?”
Crickets.
“Here’s one idea. When we evaluate a company for possible acquisition, we judge management on their capital allocations skills. We assess how well they make resource decisions. We look at how well they’ve grown their company. We decide if we’re going to keep them or axe them. It’s very much like a job interview. Or a parole board review. Or a baseball tryout.” Drawing the parallels yet?
“You can see where I’m going with this. I’ve developed a simple model to judge management’s capital allocation process. No more talking to the CEO and being wowed by their words. You can’t hide from the simple model and the scorecard.”
The room was deathly silent, like outer-space. I could see a handful of now indignant faces in the audience. When it was doctors and baseball scouts, it made sense that the models performed better. But their domain was different. They were true experts at judging management. And experts don’t like being called out.
“I have a Warren Buffett quote for you that summarizes this entire presentation.” Behind me on the screen was a short quote:
“Paradoxically, when ‘dumb’ money acknowledges its limitations, it ceases to be dumb.”
-- Warren Buffett
“I don’t mean this disrespectfully, but I can guarantee you my simple model will outperform our committee of experts in evaluating managerial talent. The odds are too much in my favor.”
Well, it’s now or never. I let out a dramatic sigh. “Now comes the difficult part of the presentation. I’m going to tell you what you’ll use as excuses for why Big Rock can’t implement a simple model like this. And after that, I’m going to tell you the real reason why Big Rock will reje
ct this obviously superior system.” Crickets in outer-space.
“First, you’ll say everyone else thinks they’re an expert, but we’re actually experts. We’re the exception. OK... but did you go to eight years of intensive schooling, then several more years of formalized residency and fellowship to learn your craft? Because the doctors in the studies did. It takes an average of 40,000 hours of training to become a doctor. They thought they were the exceptions as well, and they got trounced by the simple models. So think hard before you assume you’re the exception.”
“Second, you’ll say that this doesn’t apply to our industry. Evaluating capital allocation skills is too complicated for a simple model.” I paused to build a little suspense. “It’s not. In fact, a simple model performs best under the following conditions...”
Behind me the slide changed to read:
Simple models thrive when…
- the problem is ill-structured and complex.
- the information is incomplete, ambiguous, and changing.
- the goals are ill-defined, shifting, or competing.
- the stress is high, due to time constraints and/or high stakes.
- decisions rely upon an interaction with others.
“Don’t all of these conditions look familiar?” The faces staring back at me bore countenances that ranged from skepticism to hostility.
“You’ll say our current system is working just fine, why do we need to change it? I understand the force of corporate inertia is very strong. But if we want to be the industry leaders, heck, even rebels, don’t we need to set the standard and think differently? As Sir John Templeton said, ‘The time to reflect on your investing methods is when you are most successful, not when you are making the most mistakes.’”
“Let’s talk about the real reasons why you won’t adopt the simple model.” Awkward didn’t even begin to describe the silence.
“First, following a simple model would mean no longer needing a committee to evaluate. There’s a lot of ego and power wrapped up in being a trusted expert and decision-maker. Who wants to admit a stupid model does a better job than they do? No one in this room. Or any boardroom for that matter. As Upton Sinclair remarked, ‘It is difficult to get a man to understand something, when his salary depends upon his not understanding it.’”
“Furthermore, following a simple model is a tough sell. The model won’t be perfect. Nothing is. It will make mistakes, though all the research suggests it will make fewer than a human. But as soon as the model makes a mistake, everyone looking from the outside will scoff and say, ‘You let that crummy little model decide for you? Are you daft?! What did you expect was going to happen?’ It will be hard to stick to the model and not cave to pressure. We feel safe when we do what everyone else is doing and stay with the herd. Being a rebel requires great strength. As John Maynard Keynes quipped, ‘Worldly wisdom teaches that it is better for reputation to fail conventionally than to succeed unconventionally.’”
“But…” Let’s pause to build that drama.
“...what if…” Little bit more.
“...what if you could put your ego aside and acknowledge that you might be dumb money, to use Mr. Buffett’s term? What if the ultimate expertise was knowing when to stand down?”
“Here’s my suggestion: keep the committee. Use the model as a secret weapon, like a hidden black box. Trust what it comes up with and pretend it was your expertise that found the right answer. Then spend the rest of your time golfing and congratulating yourselves for outsmarting everyone. But do not make the common mistake of trying to modify what the model comes up with because you believe you know better. You don’t. At least that claim is not supported by any of the research.”
“Any questions?”
Career suicide achieved.
CHAPTER 42
It probably won’t surprise you to hear the Big Rock higher-ups didn’t have any immediate questions. The silence was excruciating. Finally, one of them asked if I could step outside while they conferred. I closed the door behind me and sat on a bench near the door. They were going to fire me for calling them out. These aren’t people who appreciate being challenged. They do the challenging. Normally these dire circumstances would have my mind racing a million miles an hour with swirling visions of living under a bridge, destitute and alone.
Yet I was strangely calm. I was a man who had accepted his fate and awaited his turn at the gallows. There was no protest required--nothing more to say. I would die with dignity.
It felt like I’d been sitting on the bench with my thoughts for an hour. I glanced at my phone and only ten minutes had passed. The doors reopened and I was invited back in.
The group shot leary looks my way as I re-entered and stood before them. Finally, one of them spoke. “Your presentation was very… unique. It’s not often that someone speaks so frankly to us. Apparently you must have felt like you had nothing to lose to be so brash?”
Oh, boy. Here comes the guillotine.
“And yet some of us found your idea of trusting a simple model to be something worth exploring in the future. Maybe not your exact implementation, but the concept was mildly interesting.”
“Thank you,” I mumbled. Mildly interesting--that’s it?
“However, we don’t think it’s right for Big Rock at this time.” Figures.
“I understand,” I said. “I guess this means I’m fired?”
“Fired? No, actually quite the opposite. We’ve agreed that you won the analyst promotion. We could use more independent thinkers around here.” Some of the higher-ups looked at each other and smiled, proud of their benevolence. A few looked considerably less pleased--I guess they had been outvoted?
What?! I was about to get fired, and now they’re telling me I got the promotion? That I was in for a major pay raise? And that I beat Vance? Jeez, how much of my life had I wasted in worry? Worrying about things that never came to be. Why was I in my own head so much? What was I leaving on the table by caving to crushing internal doubts?
In that instant, something inside me changed. That was it. Never again.
I spoke with a new rush of confidence, “Ladies and gentlemen, thank you very much for your time. And thank you for the offer of the promotion. It’s a very kind gesture.”
They nodded approvingly.
“However, I have to decline.” So much for the nodding.
“In fact, I’m offering my resignation, effective immediately. I believe in my simple model. So much so that I have to pursue the idea farther. Maybe I’ll start my own investment fund that identifies good capital allocators by scoring them with a simple model. And I already have the name: The Rebel Allocator Fund.”
I saw nothing but looks of confusion. It’s not often you see someone transform before your eyes. I was a new person. I knew in an instant that I didn’t want to win their game or climb their ladder. I wanted to be useful in my own unique way. I was destined to make a bigger dent in the universe than anything Big Rock could offer.
“Someone very wise once told me to follow my own inner scorecard. Now if you’ll excuse me, there’s a girl I need to keep from becoming the one that got away.”
I didn’t look back as I glided out of the room, a man with a singular purpose.
I called Stephanie as soon as I was outside the Big Rock building. She answered with a sigh and a “What do you want?” Ouch.
Before we could get derailed, I told her she had to meet me at the special place of our first hike at sunrise tomorrow morning. The energy in my voice must have been just enough to sway her. She agreed to come, though she was reluctant. It didn’t matter. She said yes, which meant I still had a chance. We hung up before I could fumble the ball.
I had a date with the jeweler and a hardware store.
CHAPTER 43
The air was chill in the mountains. The stream babbled quietly as the birds awoke and chirped their morning greetings. Nature, at her most pristine. The dark sky was slow to relinquish the twinkling stars to the c
oming sunrise. I needed to be there early to pull this off.
I had barely slept, and not just because Larry’s couch was like a torture device. My god, how many crossbars in the back are required before it breaks some Geneva Convention codes? Yet I couldn’t have been more wide awake. The sun crested the mountain range and bathed the entire scene in a soft radiance.
My eyes were fixed on the trail she’d be coming up. I shifted my weight from foot to foot, both to release a little nervous energy and because I was cold. It wasn’t working much for either.
Just when I started to feel the darkness of doubt creep in, Stephanie emerged from a grove of trees. A cold hand reached down through me and grabbed my stomach. I smiled and waved like I was flagging her down in a crowded train station. We’re the only two humans within a twenty-five-mile radius, dummy.
Her hair was drawn back and her makeup light. She wore a red checkered flannel that was tied in a knot in the front over a white ribbed tank top, tan shorts, long wool socks, and her familiar hiking boots. She looked like she’d stepped right out of the Patagonia catalog of my dreams.
“Hi,” I started.
“Hi,” she replied flatly. Still cold.
“I’ve missed you,” I said.
“You have?” she said. “I wasn’t sure you’d even noticed, you’ve been so busy.”
“I know. I’m sorry I’ve been pulled in so many different directions. I’ve been struggling to keep up.”
“How’s Mr. X?” she asked.
“He... passed away,” I could barely say. Fresh tears found their way out.
“Oh, my god, Nick. I’m so sorry,” she said.