Black Box Thinking

by Matthew Syed

  At the time, the theory that the earth moved around the sun was believed to contradict scripture. Psalm 93:1 states that “the world is firmly established, it cannot be moved.” Psalm 104:5 says: “[The Lord] set the earth on its foundations; it can never be moved.” And Ecclesiastes 1:5 says: “And the sun rises and sets and returns to its place.”

  But when Galileo invited Christian scholars to look through his telescope in order to see the new evidence, they flatly refused. They didn’t want to see any data that might count against the earth-centric view of the universe. It is difficult to think of a more revelatory episode of cognitive dissonance. They simply shut their eyes.

  As Galileo said in a letter to the German mathematician Johannes Kepler:

  My dear Kepler, I wish that we might laugh at the remarkable stupidity of the common herd. What do you have to say about the principal philosophers of this academy who are filled with the stubbornness of an asp and do not want to look at either the planets, the moon or the telescope, even though I have freely and deliberately offered them the opportunity a thousand times? Truly, just as the asp stops its ears, so do these philosophers shut their eyes to the light of truth.

  Galileo was ultimately forced to recant his views, not through rational argument, but through force. He was placed before the Inquisition and found “vehemently suspect of heresy” and ordered to “abjure, curse and detest” his opinions. He was sentenced to formal imprisonment and remained under house arrest for the rest of his life.

  According to popular legend, as Galileo retracted his views, he muttered under his breath: “But still it moves.”

  • • •

  This brief foray into the history of science shows that the basic analysis of this book is reflected in some of the most significant trends in human history. Religion was fixed in its thinking about the natural world. Knowledge was revealed from above rather than discovered through a process of learning from mistakes. That is why progress was so slow for not merely decades, but centuries.

  This takes us back to health care, where errors are also profoundly dissonant. As we have seen this has many facets, but at least one of them is the cultural insinuation that senior doctors are infallible. Is it any wonder that they find it so difficult to learn and adapt? It is noteworthy that the inability of senior doctors to embrace their flaws and weaknesses, indeed to admit that such things are even possible, is sometimes called a God complex.

  Similarly, the criminal justice system has long been infused with an almost religious air of infallibility, particularly when it comes to wrongful convictions. As we noted earlier, one district attorney said: “Innocent men are never convicted. Don’t worry about it. It is a physical impossibility.”10 But if the system is already flawless, why bother to reform it?

  Science at its best has a different approach, one based upon the bracing idea that there are things still to learn, truths yet to be discovered. As the philosopher Hilary Putnam put it: “The difference between science and previous ways of trying to find out truth is, in large part, that scientists are willing to test their ideas, because they don’t regard them as infallible . . . You have to put questions to nature and be willing to change your ideas if they don’t work.”*11

  II

  The impasse that Bacon once identified regarding natural science in the seventeenth century echoes the situation we face today with the social world. Natural science is about material objects like billiard balls, atoms and planets (physics, chemistry and the like), while social science is about human beings (such as politics, criminal justice, business, and health care). It is this world that needs to undergo a Baconian revolution.

  Take Bacon’s criticism of medieval science: that knowledge was handed down from authority figures. This tallies directly with the dogma of top-down knowledge in the social sphere today. We see this phenomenon when politicians talk about their pet ideas and ideologies—school uniform improves discipline, delinquents can be scared out of crime through prison visits, and so on. They don’t see the need for experiments or data because they think they have reached the answer through conviction or insight.

  And these habits of assumed understanding are kept in place as they once were in the natural sciences by the narrative fallacy. This is what makes us think that the world is simpler than it really is. These nice, neat, intuitive stories (think back to Scared Straight!) delude us into thinking we have a handle on real-world complexity, when often we don’t. This is not to say that narratives are not worth having; it is merely to suggest that they should be seen for what they are: rhetorical devices requiring empirical validation.

  The irony is that the social world is more complex than the natural world. We have general theories predicting the movement of the planets, but no general theories of human behavior. As we progress from physics, through chemistry and biology, out to economics, politics, and business, coming up with solutions becomes more difficult. But this strengthens rather than weakens the imperative of learning from failure.

  We need to come up with enlightened ways of making trial and error effective through the use of controlled trials and the like, and be more willing to iterate our way to success. As situations become more complex we will have to avoid the temptation to impose untested solutions from above and try to discover the world from below.

  While we have spent the last few centuries using experimentation and data in modern science, these have been largely neglected in the social world. Until 2004 there were only a few dozen controlled experiments in education, but hundreds of thousands in physics.

  And the irony is that, unlike in the medieval world, today we are fully aware of the complexity of physics. We talk about rocket science as the ultimate intellectual pursuit. We are mesmerized by Relativity and Quantum Theory. We recognize that creative people make great leaps in the natural sciences, but we also realize that this process is checked by experimentation. Scientific advance is, at least in part, precision-guided. That is Bacon’s legacy.

  But when it comes to the social world we often trust gut instinct. Political pundits range widely over various issues, making arguments on education one week, then criminal justice the next. The narratives are often powerful. But few journalists or commentators would feel entitled to argue about engineering or chemistry, at least without firm data. They would always subordinate narrative to evidence in these domains.

  And yet often in the social world this presumption is flipped. Arguments are deemed more compelling when stripped of evidence. Instead, we admire conviction, which is often a synonym for gut feeling. Chris Grayling, then the Lord Chancellor and Secretary of State for Justice in the UK, once said: “The last Government was obsessed with pilots [i.e., pilot schemes]. Sometimes you just have to believe in something and do it.” This contempt for evidence echoes the stance of the pre-scientific age.

  We noted in chapter 7 that many of the seminal thinkers of the last two centuries favored free markets and free societies precisely because they resist the human tendency to impose untested answers from above. Free markets are successful, in large part because of their capacity to clock up thousands of useful failures. Centrally planned economies are ineffective, on the other hand, because they lack this capacity.

  Markets, like other evolutionary systems, offer an antidote to our ignorance. They are not perfect, and often need government intervention to work properly. But well-functioning markets succeed because of a vital ingredient: adaptability. Different companies trying different things, with some failing and some surviving, add to the pool of knowledge. Cognitive dissonance is thwarted, in the long run, by an irrefutable failure test: bankruptcy. A company owner who runs out of money cannot pretend that his strategy was a successful one.

  Liberal societies underpinned by the values of social tolerance also harness these benefits. John Stuart Mill, the British philosopher, wrote about the importance of “experiments in living.” He based his defense of fre
edom not on an abstract value, but upon the recognition that civil society also needs trial and error. Social conformity, he argued, is catastrophic because it limits experimentation (it is the sociological equivalent of deference to authority). Criticism and dissent, far from being dangerous to the social order, are central to it. They drive new ideas and fire creativity.*

  “Protection against the tyranny of the magistrate is not enough,” Mill wrote. “[We need protection against] the tyranny of prevailing opinion and feeling; against the tendency of society to impose, by other means than civil penalties, its own ideas and practices as rules of conduct on those who dissent from them.” Mill’s notion of liberalism, like that of Popper, was largely underpinned by the insight that Bacon identified in relation to the natural sciences: the mismatch between the complexity of the world and our capacity to understand it.

  But what Mill didn’t say (unsurprisingly, given that RCTs had not become established in the culture) is that trial and error, on its own, is sometimes insufficient to drive rapid progress. Why? Because social complexity can play havoc with the interpretation of observational feedback.

  Controlled trials, where practical and ethical, have the potential to boost learning by isolating causal relationships. And yet they are not a panacea. We have to be mindful of unintended consequences and the holistic context, which are sometimes neglected by those who perform RCTs.

  Creative leaps and paradigm shifts in science, business, and technology require a capacity to connect distant concepts and ideas. Once again, we can only do this by engaging with the problems and failures that fire the imagination.

  This analysis seems to call for intellectual humility, the recognition that our ideas and theories will often be flawed. But how do we tally this with the observation that many of the most successful people are bold and sometimes even dogmatic? Entrepreneurs and scientists often risk a great deal to champion a theory or business idea. This doesn’t seem to square with the idea that science and markets are guided by learning from mistakes rather than top-down knowledge.

  Here it is necessary to distinguish between two different levels of analysis. If we return to Unilever and the nozzle, we described the approach by the mathematicians (who reasoned their way to an inadequate solution) as top-down and that of the biologists (who experimented their way to a brilliant solution) as bottom-up.

  But suppose that the team of mathematicians that came up with a defective nozzle was but one of twenty-five teams of mathematicians employed by Unilever to come up with a new design. And suppose that each of the nozzles created by these various teams was tested, with the winning nozzle used as the starting point for the teams to go back to the drawing board, to come up with a new design, and so on. Suddenly this approach starts to look very different. This is the importance of variation, a concept with parallels in biological evolution.

  When you have top-down approaches competing with each other, with a failure test to determine which of them is working, the system starts to exhibit the properties of bottom-up. That is what well-functioning markets do: entrepreneurs competing with each other, with the winning ideas replicated by the competition, which are then improved upon, and so on. Many scientists are also entrepreneurial, going against the status quo in the hope of discovering new truths.

  To put it another way, the difference between top-down and bottom-up is not just about differences in activity, it is also about the relevant perspective. It is at the level of the system that bottom-up learning is vital because of the imperative of adaptability. And that is the story of aviation, well-functioning markets, biological evolution, and, to a certain extent, the common law.

  At the level of individuals the question is more open. Do individual organizations progress faster when they iterate their way to success or when they come up with bold ideas and stick to them doggedly? In high tech, as we have seen, the world is moving so fast that entrepreneurs have found it necessary to adopt rapid iteration. They may have bold ideas, but they give them a chance to fail early through the minimum viable product (MVP). And if the idea survives the verdict of early adopters, it is iterated into better shape by harnessing the feedback of end users.

  In other words, competition has favored entrepreneurs that take bottom-up learning seriously rather than those that do not. And that is a powerful operating assumption in a rapidly changing world. If valid learning can be achieved through iteration at a fast pace and low cost, it is crazy to pass up the opportunity. Success, at the level of the individual as well as at the level of the system, will increasingly hinge on adaptability.

  In other words, learning from failure.

  III

  Having looked at the big picture, let’s narrow the focus and look at how we can wield the lessons of this book in a practical way. How can we harness the power of learning from mistakes in our jobs, our businesses, and in our lives?

  The first and most important issue is to create a revolution in the way we think about failure. For centuries, errors of all kinds have been considered embarrassing, morally egregious, almost dirty. The French Larousse dictionary historically defined error as “a vagabondage of the imagination, of the mind that is not subject to any rule.”

  This conception still lingers today. It is why children don’t dare to put their hands up in class to answer questions (how embarrassing to risk getting an answer wrong!), why doctors reframe mistakes, why politicians resist running rigorous tests on their policies, and why blame and scapegoating are so endemic.

  As business leaders, teachers, coaches, professionals, and parents, we have to transform this notion of failure. We have to conceptualize it not as dirty and embarrassing, but as bracing and educative. This is the notion we need to instil in our children: that failure is a part of life and learning, and that the desire to avoid it leads to stagnation.

  We should praise each other for trying, for experimenting, for demonstrating resilience and resolve, for daring to learn through our own critical investigations, and for having the intellectual courage to see evidence for what it is rather than what we want it to be.

  If we only ever praise each other for getting things right, for perfection, for flawlessness, we will insinuate, if only unintentionally, that it is possible to succeed without failing, to climb without falling. In a world that is complex, whose beauty is revealed in its intricacy and depth, this is misconceived. We have to challenge this misconception, in our lives and in our organizations.

  To do so would be nothing less than revolutionary. A liberating attitude to error would change almost every aspect of our professions, schools, and political institutions. It will not be easy; there will doubtless be resistance, but the battle is worth it. Instead of shying away from criticism and inconvenient evidence, we should embrace them.

  As the author Bryan Magee, drawing on the work of Karl Popper, put it:

  No one can possibly give us more service than by showing us what is wrong with what we think or do; and the bigger the fault, the bigger the improvement made possible by its revelation. The man who welcomes and acts on criticism will prize it almost above friendship: the man who fights it out of concern to maintain his position is clinging to non growth. Anything like a widespread changeover in our society toward Popperian attitudes to criticism would constitute a revolution in social and interpersonal relationships—not to mention organizational practice.12

  Once we have this new mindset, we can start to create systems that harness the power of adaptivity in our lives. What does this mean in practice? Well, let us start with how to improve our judgments and decision-making. We noted in chapter 3 that intuitive judgment improves when it is given a chance to learn from mistakes. This is how chess masters build their skill and how pediatric nurses are able to detect illnesses that are apparently invisible.

  But consider the following questions. Do you fail in your judgments? Do you ever get access to the evidence that shows where you might be
going wrong? Are your decisions ever challenged by objective data? If the answer to any of these questions is no, you are almost certainly not learning. This is not a question of motivation or diligence, but of iron logic. You are like a golfer playing in the dark.

  Think back to the example of psychotherapists from chapter 3. They are often industrious, caring and compassionate—and yet many don’t improve with time on the job. Why? The reason is simple. Most psychotherapists gauge how their clients are responding to treatment not with objective data, but by observing them in clinic. But this data is highly unreliable since patients might exaggerate how well they are to please the therapist. Moreover, psychotherapists rarely track their clients after therapy has finished. This means that they do not get any feedback on the lasting impact of the treatment.

  So, how to address this problem? It is possible to see the basic contours of an answer without even knowing much about psychotherapy itself. Psychotherapists need to access the data on where they are going wrong, so they have an opportunity to reform and refine their judgments and, at a deeper level of adaptation, the models they use to make sense of the problems they are confronting.

  With this in mind, consider what would happen if psychotherapists used a standardized and proven interview procedure to assess well-being in their patients. Suddenly they would have more objective information about how their clients are progressing. And if long-term outcomes were carefully tracked relative to valid historical data of similar cases, clinicians would have direct feedback on how patients were faring relative to established norms.

  The stage is set for meaningful evolution. The lights have been switched on. As a landmark paper by a team of psychologists, which set out these proposals in detail, put it: “Increasingly, there are reliable benchmarks for various disorders to which therapists can compare the progress of their clients. Therapists can use feedback about client progress to adjust therapy to achieve optimal outcomes.”13