The Trouble With Physics: The Rise of String Theory, The Fall of a Science, and What Comes Next

by Lee Smolin

  We’ve already discussed the easy explanations for the failure of the last twenty-five years. It’s not for lack of data; there are plenty of unexplained results to excite the imaginations of theorists. It’s not that theories take a long time to be tested; there’s rarely been more than a decade between a new theory’s prediction of new phenomena and its confirmation. It’s not for lack of effort; far more people now work on problems in fundamental physics than in the whole combined history of the subject. And it certainly cannot be blamed on a lack of talent.

  In earlier chapters, I hypothesized that what has failed is not so much a particular theory as a particular style of research. If one spends time in both the community of string theorists and the community of people working on background-independent approaches to quantum gravity, one cannot help but be struck by a great difference in style and in the values expressed by the two communities. These differences reflect a split in theoretical physics that goes back more than half a century.

  The style of the quantum-gravity world is inherited from what used to be called the relativity community. This was led by students and associates of Einstein, and by their students in turn—people like Peter Bergmann, Joshua Goldberg, and John Archibald Wheeler. The core values of this community were respect for individual ideas and research programs, suspicion of fashion, a reliance on mathematically clean arguments, and a conviction that the key problems were closely related to foundational issues about the nature of space, time, and the quantum.

  The style of the string theory community, on the other hand, is a continuation of the culture of elementary-particle theory. This has always been a more brash, aggressive, and competitive atmosphere, in which theorists vie to respond quickly to new developments (before 1980, these were usually experimental) and are distrustful of philosophical issues. This style supplanted the more reflective, philosophical style that characterized Einstein and the inventors of quantum theory, and it triumphed as the center of science moved to America and the intellectual focus moved from the exploration of fundamental new theories to their application.

  Science does need different styles, in order to address different kinds of problems. My hypothesis is that what’s wrong with string theory is the fact that it was developed using the elementary-particle-physics style of research, which is ill suited to the discovery of new theoretical frameworks. The style that led to the success of the standard model is also hard to sustain when disconnected from experiment. This competitive, fashion-driven style worked when it was fueled by experimental discoveries but failed when there was nothing driving fashion but the views and tastes of a few prominent individuals.

  When I started my studies of physics, in the mid 1970s, both these research styles were healthy. There were many more elementary-particle physicists than relativists, but there was room for both. There were not many places for people who wanted to develop their own solutions to the deep foundational issues about space, time, and the quantum, but there were enough to support the few who had good ideas. Since then, while the need for the relativists’ style has grown, their place in the academy has shrunk, due to the dominance of string theory and other large research programs. With the exception of a single research group at Pennsylvania State University, no assistant professors working on an approach to quantum gravity not based on string theory or higher dimensions have been hired by a U.S. research university since around 1990.

  Why did the style least suited to the problem at hand come to dominate physics, both here and abroad? This is a sociological question, but it is one we must answer if we are to make constructive suggestions for restoring our discipline to its former vitality.

  To put the problem in context, we should look at some pervasive changes in the academic landscape that a young person must negotiate in order to pursue a career in science.

  The most striking change is that there is much more pressure on young people to compete for the regard of influential older scientists. The great generation that made American science, now close to retirement, may have had to compete for the top spots in elite universities and institutes, but there was not a lot of pressure if all you wanted was a professorship somewhere that gave you freedom to pursue your work. From the 1940s to the 1970s, the growth of universities was exponential, and it was common for young scientists to have several offers of faculty positions at universities upon graduation. I’ve met more than one older colleague who never once actually had to apply for a job.

  Things are different now. Universities stopped growing in the early 1970s, yet the professors hired in previous eras have continued to train graduate students at a steady rate, which means that there is a significant overproduction of new PhDs in physics and other sciences. As a result, there is fierce competition for places in research universities and colleges at all levels of the academic hierarchy. There is also much more emphasis on hiring faculty who will be funded by the research agencies. This greatly narrows the options for people who want to pursue their own research programs rather than follow those initiated by senior scientists. So there are fewer corners where a creative person can hide, secure in some kind of academic job, and pursue risky and original ideas.

  Related to this is the fact that the universities are now much more professionalized than they were a generation or two ago. Whereas university faculties have stopped growing, there has been a marked increase in the number and power of administrators. Thus, in hiring, there is less reliance on the judgment of individual professors and more on statistical measures of achievement, such as funding and citation levels. This also makes it harder for young scientists to buck the mainstream and devote themselves to the invention of new research programs.

  In our attempts to make unbiased evaluations of our peers’ work, we professors tend almost reflexively to reward those who agree with us and penalize those who disagree. Even when we rise above academic politics, we often fall into the trap of evaluating fellow scientists on the basis of one-dimensional characterizations. In faculty meetings and informal discussions, we talk about who is “good” and who is not, as if we really knew what that meant. Can a person’s life work be reduced to “Angela is not as good as Chris”? It often seems as though achievements requiring nothing more than cleverness and hard work are valued more highly than probing thought or imagination. Intellectual fads are far too important, and people who ignore them have dicey academic careers.

  I once worked on a project with a retired general who had headed a college for military officers and then become a business consultant. He talked about his frustrations in trying to work with universities. I asked him what he perceived the problem to be. He said, “There is a simple but essential thing we teach to every Marine officer, that no university administrator I’ve met seems to know: There is a big difference between management and leadership. You can manage the procurement of supplies, but you must lead soldiers into battle.” I agree with him. In my time in universities, I’ve seen much more management than leadership.

  The problem is of course not confined to science. The pace of innovation in curriculum planning and teaching methods is positively medieval. Any proposal for change has to be approved by the faculty, and in general most professors see nothing wrong with how they have been teaching for decades. I learned early how resistant universities are to change. I was fortunate to have attended a college where the first-year physics course was quantum physics. This is rare. Despite the fact that quantum physics superseded Newtonian mechanics eighty years ago, most colleges and universities in North America still postpone quantum mechanics until the third year of study, and even then it is offered only to physical science majors. Since I knew how to teach a course in freshman quantum mechanics, I proposed doing so as a graduate student at Harvard. I got a young faculty member, Howard Georgi, to agree to teach it with me, but the course was vetoed by the dean of arts and sciences. This had nothing to do with our proposal, he told me, but with the fact that it hadn’t gone through the requisite committees. “I
f we let every professor teach what they wanted to,” he said, “we would have educational chaos.” I’m not sure that educational chaos is such a bad idea; in any event, Harvard still doesn’t have a quantum mechanics course for freshmen.

  It is an unfortunate fact that the number of American students graduating with degrees in physics has been declining for decades. You might think this would lessen the competition for physics positions. It doesn’t, because the decline in undergraduate degrees is more than compensated for by increases in PhDs earned by bright, ambitious students from the developing world. The same situation obtains in other advanced countries.

  I’ve sometimes had occasion—as a member of a Yale faculty committee formed to investigate the phenomenon, and since then out of my own concern—to ask undergraduates who leave physics why they did so. One reason they give is that the physics curriculum is boring—the first year just repeats what they have learned in high school and there’s no sign of exciting topics like quantum theory, cosmology, black holes, and so on. In hopes of helping to reverse the decline in physics students, I have proposed making quantum mechanics the freshman physics course at every university that has employed me. I have been refused each time, although twice I was allowed to teach small demonstrations in quantum theory. These were successful, and a few of the students who took them are now launched on good careers.

  My purpose here is not to argue for curriculum reform, but this example suggests that universities don’t function well as vehicles for innovation, even when nothing more is at stake than modernizing a curriculum that is eight decades behind the science.

  Scientists across the board lament the pace of progress in their field. I know several biologists and experimental physicists who complain bitterly about opportunities squandered because the senior scientists in control of their departments have lost the boldness and imagination they undoubtedly possessed as new PhDs. Good ideas are not taken seriously enough when they come from people of low status in the academic world; conversely, the ideas of high-status people are often taken too seriously.

  There is no way we can address these dysfunctional trends without investigating the sociology that has fostered them. If we physicists have the hubris to try to explain the fundamental laws of nature, certainly we ought to be able to think rationally about the sociology of the academy and the counterproductive decision making that plagues our academic institutions.

  It is worth noting that the word “sociology” comes up more nowadays among string theorists than among any other group of scientists I know. It seems to be shorthand for “the view of the community.” In discussing the current state of affairs with young string theorists, you often hear them say things like “I believe in the theory, but I hate the sociology.” If you comment on the narrowness of viewpoints represented at string theory conferences or on the rapid succession of fashionable research topics from one year to the next, a string theorist will agree and add, “I don’t like it, but it’s just the sociology.” More than one friend has advised me that, “The community has decided string theory is right and there is nothing you can do about it. You can’t fight sociology.”

  A real sociologist will tell you that to understand the workings of a community you have to investigate power. Who has power over whom, and how is that power exercised? The sociology of science is not a mysterious force; it refers to the influence that older, established scientists have over the careers of younger scientists. We scientists feel uncomfortable talking about it, because it forces us to confront the possibility that the organization of science may not be entirely objective and rational.

  But after thinking about this for a long time, I have become convinced we have to talk about the sociology of theoretical physics, because the phenomena we refer to collectively as its “sociology” are having a significant negative effect on its progress. Even though most string theorists are people of integrity who pursue their work with the best of intentions, there are aspects of the field’s sociology that are aberrant, compared with the ideals that define the larger scientific community. These have led to pathologies in the methodology of theoretical physics that delay progress. The issue is not whether string theory is worth doing or should be supported, but why string theory, in spite of a dearth of experimental predictions, has monopolized the resources available to advance fundamental physics, thus choking off the investigation of equally promising alternative approaches. There is good evidence that the progress of string theory itself has been slowed by a sociology that restricts the set of questions investigated and excludes the kind of imaginative and independent-minded scientists that progress requires.

  I should point out that there has always been a dominant field within theoretical physics. At one time it was nuclear physics, then it was elementary-particle physics. String theory is only the most recent example. Perhaps the physics community is organized in such a way that there will always be a dominant field at any given moment. If so, then we need to examine why.

  The first thing that outsiders notice about the string theory community is its tremendous self-confidence. As a witness to the first superstring revolution, in 1984, I remember the sense of triumph that greeted the new theory. “It will all be over in the next twelve to eighteen months,” Dan Friedan, one of the young stars of the field, advised me. “You’d better get in while there’s still something left to do in theoretical physics.” This was just one of many assertions that things would be wrapped up quickly.

  Of course, they weren’t. But through all the subsequent highs and lows, many string theorists have continued to be supremely confident both of the truth of string theory and of their superiority over those unable or unwilling to do it. To many string theorists, especially the young ones with no memory of physics before their time, it is incomprehensible that a talented physicist, given the chance, would choose to be anything other than a string theorist.

  This attitude of course puts off physicists in other fields. Here are the thoughts of JoAnne Hewett, a particle physicist at the Stanford Linear Accelerator Center, on her blog:

  I find the arrogance of some string theorists astounding, even by physicists’ standards. Some truly believe that all non-stringy theorists are inferior scientists. It’s all over their letters of recommendation for each other, and I’ve actually had some of them tell me this to my face. . . . String theory [is perceived to be] so important that it must be practiced at the expense of all other theory. There are two manifestations of this: String theorists have been hired into faculty positions at a disproportionally high level not necessarily commensurate with ability in all cases, and the younger string theorists are usually not well educated in particle physics. Some literally have a hard time naming the fundamental particles of nature. Both of these manifestations are worrying for the long-term future of our field.1

  The arrogance Dr. Hewett describes has been a feature of the community of string theorists from the very beginning. Subrahmanyan Chandrasekhar, perhaps the greatest astrophysicist of the twentieth century, loved to tell the story of a visit to Princeton in the mid 1980s, where he was feted in honor of his recent Nobel Prize. At the dinner, he found himself seated next to an earnest young man. As physicists often do to make conversation, he asked his dinner companion, “What are you working on these days?” The reply was, “I work on string theory, which is the most important advance in physics in the twentieth century.” The young string theorist went on to advise Chandra to drop what he was doing and switch to string theory or risk becoming as obsolete as those in the 1920s who did not immediately take up quantum theory.

  “Young man,” Chandra replied, “I knew Werner Heisenberg. I can promise you that Heisenberg would never have been so rude as to tell someone to stop what they were doing and work on quantum theory. And he certainly would never have been so disrespectful as to tell someone who got his PhD fifty years ago that he was about to become obsolete.”

  Anyone who hangs out with string theorists encounters this kind of sup
reme confidence regularly. No matter what the problem under discussion, the one option that never comes up (unless introduced by an outsider) is that the theory might simply be wrong. If the discussion veers to the fact that string theory predicts a landscape and hence makes no predictions, some string theorists will rhapsodize about changing the definition of science.

  Some string theorists prefer to believe that string theory is too arcane to be understood by human beings, rather than consider the possibility that it might just be wrong. One recent posting on a physics blog laid this out beautifully: “We can’t expect a dog to understand quantum mechanics, and it may be that we are reaching the limit of what humans can understand about string theory. Maybe there are advanced civilizations out there to whom we appear as smart as dogs do to us, and maybe they have figured out string theory well enough to have moved on to a better theory. . . . ”2 Indeed, string theorists seem to have no problem believing that string theory must be right while acknowledging that they have no idea what it really is. In other words, string theory will subsume whatever comes after it. The first time I heard this view expressed, I thought it was a joke, but the fourth iteration convinced me that the speaker was serious. Even Nathan Seiberg, who is a celebrated theorist at the Institute for Advanced Study, was quoted in a recent interview as saying (“with a smile”), “If there is something [beyond string theory], we will call it string theory.”3

  A related characteristic is a sense of entitlement and a lack of regard for those who work on alternative approaches to the problems string theory claims to solve. Indeed, string theorists are normally uninterested in, and often ignorant of, anything that is not labeled string theory. In contrast to the practice at quantum-gravity meetings, the major string theory conferences never invite scientists working on rival approaches to give papers. This of course serves only to bolster the assertions of string theorists that string theory is the only approach yielding successful results on quantum gravity. The disregard of alternative approaches sometimes borders on disdain. At a recent string theory conference, an editor from Cambridge University Press confided to me that a string theorist had told him he would never consider publishing with the press because it had put out a book on loop quantum gravity. This kind of thing is not as rare as it should be.