The Trouble With Physics: The Rise of String Theory, The Fall of a Science, and What Comes Next

by Lee Smolin

  The big excitement of 1984–85 was due partly to the five original superstring theories being proved finite to the first approximation. A few years later, a paper was published by the well-established theorist Stanley Mandelstam that was taken as proving all of the infinite number of terms finite.12

  At the time, the response to Mandelstam’s paper was mixed. Indeed, there is an intuitive argument—which many string theorists believe—strongly suggesting that if the theory exists at all, it will give finite answers. At the same time, several mathematicians I knew who were experts in the technical issues involved denied that the argument was a complete proof.

  I didn’t hear much about the issue of finiteness for many years. It simply faded into the background as the field moved on to other problems. From time to time, a paper would appear on the Internet addressing the issue, but I paid little attention. Indeed, I don’t recall doubting the finiteness of the theory at all, until recently. Most of the developments I followed in the last twenty years, and a good deal of my own work in the area, were based on the assumption that string theory was finite. I heard many talks by string theorists over the years that began with the claim that the theory gave a “finite quantum theory of gravity,” before going on to deal with a problem of current interest. Many books were written, and talks given for the public, asserting that string theory was a sensible quantum theory of gravity and either explicitly or implicitly claiming that the theory was finite. As far as my own work was concerned, I believed that string theory had been proved finite (or almost proved finite, up to filling in some technical detail only a mathematician would worry about), and this was a major reason for my continuing interest in it.

  In 2002, I was asked to write and present a review of the whole field of quantum gravity to a conference being organized in honor of John Wheeler, one of its founders. I decided the best way to review the subject would be to write down a list of all the major results established so far by the various approaches. My hope was to make an objective comparison of how well each approach was doing in the drive toward the goal of a theory of quantum gravity. I wrote a draft of the paper and, naturally, one of the results on my list was the finiteness of superstring theory.

  To finish the paper, I of course had to find proper citations to papers where each of the results listed was demonstrated. For most of them, this proved no problem, but I ran into trouble in my search for the right citation for the proof of the finiteness of string theory. Looking at different sources, I found referenced only the original paper by Mandelstam—the one that, I had been told by mathematicians, was incomplete. I found a few other papers on the problem, none of them claiming a final result. I then began asking string theorists I knew, in person and by e-mail, about the status of finiteness and where I could find the paper containing the proof. I asked a dozen or so string theorists, young and old. Almost all who answered told me that the result was true. Most didn’t have the citation for the proof, and those who did gave me the paper by Mandelstam. In frustration, I consulted review papers—these are papers written to survey the main results of a field. Of more than fifteen review articles I consulted, most either said or implied that the theory was finite.13 For citations, I found only earlier review papers or the paper of Mandelstam. I did find one review paper, by a Russian physicist, explaining that the result was unproved.14 But it was hard to believe that he was right and all the reviews by better-known people, most of whom I knew and admired, were wrong.

  Finally, I queried my Perimeter colleague Robert Myers. He told me, with his usual refreshing candor, that he didn’t know whether finiteness had been completely proved, but he suggested that someone named Eric D’Hoker might. I looked him up, and this is how I finally found that D’Hoker and Phong had, just in 2001, succeeded in proving the finiteness to the second order of approximation (see chapter 12). Until then, over the seventeen years since 1984, no substantial progress had been made. (As I mentioned in chapter 12, there has been some progress in the four years since D’Hoker and Phong’s paper, mainly by Nathan Berkovits. But his proof relies on additional unproved assumptions, so, while it is a step forward, it is not yet a complete proof of finiteness.) Thus, the fact was that only the first three out of an infinite number of terms in the approximation were known to be finite. Beyond that, whether the theory is finite or infinite was (and is) simply not known.

  When I described this situation in my review paper, it was greeted with disbelief. I got several e-mails, not all of them polite, claiming that I was mistaken, that the theory was finite, and that Mandelstam had proved it. I had a similar experience talking to string theorists; some of them were shocked to hear that the proof of finiteness had never been completed. But their shock was as nothing compared with that of those physicists and mathematicians I talked to who were not string theorists, and who had believed that string theory was finite because they had been told that it was. For all of us, the impression of string theory as finite had had a great deal to do with our acknowledgment of its importance. None of us could recall ever having heard a string theorist point to it as an unsolved problem.

  I also felt somewhat peculiar at having to present a paper that aimed to make a detailed assessment of the evidence supporting various conjectures in string theory. Certainly, I thought, this was something that one of the leaders of the field should be doing periodically. This kind of critical review paper, emphasizing the key unsolved problems, is common in quantum gravity, cosmology, and, I suspect, most other fields of science. Because this was not done by any of the leaders of string theory, it was left to someone like me, as a quasi “insider” who had the technical knowledge but not the sociological commitment, to take on that responsibility. And I had done so because of my own interest in string theory, which I was working on almost exclusively at the time. Nevertheless, some string theorists regarded the review as a hostile act.

  Carlo Rovelli, of the Centre de Physique Théorique in Marseille, is a good friend who works in quantum gravity. He had the same experience when he incorporated the statement that string theory had never been proved finite into a dialogue he wrote dramatizing the debate between the different approaches to quantum gravity. He got so many e-mails asserting that Mandelstam had proved the theory finite that he decided to write to Mandelstam himself and ask his view. Mandelstam is retired, but he responded quickly. He explained that what he had proved is that a certain kind of infinite term does not appear anywhere in the theory. But he told us that he had not actually proved that the theory itself was finite, because other kinds of infinite terms might appear.15 No such term has ever been seen in any calculation done so far, but neither has anyone proved that one couldn’t appear.

  None of the string theorists I’ve discussed these issues with have decided, on learning that the theory has not been proved finite, to stop working on string theory. I’ve also encountered well-known string theorists who insisted that they had proved the theory’s finiteness decades ago and didn’t publish only because of some technical issues that remained unresolved.

  But when and if the issue of finiteness is settled, we will have to ask how it happened that so many members of a research program were unaware of the status of one of the key results in their field. Should it not be of concern that between 1984 and 2001 many string theorists talked and wrote as if it were a fact that the theory was finite? Why did many string theorists feel comfortable talking to outsiders and insiders alike, using language that implied the theory was fully finite and consistent?

  Finiteness is not the only example in string theory of a conjecture that is widely believed but so far unproved. As we discussed, there are several versions of the Maldacena conjecture in the literature, and they have very different implications. What is sure is that the strongest of these conjectures is far from proved, although some weak version is certainly well supported. But this is not how some string theorists see it. In a recent review of the Maldacena conjecture, Gary Horowitz and Joseph Polchinski compare it to a well-kno
wn unsolved conjecture in mathematics, the Riemann hypothesis.

  In summary, we see convincing reason to place [Maldacena’s duality conjecture] in the category of true but not proven. Indeed, we regard it on much the same footing as such mathematical conjectures as the Riemann hypothesis. Both provide unexpected connections between seemingly different structures . . . and each has resisted either proof or disproof in spite of concentrated attention.16

  I’ve never heard a mathematician refer to a result as “true but not proven,” but beyond that, what is astounding about this assertion is that the authors, two very smart people, ignore an obvious difference between the two cases they discuss. We know that the structures related by the Riemann hypothesis both exist mathematically; what is in question is only a conjectured relation between them. But we do not know that either string theory or the supersymmetric gauge theories really exist as mathematical structures; indeed, their existence is part of what is in question. What this quote makes clear is that these authors reason from the assumption that string theory is a well-defined mathematical structure—despite wide agreement that even if it is true, we have no idea what that structure is. If you don’t make this unproved assumption, then your evaluation of the evidence for the strongest version of the Maldacena conjecture must disagree with theirs.

  When it comes to defending their belief in these unproved conjectures, string theorists often note that something is “generally believed” in the string theory community, or that “no sensible person doubts that it’s true.” They seem to feel that appeal to consensus within their community is equivalent to rational argument. Here is a typical example, from the blog of a well-known string theorist:

  Anyone who hasn’t been asleep for the past 6 years knows that quantum gravity in asymptotically anti–de Sitter space has unitary time evolution. . . . With the large accumulation of evidence for AdS/CFT, I doubt there are many hold-outs left who doubt that the above statement holds, not just in the semiclassical limit that Hawking considers, but in the full nonperturbative theory.17 (Italics mine.)

  It doesn’t feel good to have to admit to being one of the hold-outs, but that is what a detailed examination of the evidence forces me to be.

  This cavalier attitude toward precise support for key conjectures is counterproductive for several reasons. First, in combination with the tendencies described earlier, it means that almost no one works on these important open problems—making it more likely that they will remain unsolved. It also leads to a corrosion of the ethics and methods of science, because a large community of smart people are willing to believe key conjectures without demanding to see them proved.

  Moreover, when great results are discovered, they are often exaggerated. Several non–string theorists have asked me why I work on anything else, when string theory has completely explained black-hole entropy. While I greatly admire the work on extremal black holes by Strominger and Vafa and others (see chapter 9), I must reiterate that, for what appear to be good reasons, the precise results have not been extended to black holes in general.

  Similarly, the claim that a vast number of string theories exist with a positive cosmological constant (the much-discussed “landscape”) is far from secure. Yet some leading string theorists are willing, on the basis of these weak results, to make grand pronouncements about string theory’s success and future prospects.

  It may well be that the persistent exaggeration has benefited string theory over its rivals. If you were a department head or an officer of a granting agency, wouldn’t you be more likely to fund or hire a scientist who worked on a program said to solve the big problems in the field over a scientist who could claim only that he or she had evidence that there might exist a theory—so far unformulated—that had the potential to solve the problems?

  Let me summarize, so we can see where this is taking us. The discussion has brought out seven unusual aspects of the string theory community.

  Tremendous self-confidence, leading to a sense of entitlement and of belonging to an elite community of experts.

  An unusually monolithic community, with a strong sense of consensus, whether driven by the evidence or not, and an unusual uniformity of views on open questions. These views seem related to the existence of a hierarchical structure in which the ideas of a few leaders dictate the viewpoint, strategy, and direction of the field.

  In some cases, a sense of identification with the group, akin to identification with a religious faith or political platform.

  A strong sense of the boundary between the group and other experts.

  A disregard for and disinterest in the ideas, opinions, and work of experts who are not part of the group, and a preference for talking only with other members of the community.

  A tendency to interpret evidence optimistically, to believe exaggerated or incorrect statements of results, and to disregard the possibility that the theory might be wrong. This is coupled with a tendency to believe results are true because they are “widely believed,” even if one has not checked (or even seen) the proof oneself.

  A lack of appreciation for the extent to which a research program ought to involve risk.

  Of course, not all string theorists can be described this way, but few observers, inside or outside the string theory community, will disagree that some or all of these attitudes characterize that community.

  I want to be clear that I am not criticizing the behavior of specific individuals. Many string theorists are personally open-minded and self-critical, and if asked, they will say that they deplore these characteristics of their community.

  I must also be clear that I am as much at fault as my colleagues in string theory. For many years, I believed that basic conjectures such as finiteness were proved. This is largely why I invested years of work in string theory. More than just my own work was affected, for among the community of people who work on quantum gravity, I was the strongest advocate for taking string theory seriously. Yet I did not take the time to check the literature, so I, too, was willing to let the leaders of the string theory community do my critical thinking for me. And during the years I worked on string theory, I cared very much what the leaders of the community thought of my work. Just like an adolescent, I wanted to be accepted by those who were the most influential in my little circle. If I didn’t actually take their advice and devote my life to the theory, it’s only because I have a stubborn streak that usually wins out in these situations. For me, this is not an issue of “us” versus “them,” or a struggle between two communities for dominance. These are very personal problems, which I have been contending with internally for as long as I have been a scientist.

  So I sympathize strongly with the plight of string theorists, who want both to be good scientists and to have the approval of the powerful people in their field. I understand the difficulty of thinking clearly and independently when acceptance in your community requires belief in a complicated set of ideas that you don’t know how to prove yourself. This is a trap it took me years to think my way out of.

  All of which bolsters my conviction that we theoretical physicists are in trouble. If you ask many string theorists why scientists working on alternatives to string theory are never invited to string theory conferences, they will agree with you that such people should be invited, they will deplore the current state of affairs, but they will insist that there’s nothing they can do about it. If you ask them why string theory groups never hire young people working on alternatives as postdocs or faculty or invite them as visitors, they will agree with you that this would be a good thing to do, and they will lament the fact that it isn’t being done. The situation is one in which there are big issues that many agree on but no one feels responsible for.

  I strongly believe in my string theory friends. I believe that as individuals, they are almost all more open-minded and self-critical and less dogmatic than they are en masse.

  How could a community act in a way so at odds with the goodwill and good sense of its individual memb
ers?

  It turns out that sociologists have no problem recognizing this phenomenon. It afflicts communities of highly credentialed experts, who by choice or circumstance communicate only among themselves. It has been studied in the context of intelligence agencies and governmental policy-making bodies and major corporations. Because the consequences have sometimes been tragic, there is a literature describing the phenomenon, which is called groupthink.

  Yale psychologist Irving Janis, who coined the term in the 1970s, defines groupthink as “a mode of thinking that people engage in when they are deeply involved in a cohesive in-group, when the members’ strivings for unanimity override their motivation to realistically appraise alternative courses of action.”18 According to this definition, groupthink occurs only when cohesiveness is high. It requires that members share a strong “we-feeling” of solidarity and a desire to maintain relationships within the group at all costs. When colleagues operate in a groupthink mode, they automatically apply the “preserve group harmony” test to every decision they face.19

  Janis was studying failures of decision making by groups of experts, such as the Bay of Pigs. The term has since been applied to many other examples, including the failure of NASA to prevent the Challenger disaster, the failure of the West to anticipate the collapse of the Soviet Union, the failure of the American automobile companies to foresee the demand for smaller cars, and most recently—and perhaps most calamitously—the Bush administration’s rush to war on the basis of a false belief that Iraq had weapons of mass destruction.