The Half-Life of Facts

by Samuel Arbesman

  Here’s what Ising recognized: There is a temperature value that can be solved for mathematically and that precisely represents the transition between the two phases: between when everything is the same color and when it’s all random.

  The Ising model—which is referenced by thousands of articles in fields that range from biology to the social sciences—is a mathematical abstraction that has been used to explain all sorts of phase transitions, many outside the realm of physics. Phase transitions have been used to understand all types of rapid change, from ecological models that explore ecosystem collapse and abrupt climate change to tipping points for how fads and fashions spread.

  But what about facts? Can these sorts of models provide insight into how changes in knowledge operate?

  It turns out that knowledge is not that different from abstract magnetic models. While the underlying changes in our knowledge develop over time, we can have big sudden jumps in something else we might be examining. In the Ising model, we change something slowly and continuously—temperature—and this change yields a sudden jump in something else: the state of the system, from gas to solid, for example. In the world of facts, this sort of thing can also happen: A sudden jump in our knowledge might be due to some other facts changing and being slowly accumulated. But this still sounds very abstract; how might this work in practice?

  • • •

  ONE of the biggest facts in human history was overturned abruptly, and it was done in a single footprint: Since the dawn of history, no human had walked on the moon. That was a fact whose status hadn’t budged at all for thousands of years. At no point did someone walk on the moon even a little bit. At 10:56 P.M. EDT on July 20, 1969, this all changed with a single step.

  But could this phase transition have been predicted as growing out of some underlying regular change? Yes. There were underlying quantifiable patterns occurring over the previous decade that would have allowed the prediction of this phase transition in humanity’s place in the solar system.

  Prior to Apollo 11, multiple unmanned and manned Apollo missions had been launched. Apollo 10 actually did everything but land on the Moon—it left the earth’s orbit, circled the moon, and returned home. It even came within fifteen kilometers of the moon’s surface. Despite what appears to be an abrupt transition in what we know, there was a steady progression of underlying changes that can explain the jump.

  In fact, it’s not even the few missions just prior to the moon landing that could help explain the transition; hundreds of years of smooth data can actually be used to explain this steady march. In 1953,2 Air Force scientists in the Office of Scientific Research and Development created a simple chart of the fastest man-made vehicles at each moment in history for the previous couple of centuries. They found something incredible: If they extrapolated the exponential curve outward, the data showed that speeds necessary to leave the Earth’s surface could be reached within four years! A curve had implied the existence of the first artificial satellite well before Sputnik’s launch, and exactly as predicted (Sputnik went into orbit on October 4, 1957).

  Furthermore, the Air Force scientists realized that the chart predicted that the speeds required to land on the moon were reachable a few years after that. And the moon landing came to pass, also right on schedule. Interstellar spacecraft are also included in the plot, and while we haven’t succeeded in that domain yet, NASA recently announced an initiative to begin thinking about ships with the capability of reaching the stars.

  What this means is that the factual phase transition of the moon landing was long in the cards—it was simply due to a steady change in the underlying speeds we could achieve.

  These sorts of phase transitions occur in many places in the world of facts. As mentioned before, Pluto was a planet for nearly a hundred years, and then, suddenly, it wasn’t. Everyone knew that smoking was fine for you. Doctors even endorsed certain brands. Then, in the 1950s, it became abundantly clear that smoking kills.

  But these can likely be explained by underlying steadier changes. The detection of larger and larger objects at the fringe of the solar system meant that Pluto’s status was due for reconsideration. And the proliferation of medical studies meant that it was only a matter of time before we would learn of the dangers of smoking.

  In contrast, the number of mammals that we know of is a slower, steadier progression of knowledge. No single discovery greatly changes our awareness of the number of species. If we know of thousands of mammals, no single additional discovery is going to overturn that fact.

  Sometimes, though, the line between these two types of knowledge change is a bit less clear. For example, while finding the first extrasolar planet is a phase transition, going from 400 to 401 known extrasolar planets is a more subtle and steady progression. In these cases, and many others like them, knowledge accretes slowly, and often according to the regular patterns discussed in chapter 2.

  But understanding these steady progressions is the key to understanding the rapid shifts in our knowledge. These steady progressions are the underlying slow temperature changes that result in the fast phase transition that we see when looking at everything from a different scale. Of course, it’s not always that easy. Figuring out the right underlying change to measure3 requires a bit of ingenuity. But doing this can help explain the many rapid shifts in our knowledge that are all around us.

  • • •

  EVER since the first planet was discovered outside our solar system in 1995, anticipated by Carl Sagan in Cosmos, we have actually been waiting for another large-scale shift in our awareness of our place in the universe. We have been searching for a planet that is truly like Earth, or one that could at least potentially be habitable. As one of the leaders in the field told me, this is ultimately what many of the astronomers in the area of extrasolar planetary research are after. Of course, there are many other topics to study, such as the true distribution of the types and sizes of planets that exist in our stellar neighborhood. But a paramount goal has been to find a planet truly similar to our own.

  Finding one that actually harbors life will be a huge milestone for humanity, and it will be one of those massive phase transitions in how we view the world around us.

  But just like landing on the moon, a path to such a discovery proceeds by increments. While we are fairly certain that life can exist in far stranger spaces than we can possibly imagine, good initial spaces to look are planets that are similar to Earth. And short of being able to examine the atmospheres of these planets for such hallmarks of life as oxygen, we use simpler proxies. These proxies take the form of mass and temperature. If a planet can have something like liquid water due to its surface temperature, and is about the same size as Earth (meaning our style of life could have evolved there), this planet is deemed potentially habitable, at least as a first pass.

  So while detecting the first potentially habitable planet outside our solar system is a stunningly discontinuous jump in our knowledge,4 can it be predicted? This is what Gregory Laughlin, an astronomer at the University of California, Santa Cruz, and I tried to do in the summer of 2010. Greg is an expert on extrasolar planets, has discovered numerous ones himself, and writes the premier blog for the field, Systemic at oklo.org.

  Greg and I knew that the discovery of a roughly Earth-like planet was on the horizon. NASA’s Kepler mission had been running for more than a year and, in fact, in mid–June 2010 had released a whole slew of exoplanet candidates, tantalizingly withholding the most promising candidates until the following February. We knew that we were in a special window of time, both for the discovery itself and for making a prediction.

  We created a simple metric of habitability.5 Each previously discovered planet was rated on a scale from 0 to 1, where 0 is not habitable at all and 1 is quite like Earth, at least when it comes to mass and temperature. Most planets are 0 or very close to 0, since most planets are either scorchingly hot or
unbelievably cold (and sometimes both at different parts of their year). But when we looked at the highest habitability values over time, we saw a clear upward trend: By focusing on the most Earth-like planet discovered each year and charting their habitability values, we found that these have been steadily proceeding upward over time. In fact, this upward trend conformed to our old friend the logistic curve. Just as scientific output in general fits exponential and logistic curves, the march toward the discovery of a potentially Earth-like planet fits one of these omnipresent functions as well.

  Since it conformed to a mathematical shape, we could predict when these values would reach something that was habitable by extrapolating these curves into the future. Essentially, the properties of the discovered planets and their values of habitability act as the underlying temperature of the system. As time proceeds, the highest habitability values of discovered planets steadily increase. When we reach a very high habitable value, we jump in our knowledge: We go from not knowing of any planets that are potentially like Earth to knowing of one. Extrasolar planet discovery data allow us to see the microscopic behavior, the small underlying changes in what we know. And checking the news for the discovery of an Earth-like planet provides a metric for whether a phase transition in our knowledge has actually occurred.

  After running such an analysis ten thousand times (in order to make sure our calculations were robust), we found that there was a two-thirds chance that an Earth-like planet would be discovered by the end of 2013. But even more exciting, there was a 50 percent chance of such a discovery happening by early to mid-2011!

  Soon after our paper was accepted, on September 1, 2010, we published it online. Four weeks later, on September 29, 2010, a team from UC Santa Cruz and the Carnegie Institution for Science announced the discovery of the first planet truly similar to Earth—one that could actually sustain life—called Gliese 581g.

  This planet has some curious properties: For example, one side always faces its star and one side always faces toward the night sky. Due to this, only in the region of the planet in permanent dusk is the temperature potentially habitable. And barely; it rates a balmy Moscow winter. But in addition to having a dimmer red sun than our own (much like the sun of Superman’s home planet of Krypton), it has a most intriguing property: It might not exist.

  It turns out that there is debate over whether its signature is authentic or simply a whole lot of really exciting noise. When another team examined a subset of the data, they found no evidence of Gliese 581g. Now it might just be such a subtle signal that it requires a lot of data to detect its presence. It might also not be there. However, since then, a somewhat potentially Earth-like planet, Kepler 22b,6 has been discovered orbiting a star about six hundred light-years away.

  I have heard an astronomer involved in exoplanet discovery mention that nothing would make him happier than to become a biologist once the target planet has been discovered. And it seems that we now have crossed a certain threshold in our scientific knowledge. Where there were once only hints of such a planet, gleaned from bits of data and inferences, we now seem to be at the dawn of a new type of awareness. And this phase transition occurred due to the regular underlying variation in habitability, which was deduced from discoveries of the masses and temperatures of planets.

  • • •

  SOMETIMES, though, it’s not that easy to determine the underlying change and see when a phase transition will happen. Even so, there are still mathematical regularities to how sudden changes in our knowledge occur, at least in the aggregate. One of these areas is in the proof of mathematical concepts.

  Back at the end of 2010 I attempted to use mathematical modeling to determine when a famous unsolved problem in mathematics would be proved. This was a lot harder than the planetary discovery area. There wasn’t some fundamental underlying property that was slowly changing, so that when all the individual increments were added up we would get something qualitatively new, like a planet that is potentially like Earth. Instead, when it comes to proofs, it often seems that it’s just a lot of mathematicians being wrong, year after year, with little hope of finding the correct solution.

  In fact, it’s not quite like that. Even in failure there can be success when trying to solve something, although it’s not exactly what we might have hoped for. For example, Fermat’s Last Theorem was a famously long-unsolved problem. This idea was created by the Frenchman Pierre de Fermat, a lawyer by profession and mathematician by hobby, in 1637. Fermat wrote that there are no three positive numbers a, b, or c that can satisfy the equation an + bn = cn, if n is greater than 2. If n is 2, we get the Pythagorean Theorem, which has tons of solutions. But make n larger, and Fermat stated that no number would work in the equation. Fermat didn’t prove this. Instead, in one of the most maddening episodes in math history, he scribbled this idea in the margins of a book and wrote that he had a brilliant proof, but, alas, the margin was too small to contain it.

  We now think he might have been mistaken. But no one had found any numbers greater than 2 that fit the equation since he wrote this statement. So it was assumed to be true, but no one could prove it. This elegant problem in number theory had gone unproven since the seventeenth century, until Andrew Wiles completed a proof in 1995, using pages and pages of very complex math, which would most certainly not have fit in Fermat’s margin. But, crucially, along the way, mathematicians proved other, smaller, proofs in their quest to crack Fermat’s Last Theorem. When finally solved, whole new pieces of math were involved7 in the construction of the proof.

  I decided to tackle predicting the proof of one of the most famous unsolved problems, something known as P versus NP. It’s so tough and important that the Clay Mathematics Institute has offered a prize of $1 million for its solution. It essentially asks whether two classes of mathematical problems are equivalent. P problems are easy, and can be solved in a straightforward fashion by today’s computers, and sometimes even by pencil and paper. NP problems are problems in which we can check whether an answer is correct very easily, independent of whether they seem easy to solve. Specifically, some NP problems appear to be incredibly difficult, but vexingly, if I were to give you the right answer to any NP problem, you would know right away whether it was a correct solution.

  For example, finding the prime factors of a number is an NP problem. Given a large enough number (like something that’s about four hundred digits long), we currently think that we would have to use all the computational resources on the planet to try to factor it. And even then it would take far longer than the lifetime of the Earth to determine what two prime numbers were multiplied to get the four-hundred-digit number.

  But if I gave you the correct factors, you could simply multiply them together and see if it yielded the really big solution. The question that people have been trying to solve for the past few decades is whether NP problems—which are easy to check, such as factoring primes—are also easy to solve. Thus far, no one has found any shortcuts to solving many fiendishly difficult NP problems, and they seem to be separate from P problems, making us think that P does not equal NP. But we could just be missing some basic insight or mathematical trick that would make solving these problems much easier, and which would imply that P equals NP. For example, trying to see if a big number is divisible by three might be hard, unless you know the trick: If the sum of a number’s digits is divisible by three, then so is the number itself.

  Assuming that NP problems are very hard is actually the basic assumption behind modern cryptography. Codes are only strong so long as P emphatically does not equal NP. And many other things remain hard to solve if P is not equal to NP, from handling resources and logistics in large organizations, to managing complex financial portfolios, to dealing with games and puzzles such as crosswords and Minesweeper. But we don’t know if they’re equal or not. And we haven’t known for over forty years.

  When will we know? Possibly never. There are some prob
lems8 that can never be solved, and in fact have been proven unsolvable. But there are many problems that have gone unsolved for hundreds of years, and then one day a mathematician puts the finishing touches on the proof. Therein lies the key for tackling this problem. I realized that I could look at the distribution of the amount of time it has taken for lots and lots of well-known unsolved problems to be solved eventually and compare that to the amount of time that P versus NP has been an outstanding problem. If P versus NP is not special but is similar to many other famous problems that have been hard to decipher, perhaps then I could gain some insight into looking statistically at how long it takes for other problems to eventually be solved.

  So while I can’t know exactly when any individual problem will be solved in advance, I can look at the aggregate of hard problems and see if there are any regularities. In doing this we can begin to put some bounds on our uncertainty about solving hard problems.

  It turns out that if we do this, we get some very interesting distributions, known as heavy-tailed distributions. This means that there is no single amount of time we can expect to wait for a solution: Some famous problems go decades before being solved, and some, those that exist far out in the tail of the distribution, remain outstanding for hundreds of years. There was even a famous conjecture (or unproven statement) in the data set that took more than fifteen hundred years before it was eventually proven.

  But using this distribution, we can see the number of years it takes for half of all of these problems to be solved, taking that 50 percent mark and using it as a likely timeframe for its solution. So given that it is soluble, it turns out we will have to wait until 2024,9 when P versus NP turns fifty-three, for a solution, assuming it behaves like other long-unsolved problems.

  There’s still a lot of uncertainty in this year. But through probability, we can now understand how rapid changes in mathematical knowledge can occur, at least in the aggregate. Given that we eventually do discover whether P equals NP, we can be certain that this change in knowledge will occur rapidly, descending upon the greater world without much warning.