What Just Happened: A Chronicle From the Information Frontier

by James Gleick

  In the social sciences, the direct influence of information theorists had passed its peak. The specialized mathematics had less and less to contribute to psychology and more and more to computer science. But their contributions had been real. They had catalyzed the social sciences and prepared them for the new age under way. The work had begun; the informational turn could not be undone.

  * * *

  ♦ As Jean-Pierre Dupuy remarks: “It was, at bottom, a perfectly ordinary situation, in which scientists blamed nonscientists for taking them at their word. Having planted the idea in the public mind that thinking machines were just around the corner, the cyberneticians hastened to dissociate themselves from anyone gullible enough to believe such a thing.”

  9 | ENTROPY AND ITS DEMONS

  (You Cannot Stir Things Apart)

  Thought interferes with the probability of events, and, in the long run therefore, with entropy.

  —David L. Watson (1930)♦

  IT WOULD BE AN EXAGGERATION TO SAY that no one knew what entropy meant. Still, it was one of those words. The rumor at Bell Labs was that Shannon had gotten it from John von Neumann, who advised him he would win every argument because no one would understand it.♦ Untrue, but plausible. The word began by meaning the opposite of itself. It remains excruciatingly difficult to define. The Oxford English Dictionary, uncharacteristically, punts:

  1. The name given to one of the quantitative elements which determine the thermodynamic condition of a portion of matter.

  Rudolf Clausius coined the word in 1865, in the course of creating a science of thermodynamics. He needed to name a certain quantity that he had discovered—a quantity related to energy, but not energy.

  Thermodynamics arose hand in hand with steam engines; it was at first nothing more than “the theoretical study of the steam engine.”♦ It concerned itself with the conversion of heat, or energy, into work. As this occurs—heat drives an engine—Clausius observed that the heat does not actually get lost; it merely passes from a hotter body into a cooler body. On its way, it accomplishes something. This is like a waterwheel, as Nicolas Sadi Carnot kept pointing out in France: water begins at the top and ends at the bottom, and no water is gained or lost, but the water performs work on the way down. Carnot imagined heat as just such a substance. The ability of a thermodynamic system to produce work depends not on the heat itself, but on the contrast between hot and cold. A hot stone plunged into cold water can generate work—for example, by creating steam that drives a turbine—but the total heat in the system (stone plus water) remains constant. Eventually, the stone and the water reach the same temperature. No matter how much energy a closed system contains, when everything is the same temperature, no work can be done.

  It is the unavailability of this energy—its uselessness for work—that Clausius wanted to measure. He came up with the word entropy, formed from Greek to mean “transformation content.” His English counterparts immediately saw the point but decided Clausius had it backward in focusing on the negative. James Clerk Maxwell suggested in his Theory of Heat that it would be “more convenient” to make entropy mean the opposite: “the part which can be converted into mechanical work.” Thus:

  When the pressure and temperature of the system have become uniform the entropy is exhausted.

  Within a few years, though, Maxwell turned about-face and decided to follow Clausius.♦ He rewrote his book and added an abashed footnote:

  In former editions of this book the meaning of the term Entropy, as introduced by Clausius, was erroneously stated to be that part of the energy which cannot be converted into work. The book then proceeded to use the term as equivalent to the available energy; thus introducing great confusion into the language of thermodynamics. In this edition I have endeavoured to use the word Entropy according to its original definition by Clausius.

  The problem was not just in choosing between positive and negative. It was subtler than that. Maxwell had first considered entropy as a subtype of energy: the energy available for work. On reconsideration, he recognized that thermodynamics needed an entirely different measure. Entropy was not a kind of energy or an amount of energy; it was, as Clausius had said, the unavailability of energy. Abstract though this was, it turned out to be a quantity as measurable as temperature, volume, or pressure.

  It became a totemic concept. With entropy, the “laws” of thermodynamics could be neatly expressed:

  First law: The energy of the universe is constant.

  Second law: The entropy of the universe always increases.

  There are many other formulations of these laws, from the mathematical to the whimsical, e.g., “1. You can’t win; 2. You can’t break even either.”♦ But this is the cosmic, fateful one. The universe is running down. It is a degenerative one-way street. The final state of maximum entropy is our destiny.

  William Thomson, Lord Kelvin, imprinted the second law on the popular imagination by reveling in its bleakness: “Although mechanical energy is indestructible,” he declared in 1862, “there is a universal tendency to its dissipation, which produces gradual augmentation and diffusion of heat, cessation of motion, and exhaustion of potential energy through the material universe. The result of this would be a state of universal rest and death.”♦ Thus entropy dictated the universe’s fate in H. G. Wells’s novel The Time Machine: the life ebbing away, the dying sun, the “abominable desolation that hung over the world.” Heat death is not cold; it is lukewarm and dull. Freud thought he saw something useful there in 1918, though he muddled it: “In considering the conversion of psychical energy no less than of physical, we must make use of the concept of an entropy, which opposes the undoing of what has already occurred.”♦

  Thomson liked the word dissipation for this. Energy is not lost, but it dissipates. Dissipated energy is present but useless. It was Maxwell, though, who began to focus on the confusion itself—the disorder—as entropy’s essential quality. Disorder seemed strangely unphysical. It implied that a piece of the equation must be something like knowledge, or intelligence, or judgment. “The idea of dissipation of energy depends on the extent of our knowledge,” Maxwell said. “Available energy is energy which we can direct into any desired channel. Dissipated energy is energy which we cannot lay hold of and direct at pleasure, such as the energy of the confused agitation of molecules which we call heat.” What we can do, or know, became part of the definition. It seemed impossible to talk about order and disorder without involving an agent or an observer—without talking about the mind:

  Confusion, like the correlative term order, is not a property of material things in themselves, but only in relation to the mind which perceives them. A memorandum-book does not, provided it is neatly written, appear confused to an illiterate person, or to the owner who understands it thoroughly, but to any other person able to read it appears to be inextricably confused. Similarly the notion of dissipated energy could not occur to a being who could not turn any of the energies of nature to his own account, or to one who could trace the motion of every molecule and seize it at the right moment.♦

  Order is subjective—in the eye of the beholder. Order and confusion are not the sorts of things a mathematician would try to define or measure. Or are they? If disorder corresponded to entropy, maybe it was ready for scientific treatment after all.

  As an ideal case, the pioneers of thermodynamics considered a box of gas. Being made of atoms, it is far from simple or calm. It is a vast ensemble of agitating particles. Atoms were unseen and hypothetical, but these theorists—Clausius, Kelvin, Maxwell, Ludwig Boltzmann, Willard Gibbs—accepted the atomic nature of a fluid and tried to work out the consequences: mixing, violence, continuous motion. This motion constitutes heat, they now understood. Heat is no substance, no fluid, no “phlogiston”—just the motion of molecules.

  Individually the molecules must be obeying Newton’s laws—every action, every collision, measurable and calculable, in theory. But there were too many to measure and calculate individually
. Probability entered the picture. The new science of statistical mechanics made a bridge between the microscopic details and the macroscopic behavior. Suppose the box of gas is divided by a diaphragm. The gas on side A is hotter than the gas on side B—that is, the A molecules are moving faster, with greater energy. As soon as the divider is removed, the molecules begin to mix; the fast collide with the slow; energy is exchanged; and after some time the gas reaches a uniform temperature. The mystery is this: Why can the process not be reversed? In Newton’s equations of motion, time can have a plus sign or a minus sign; the mathematics works either way. In the real world past and future cannot be interchanged so easily.

  “Time flows on, never comes back,” said Léon Brillouin in 1949. “When the physicist is confronted with this fact he is greatly disturbed.”♦ Maxwell had been mildly disturbed. He wrote to Lord Rayleigh:

  If this world is a purely dynamical system, and if you accurately reverse the motion of every particle of it at the same instant, then all things will happen backwards to the beginning of things, the raindrops will collect themselves from the ground and fly up to the clouds, etc, etc, and men will see their friends passing from the grave to the cradle till we ourselves become the reverse of born, whatever that is.

  His point was that in the microscopic details, if we watch the motions of individual molecules, their behavior is the same forward and backward in time. We can run the film backward. But pan out, watch the box of gas as an ensemble, and statistically the mixing process becomes a one-way street. We can watch the fluid for all eternity, and it will never divide itself into hot molecules on one side and cool on the other. The clever young Thomasina says in Tom Stoppard’s Arcadia, “You cannot stir things apart,” and this is precisely the same as “Time flows on, never comes back.” Such processes run in one direction only. Probability is the reason. What is remarkable—physicists took a long time to accept it—is that every irreversible process must be explained the same way. Time itself depends on chance, or “the accidents of life,” as Richard Feynman liked to say: “Well, you see that all there is to it is that the irreversibility is caused by the general accidents of life.”♦ For the box of gas to come unmixed is not physically impossible; it is just improbable in the extreme. So the second law is merely probabilistic. Statistically, everything tends toward maximum entropy.

  Yet probability is enough: enough for the second law to stand as a pillar of science. As Maxwell put it:

  Moral. The 2nd law of Thermodynamics has the same degree of truth as the statement that if you throw a tumblerful of water into the sea, you cannot get the same tumblerful of water out again.♦

  The improbability of heat passing from a colder to a warmer body (without help from elsewhere) is identical to the improbability of order arranging itself from disorder (without help from elsewhere). Both, fundamentally, are due only to statistics. Counting all the possible ways a system can be arranged, the disorderly ones far outnumber the orderly ones. There are many arrangements, or “states,” in which molecules are all jumbled, and few in which they are neatly sorted. The orderly states have low probability and low entropy. For impressive degrees of orderliness, the probabilities may be very low. Alan Turing once whimsically proposed a number N, defined as “the odds against a piece of chalk leaping across the room and writing a line of Shakespeare on the board.”♦

  Eventually physicists began speaking of microstates and macrostates. A macrostate might be: all the gas in the top half of the box. The corresponding microstates would be all the possible arrangements of all particles—positions and velocities. Entropy thus became a physical equivalent of probability: the entropy of a given macrostate is the logarithm of the number of its possible microstates. The second law, then, is the tendency of the universe to flow from less likely (orderly) to more likely (disorderly) macrostates.

  It was still puzzling, though, to hang so much of physics on a matter of mere probability. Can it be right to say that nothing in physics is stopping a gas from dividing itself into hot and cold—that it is only a matter of chance and statistics? Maxwell illustrated this conundrum with a thought experiment. Imagine, he suggested, “a finite being” who stands watch over a tiny hole in the diaphragm dividing the box of gas. This creature can see molecules coming, can tell whether they are fast or slow, and can choose whether or not to let them pass. Thus he could tilt the odds. By sorting fast from slow, he could make side A hotter and side B colder—“and yet no work has been done, only the intelligence of a very observant and neat-fingered being has been employed.”♦ The being defies ordinary probabilities. The chances are, things get mixed together. To sort them out requires information.

  Thomson loved this idea. He dubbed the notional creature a demon:

  (Illustration credit 9.1)

  “Maxwell’s intelligent demon,” “Maxwell’s sorting demon,” and soon just “Maxwell’s demon.” Thomson waxed eloquent about the little fellow: “He differs from real living animals only [only!] in extreme smallness and agility.”♦ Lecturing to an evening crowd at the Royal Institution of Great Britain, with the help of tubes of liquid dyed two different colors, Thomson demonstrated the apparently irreversible process of diffusion and declared that only the demon can counteract it:

  He can cause one-half of a closed jar of air, or of a bar of iron, to become glowingly hot and the other ice cold; can direct the energy of the moving molecules of a basin of water to throw the water up to a height and leave it there proportionately cooled; can “sort” the molecules in a solution of salt or in a mixture of two gases, so as to reverse the natural process of diffusion, and produce concentration of the solution in one portion of the water, leaving pure water in the remainder of the space occupied; or, in the other case, separate the gases into different parts of the containing vessel.

  The reporter for The Popular Science Monthly thought this was ridiculous. “All nature is supposed to be filled with infinite swarms of absurd little microscopic imps,” he sniffed. “When men like Maxwell, of Cambridge, and Thomson, of Glasgow, lend their sanction to such a crude hypothetical fancy as that of little devils knocking and kicking the atoms this way and that …, we may well ask, What next?”♦ He missed the point. Maxwell had not meant his demon to exist, except as a teaching device.

  The demon sees what we cannot—because we are so gross and slow—namely, that the second law is statistical, not mechanical. At the level of molecules, it is violated all the time, here and there, purely by chance. The demon replaces chance with purpose. It uses information to reduce entropy. Maxwell never imagined how popular his demon would become, nor how long-lived. Henry Adams, who wanted to work some version of entropy into his theory of history, wrote to his brother Brooks in 1903, “Clerk Maxwell’s demon who runs the second law of Thermo-dynamics ought to be made President.”♦ The demon presided over a gateway—at first, a magical gateway—from the world of physics to the world of information.

  (Illustration credit 9.2)

  (Illustration credit 9.3)

  (Illustration credit 9.4)

  (Illustration credit 9.5)

  Scientists envied the demon’s powers. It became a familiar character in cartoons enlivening physics journals. To be sure, the creature was a fantasy, but the atom itself had seemed fantastic, and the demon had helped tame it. Implacable as the laws of nature now seemed, the demon defied these laws. It was a burglar, picking the lock one molecule at a time. It had “infinitely subtile senses,” wrote Henri Poincaré, and “could turn back the course of the universe.”♦ Was this not just what humans dreamed of doing?

  Through their ever better microscopes, scientists of the early twentieth century examined the active, sorting processes of biological membranes. They discovered that living cells act as pumps, filters, and factories. Purposeful processes seemed to operate at tiny scales. Who or what was in control? Life itself seemed an organizing force. “Now we must not introduce demonology into science,” wrote the British biologist James Johnstone in 1914. I
n physics, he said, individual molecules must remain beyond our control. “These motions and paths are un-co-ordinated—‘helter-skelter’—if we like so to term them. Physics considers only the statistical mean velocities.” That is why the phenomena of physics are irreversible, “so that for the latter science Maxwell’s demons do not exist.” But what of life? What of physiology? The processes of terrestrial life are reversible, he argued. “We must therefore seek for evidence that the organism can control the, otherwise, un-co-ordinated motions of the individual molecules.”♦

  Is it not strange that while we see that most of our human effort is that of directing natural agencies and energies into paths which they would not otherwise take, we should yet have failed to think of primitive organisms, or even of the tissue elements in the bodies of the higher organisms, as possessing also the power of directing physico-chemical processes?

  When life remained so mysterious, maybe Maxwell’s demon was not just a cartoon.

  Then the demon began to haunt Leó Szilárd, a very young Hungarian physicist with a productive imagination who would later conceive the electron microscope and, not incidentally, the nuclear chain reaction. One of his more famous teachers, Albert Einstein, advised him out of avuncular protectiveness to take a paying job with the patent office, but Szilárd ignored the advice. He was thinking in the 1920s about how thermodynamics should deal with incessant molecular fluctuations. By definition, fluctuations ran counter to averages, like fish swimming momentarily upstream, and people naturally wondered: what if you could harness them? This irresistible idea led to a version of the perpetual motion machine, perpetuum mobile, holy grail of cranks and hucksters. It was another way of saying, “All that heat—why can’t we use it?”