by Seth Lloyd
For example, if you look through a microscope at tiny dust particles suspended in a liquid, you see them doing a jiggly dance called Brownian motion. This jiggling is caused by the dust particles being bombarded from all directions by molecules of the liquid in which the dust is suspended. When, by chance, more molecules hit the dust from the left than from the right, the dust particle recoils to the right. When more molecules hit the dust from the top, it recoils to the bottom. At the beginning of the twentieth century, Einstein provided an elegant quantitative theory of Brownian motion, showing that the observed motion was consistent with bombardment of the suspended dust particles by much smaller particles, of a particular size and mass. The atomic hypothesis was back.
Prior to Einstein’s work, the atomic hypothesis had been used to provide a rigorous basis for the behavior of heat and energy. Heat had long been known to be a form of energy. In the eighteenth century, James Watt performed a famous demonstration in which he immersed in water a cannon that was being bored by a horse-powered drill. The horses went round and round in a circle, turning a drill that removed metal to form the cannon’s bore. The water eventually boiled, showing the conversion of horsepower into heat. The quantitative trade-off between mechanical energy and heat was even more firmly established by the mid-nineteenth century and enshrined as the first law of thermodynamics: energy is conserved when mechanical energy is converted to heat. Unlike mechanical energy, however, energy in the form of heat seemed to possess the mysterious property called entropy, which prevented some of the heat from being transformed into useful work. Like energy, entropy could be quantified experimentally: Whenever mechanical energy was turned into heat, an amount of entropy equal to the energy divided by the temperature was created. When heat was turned into mechanical energy, as in one of Watt’s steam engines, the amount of entropy in the cooler steam of the engine’s exhaust was discovered to be either greater than or equal to the amount of entropy in the hot steam driving the engine. Entropy, whatever it was, never decreased.
Just what is this entropy stuff, anyway? The atomic hypothesis provides an answer. Heat is a form of energy, and entropy is associated with heat. If things are made out of atoms, then there is a simple explanation of heat: heat is just the energy in the jiggling of atoms. Entropy, then, has a simple interpretation, too: To describe the motion of atoms requires a large number of bits of information. The quantity called entropy is proportional to the number of bits required to describe the way atoms are jiggling.
For the scientists of the nineteenth century, it wasn’t a stretch of the imagination to think of heat as the energy in jiggling atoms. After all, since the work of Galileo and Newton hundreds of years earlier, it had been known that everything that moves has energy—called kinetic energy, from the Greek kinesis, “motion”—associated with that motion. The faster a thing moves, the more kinetic energy it has. When mechanical energy is converted to heat, as when horses bore a cannon and heat water, the mechanical energy generated by the horses is converted into the kinetic energy of the molecules of water. Similarly, when hot gas moves a piston in a steam engine, it is because the water molecules that form the steam are exerting pressure on the piston head as they bounce off it. When mechanical energy is converted into the kinetic energy of atoms and molecules, and vice versa, the first law of thermodynamics guarantees that the overall energy is conserved. It was not so natural, though, for nineteenth-century scientists to conceive of entropy as information. Nowadays, in the midst of the information-processing revolution, it is no longer surprising that information should be just as fundamental a quantity as energy, but at the end of the nineteenth century, it was not even clear that information was a quantity at all.
In the middle of the nineteenth century, James Clerk Maxwell developed a detailed theory of heat in terms of the motion of atoms. He figured out how fast the atoms were moving as a function of temperature: the kinetic energy of an atom is proportional to its temperature. The hotter something is, the faster its atoms are jiggling around.
This jiggling is also associated with entropy: the faster the atoms jiggle, the more information is required to describe their jiggling, and thus, the more entropy they possess. Temperature is a measure of the trade-off between information and energy: atoms at a high temperature require more energy to register a bit of information, and atoms at a low temperature require less energy to register a bit. Temperature is energy per bit. When energy in the form of heat flows from a hot thing to a cold thing, entropy increases: the same amount of energy registers less information when it’s hot than when it’s cold. The state of maximum entropy is obtained when everything is at the same temperature.
Maxwell realized that if one could gain information about the microscopic behavior of the atoms in a gas, one could reduce its entropy: entropy was somehow associated with information. In a famous letter, “On the Decrease of Entropy by Intelligent Beings,” Maxwell imagined a tiny intelligent being, or “demon,” who could make heat flow from cold bodies to hot, thereby apparently violating the second law of thermodynamics.
Figure 5. Maxwell’s Demon
Maxwell’s demon is an imaginary being that shunts fast, hot molecules to one side of a container and slow, cold molecules to the other side in apparent violation of the second law of thermodynamics.
Suppose a container of helium gas is divided in half by a partition. In the partition is a small door, just big enough for a few gas atoms to pass through at a time. The demon monitors the atoms in the neighborhood of the door and opens the door whenever the atoms that are approaching the door from the colder side are jiggling faster (thus are hotter) than the atoms approaching it from the hotter side. Each time the demon opens the door, hotter atoms move to the hot side, and cooler atoms move to the cold side. As the demon sorts more and more fast-jiggling atoms into the hot side and more and more slow-jiggling atoms into the cold side, the hot side of the gas grows hotter and the cold side grows colder. This demon-mediated flow of heat from the cold side to the hot side apparently violates the second law of thermodynamics, which implies that heat flows from hot to cold but not from cold to hot. It is the demon’s ability to get information about the atoms that allows him to accomplish this apparent violation of physical law.
As we will see, the demon cannot actually violate the second law, which states that the total entropy/information of the gas and the demon combined can’t decrease. The second law of thermodynamics remains intact. But still, a clear connection between information and entropy is illustrated by his demonic feat.
As the nineteenth century wound on, Boltzmann, Gibbs, and the German physicist Max Planck refined their formulas describing the energy and entropy of systems made up of atoms. In particular, they discovered that the entropy of a system was proportional to the number of bits required to describe the microscopic state of the atoms. This result was so useful in describing the trade-offs between heat and energy that the formula that encompasses it is inscribed on Boltzmann’s tomb. Entropy is traditionally written S, and the number of different possible microscopic states (or “complexions,” as Planck called them) is W. W could be the number of complexions of an individual atom or of a system made of atoms. The epitaph on Boltzmann’s tomb reads “S = k log W,” which is just a fancy way of saying that the entropy of something is proportional to the number of bits registered by its microscopic state. Another way of saying the same thing is that the entropy is proportional to the length, in bits, of the number of the microscopic states. In this formula, k is known as Boltzmann’s constant.
Paul and Tatyana Ehrenfest, who made many original contributions to the study of entropy, point out that this formula was actually first written down by Planck, and they therefore refer to the constant that we call Boltzmann’s constant as “Planck’s constant.” But, as we’ll see when we look at quantum mechanics, Planck already had a rather important constant named after him, so in order to avoid confusion, as well as to honor Boltzmann, k was renamed for him. Boltzmann was fa
mously moody, and died by his own hand shortly after a visit to the United States in 1906. One wonders what he would have made of another man’s formula being inscribed on his tomb.
Maxwell, Boltzmann, Gibbs, and Planck discovered that entropy is proportional to the number of bits of information registered by the microscopic motions of atoms. Of course, these nineteenth-century scientists did not think of their discovery as being primarily about information. At the time, however, entropy was not measured in bits, so they considered their discovery to be the proper expression for thermodynamic entropy—the quantity that limits the efficiency of heat engines. They were correct. And since entropy was not measured in bits at that time, the measurement had to be multiplied by Boltzmann’s constant to relate entropy as measured in terms of information to ordinary thermodynamic entropy. But whether they knew it or not, the pioneers of statistical mechanics discovered the formula for information fifty years before the mathematical theory of information was in place.
Just how does a physical system, such as a gas, register information? Consider a child’s balloon filled with helium. The helium atoms in the balloon are zipping around from place to place inside the balloon, bouncing off each other and off the walls of the balloon. Each helium atom registers information: the amount of information required to describe where it is (position) and where and how fast it is going (velocity). In order to measure the amount of information an atom registers, you must define the smallest scale (degree of precision) to which an atom’s position and velocity can be described. Then the number of bits a given atom registers is equal to the number of bits required to specify its position and velocity to a precision given by that smallest scale. Later, we’ll see that quantum mechanics defines the smallest scale to which position and velocity can be specified. Based on that scale, we can determine that each atom in the balloon registers about 20 bits. The amount of information registered by all the helium atoms in the balloon, then, is the number of atoms—6 × 1023—times the number of bits per atom: a million billion billion (1025) bits of information.
That’s a lot of information. This book contains a few million bits of information. The millions of books in the Library of Congress contain some million millions of bits. All the computers in the world at present contain some billion billion bits. All the bits of information generated in written or electronic form by the human species as a whole still falls short of the amount of information registered by the atoms of helium in a balloon.
Mind you, the bits of information registered by the helium atoms in a balloon wouldn’t make a very good read. Like the texts written by a monkey on a typewriter, the bits registered by the atoms would have a very high probability of looking like a bunch of random gibberish. Even if the positions and velocities of the helium atoms happened to spell out the whole of Hamlet at one instant in time (and we already know how unlikely that is), a second later their bits would bounce back into randomness.
Landauer’s Principle
The second law of thermodynamics holds that the total amount of information never decreases. In our balloon, for example, the second law implies that the number of bits of information registered by the helium atoms doesn’t get smaller if the balloon is left alone. If you cool the balloon, or squeeze it, or pop it, the number of bits registered by the atoms in the balloon can indeed decrease—but only at the expense of increasing the number of bits registered by atoms outside the balloon.
Information can be created but it can’t be destroyed. Consider flipping a bit. Flipping a bit transforms information: 0 goes to 1 and vice versa. It also preserves information: if you knew that the bit was 0 before the flip, then you know that it is 1 after the flip.
By contrast, erasure is a process that destroys information. During erasure, a bit that is initially 0 stays 0, and a bit that is initially 1 goes to 0. Erasure destroys the information in the bit. But the laws of physics do not allow processes that do nothing but erase a bit. Any process that erases a bit in one place must transfer that same amount of information somewhere else. This is known as Landauer’s principle, after Rolf Landauer, the pioneer of the physics of information, who discovered it in the early 1960s.
To see Landauer’s principle in action, look at how bits are erased in computers. As noted in chapter 2, in a contemporary electronic computer a bit is stored on a capacitor. A capacitor is a bucket for electrons. When you charge up the capacitor, you put electrons in the bucket; when you discharge it, you dump the electrons out of the bucket. In a computer, an uncharged capacitor registers a 0 and a charged capacitor registers a 1.
To erase a bit in an electronic computer, just empty the bucket: close a switch and let the electrons on the capacitor flow out. When the capacitor has been discharged, the bucket is empty: the bit has been restored to 0. But the microscopic state of the electrons “remembers” whether the capacitor was charged or not; that is, as they flow out of the capacitor, they heat up. This temperature change remains as evidence of the initial state of the capacitor. Its bit of information has been transferred to the microscopic motion of the electrons.
Another way to erase a bit is to swap it with another bit that reads 0. Swapping information between bits preserves information; to get back the original values of the bits, just swap them again. Before the swap, the first bit could read either 0 or 1; it has a bit’s worth of entropy. The second bit reads 0; it has no entropy. After the swap, the first bit reads 0; it has been restored to 0, or erased. The second bit reads 0 or 1; it has a bit’s worth of entropy—the same entropy that the first bit had before the swap. Swapping moves information and entropy from one place to another, but the overall amount of information remains constant. Swapping can be used to erase a bit in one register while retaining a copy of the bit in another register. To return to the example of a computer’s capacitor, discharging, or erasing its bit, essentially “swaps” the information registered by the capacitor with the information registered by the electrons.
The laws of physics preserve information as it is transformed. In mathematical parlance, the dynamical laws of physics of a closed physical system are one-to-one: Each input state goes to one and only one output state, and each output state can have come from one and only one input state. Thus, you can work backwards: if you know the physical state of a system now, then in principle you can follow its physical dynamics to determine the state of the system at earlier or later times.
So, for example, if you knew the exact state of the gas of helium atoms in the balloon at one point in time and were able to follow the detailed dynamics of the atoms bouncing off one another and off the inner wall of the balloon, then—because each state evolves dynamically into a completely determined state—you would know the state of the helium atoms at later times as well. Conversely, because each state evolves from a completely determined state, knowledge of the state now, together with the ability to follow the detailed dynamics, determines the state at previous times. In the case of flipping a bit, if you know what the state of the bit is before the flip, then you still know it afterward. Physical dynamics preserve information.
This preservation is what prevents heat engines like steam engines or automobile engines from extracting all the energy from heat. There’s lots of energy in hot gas, but lots of bits as well. The temperature of the gas is proportional to the average energy per bit. Hot gas has more energy per bit; cold gas has less energy per bit. When the heat energy of the gas is extracted—for example, by having the gas push against a piston—the bits are left behind. The moving piston turns heat energy into mechanical energy, the energy per atom (and hence per bit), decreases, and the expanding gas cools down. As long as the temperature of the gas does not go to absolute zero, each atom (and hence each of its bits) still requires some energy, so that amount of energy must remain in the gas, rather than becoming mechanical energy. Since some energy must remain behind, not all of the energy can be extracted in the form of work.
Over the centuries, many ingenious inventors have propos
ed machines that promise to extract more energy than should be possible by this rationale. They’ve attempted to defy the second law of thermodynamics. Such a machine is traditionally called a perpetuum mobile, a perpetual motion machine.6 As you might guess, these machines do not work because they fail to provide for the extra information. You would think that after centuries of failed effort, people would have given up on perpetual motion machines. Over the last fifteen years, however, I have been the referee for a number of ingenious proposals that try to extract more energy from physical systems than is allowed by the second law. They all fail. With practice, one can look at the most complicated of these schemes and see just where the inventor has swept the information under the rug.
The Spread of Ignorance
The laws of physics preserve information. The number of bits registered by a system (such as the helium-filled balloon) does not decrease. This information-preserving feature puts limits on the efficiency of heat engines and is responsible for the second law of thermodynamics. But now there’s a problem. According to the laws of physics, total information cannot increase either. In fact, the laws of physics say that in the absence of interaction with another system, the amount of information in a system stays the same. So, how can entropy—a form of information—increase without increasing the overall information content of a physical system? How does known information become unknown?
As originally conceived, entropy is a quantity that measures how useful energy is. Energy with a small amount of entropy is useful (free) energy; energy with lots of entropy is useless. It is perhaps easier to conceive of an increase in entropy in these terms: energy degrading from useful to useless forms. Hot baths grow cold. Cars run out of gas. Milk goes sour. So how can we think of this process in terms of information? The answer lies in a fundamental fact of nature that I call “the spread of ignorance.” Unknown bits infect known bits.