by James Gleick
Quantifying predictability and redundancy in this way is a backward way of measuring information content. If a letter can be guessed from what comes before, it is redundant; to the extent that it is redundant, it provides no new information. If English is 75 percent redundant, then a thousand-letter message in English carries only 25 percent as much information as one thousand letters chosen at random. Paradoxical though it sounded, random messages carry more information. The implication was that natural-language text could be encoded more efficiently for transmission or storage.
Shannon demonstrated one way to do this, an algorithm that exploits differing probabilities of different symbols. And he delivered a stunning package of fundamental results. One was a formula for channel capacity, the absolute speed limit of any communication channel (now known simply as the Shannon limit). Another was the discovery that, within that limit, it must always be possible to devise schemes of error correction that will overcome any level of noise. The sender may have to devote more and more bits to correcting errors, making transmission slower and slower, but the message will ultimately get through. Shannon did not show how to design such schemes; he only proved that it was possible, thereby inspiring a future branch of computer science. “To make the chance of error as small as you wish? Nobody had thought of that,” his colleague Robert Fano recalled years later. “How he got that insight, how he came to believe such a thing, I don’t know. But almost all modern communication theory is based on that work.”♦ Whether removing redundancy to increase efficiency or adding redundancy to enable error correction, the encoding depends on knowledge of the language’s statistical structure to do the encoding. Information cannot be separated from probabilities. A bit, fundamentally, is always a coin toss.
If the two sides of a coin were one way of representing a bit, Shannon offered a more practical hardware example as well:
A device with two stable positions, such as a relay or a flip-flop circuit, can store one bit of information. N such devices can store N bits, since the total number of possible states is 2N and log22N = N.
Shannon had seen devices—arrays of relays, for example—that could store hundreds, even thousands of bits. That seemed like a great many. As he was finishing his write-up, he wandered one day into the office of a Bell Labs colleague, William Shockley, an Englishman in his thirties. Shockley belonged to a group of solid-state physicists working on alternatives to vacuum tubes for electronics, and sitting on his desk was a tiny prototype, a piece of semiconducting crystal. “It’s a solid-state amplifier,” Shockley told Shannon.♦ At that point it still needed a name.
One day in the summer of 1949, before the book version of The Mathematical Theory of Communication appeared, Shannon took a pencil and a piece of notebook paper, drew a line from top to bottom, and wrote the powers of ten from 100 to 1013. He labeled this axis “bits storage capacity.”♦ He began listing some items that might be said to “store” information. A digit wheel, of the kind used in a desktop adding machine—ten decimal digits—represents just over 3 bits. At just under 103 bits, he wrote “punched card (all config. allowed).” At 104 he put “page single spaced typing (32 possible symbols).” Near 105 he wrote something offbeat: “genetic constitution of man.” There was no real precedent for this in current scientific thinking. James D. Watson was a twenty-one-year-old student of zoology in Indiana; the discovery of the structure of DNA lay several years in the future. This was the first time anyone suggested the genome was an information store measurable in bits. Shannon’s guess was conservative, by at least four orders of magnitude. He thought a “phono record (128 levels)” held more information: about 300,000 bits. To the 10 million level he assigned a thick professional journal (Proceedings of the Institute of Radio Engineers) and to 1 billion the Encyclopaedia Britannica. He estimated one hour of broadcast television at 1011 bits and one hour of “technicolor movie” at more than a trillion. Finally, just under his pencil mark for 1014, 100 trillion bits, he put the largest information stockpile he could think of: the Library of Congress.
(Illustration credit 7.5)
* * *
♦ Toward the end of his life Gödel wrote, “It was only by Turing’s work that it became completely clear, that my proof is applicable to every formal system containing arithmetic.”
♦ “not considering statistical structure over greater distances than about eight letters.”
8 | THE INFORMATIONAL TURN
(The Basic Ingredient in Building a Mind)
It is probably dangerous to use this theory of information in fields for which it was not designed, but I think the danger will not keep people from using it.
—J. C. R. Licklider (1950)♦
MOST MATHEMATICAL THEORIES take shape slowly; Shannon’s information theory sprang forth like Athena, fully formed. Yet the little book of Shannon and Weaver drew scant public attention when it appeared in 1949. The first review came from a mathematician, Joseph L. Doob, who complained that it was more “suggestive” than mathematical—“and it is not always clear that the author’s mathematical intentions are honorable.”♦ A biology journal said, “At first glance, it might appear that this is primarily an engineering monograph with little or no application to human problems. Actually, the theory has some rather exciting implications.”♦ The Philosophical Review said it would be a mistake for philosophers to overlook this book: “Shannon develops a concept of information which, surprisingly enough, turns out to be an extension of the thermodynamic concept of entropy.”♦ The strangest review was barely a review at all: five paragraphs in Physics Today, September 1950, signed by Norbert Wiener, Massachusetts Institute of Technology.
Wiener began with a faintly patronizing anecdote:
Some fifteen years ago, a very bright young student came to the authorities at MIT with an idea for a theory of electric switching dependent on the algebra of logic. The student was Claude E. Shannon.
In the present book (Wiener continued), Shannon, along with Warren Weaver, “has summed up his views on communication engineering.”
The fundamental idea developed by Shannon, said Wiener, “is that of the amount of information as negative entropy.” He added that he himself—“the author of the present review”—had developed the same idea at about the same time.
Wiener declared the book to be work “whose origins were independent of my own work, but which has been bound from the beginning to my investigations by cross influences spreading in both directions.” He mentioned “those of us who have tried to pursue this analogy into the study of Maxwell’s demon” and added that much work remained to be done.
Then he suggested that the treatment of language was incomplete without greater emphasis on the human nervous system: “nervous reception and the transmission of language into the brain. I say these things not as a hostile criticism.”
Finally, Wiener concluded with a paragraph devoted to another new book: “my own Cybernetics.” Both books, he said, represent opening salvos in a field that promises to grow rapidly.
In my book, I have taken the privilege of an author to be more speculative, and to cover a wider range than Drs. Shannon and Weaver have chosen to do.… There is not only room, but a definite need for different books.
He saluted his colleagues for their well-worked and independent approach—to cybernetics.
Shannon, meanwhile, had already contributed a short review of Wiener’s book to the Proceedings of the Institute of Radio Engineers, offering praise that could be described as faint. It is “an excellent introduction,” he said.♦ There was a little tension between these men. It could be felt weighing down the long footnote that anchored the opening page of Weaver’s portion of The Mathematical Theory of Communication:
Dr. Shannon has himself emphasized that communication theory owes a great debt to Professor Norbert Wiener for most of its basic philosophy. Professor Wiener, on the other hand, points out that much of Shannon’s early work on switching and mathematical logic antedated his own i
nterest in this field; and generously adds that Shannon certainly deserves credit for independent development of such fundamental aspects of the theory as the introduction of entropic ideas.
Shannon’s colleague John Pierce wrote later: “Wiener’s head was full of his own work.… Competent people have told me that Wiener, under the misapprehension that he already knew what Shannon had done, never actually found out.”♦
Cybernetics was a coinage, future buzzword, proposed field of study, would-be philosophical movement entirely conceived by this brilliant and prickly thinker. The word he took from the Greek for steersman: κυβερνιτησ, kubernites, from which comes also (not coincidentally) the word governor.♦ He meant cybernetics to be a field that would synthesize the study of communication and control, also the study of human and machine. Norbert Wiener had first become known to the world as a curiosity: a sport, a prodigy, driven and promoted by his father, a professor at Harvard. “A lad who has been proudly termed by his friends the brightest boy in the world,” The New York Times reported on page 1 when he was fourteen years old, “will graduate next month from Tufts College.… Aside from the fact that Norbert Wiener’s capacity for learning is phenomenal, he is as other boys.… His intense black eyes are his most striking feature.”♦ When he wrote his memoirs, he always used the word prodigy in the titles: Ex-Prodigy: My Childhood and Youth and I Am a Mathematician: The Later Life of a Prodigy.
After Tufts (mathematics), Harvard graduate school (zoology), Cornell (philosophy), and Harvard again, Wiener left for Cambridge, England, where he studied symbolic logic and Principia Mathematica with Bertrand Russell himself. Russell was not entirely charmed. “An infant prodigy named Wiener, Ph.D. (Harvard), aged 18, turned up,” he wrote a friend. “The youth has been flattered, and thinks himself God Almighty—there is a perpetual contest between him and me as to which is to do the teaching.”♦ For his part, Wiener detested Russell: “He is an iceberg. His mind impresses one as a keen, cold, narrow logical machine, that cuts the universe into neat little packets, that measure, as it were, just three inches each way.”♦ On his return to the United States, Wiener joined the faculty of MIT in 1919, the same year as Vannevar Bush. When Shannon got there in 1936, he took one of Wiener’s mathematics courses. When war loomed, Wiener was one of the first to join the hidden, scattered teams of mathematicians working on antiaircraft fire control.
NORBERT WIENER (1956) (Illustration credit 8.1)
He was short and rotund, with heavy glasses and a Mephistophelian goatee. Where Shannon’s fire-control work drilled down to the signal amid the noise, Wiener stayed with the noise: swarming fluctuations in the radar receiver, unpredictable deviations in flight paths. The noise behaved statistically, he understood, like Brownian motion, the “extremely lively and wholly haphazard movement” that van Leeuwenhoek had observed through his microscope in the seventeenth century. Wiener had undertaken a thoroughgoing mathematical treatment of Brownian motion in the 1920s. The very discontinuity appealed to him—not just the particle trajectories but the mathematical functions, too, seemed to misbehave. This was, as he wrote, discrete chaos, a term that would not be well understood for several generations. On the fire-control project, where Shannon made a modest contribution to the Bell Labs team, Wiener and his colleague Julian Bigelow produced a legendary 120-page monograph, classified and known to the several dozen people allowed to see it as the Yellow Peril because of the color of its binder and the difficulty of its treatment. The formal title was Extrapolation, Interpolation, and Smoothing of Stationary Time Series. In it Wiener developed a statistical method for predicting the future from noisy, uncertain, and corrupted data about the past. It was too ambitious for the existing gun machinery, but he tested it on Vannevar Bush’s Differential Analyzer. Both the antiaircraft gun, with its operator, and the target airplane, with its pilot, were hybrids of machine and human. One had to predict the behavior of the other.
Wiener was as worldly as Shannon was reticent. He was well traveled and polyglot, ambitious and socially aware; he took science personally and passionately. His expression of the second law of thermodynamics, for example, was a cry of the heart:
We are swimming upstream against a great torrent of disorganization, which tends to reduce everything to the heat death of equilibrium and sameness.… This heat death in physics has a counterpart in the ethics of Kierkegaard, who pointed out that we live in a chaotic moral universe. In this, our main obligation is to establish arbitrary enclaves of order and system.… Like the Red Queen, we cannot stay where we are without running as fast as we can.♦
He was concerned for his place in intellectual history, and he aimed high. Cybernetics, he wrote in his memoirs, amounted to “a new interpretation of man, of man’s knowledge of the universe, and of society.”♦ Where Shannon saw himself as a mathematician and an engineer, Wiener considered himself foremost a philosopher, and from his fire-control work he drew philosophical lessons about purpose and behavior. If one defines behavior cleverly—“any change of an entity with respect to its surroundings”♦—then the word can apply to machines as well as animals. Behavior directed toward a goal is purposeful, and the purpose can sometimes be imputed to the machine rather than a human operator: for example, in the case of a target-seeking mechanism. “The term servomechanisms has been coined precisely to designate machines with an intrinsic purposeful behavior.” The key was control, or self-regulation.
To analyze it properly he borrowed an obscure term from electrical engineering: “feed-back,” the return of energy from a circuit’s output back to its input. When feedback is positive, as when the sound from loudspeakers is re-amplified through a microphone, it grows wildly out of control. But when feedback is negative—as in the original mechanical governor of steam engines, first analyzed by James Clerk Maxwell—it can guide a system toward equilibrium; it serves as an agent of stability. Feedback can be mechanical: the faster Maxwell’s governor spins, the wider its arms extend, and the wider its arms extend, the slower it must spin. Or it can be electrical. Either way, the key to the process is information. What governs the antiaircraft gun, for example, is information about the plane’s coordinates and about the previous position of the gun itself. Wiener’s friend Bigelow emphasized this: “that it was not some particular physical thing such as energy or length or voltage, but only information (conveyed by any means).”♦
Negative feedback must be ubiquitous, Wiener felt. He could see it at work in the coordination of eye and hand, guiding the nervous system of a person performing an action as ordinary as picking up a pencil. He focused especially on neurological disorders, maladies that disrupted physical coordination or language. He saw them quite specifically as cases of information feedback gone awry: varieties of ataxia, for example, where sense messages are either interrupted in the spinal cord or misinterpreted in the cerebellum. His analysis was detailed and mathematical, with equations—almost unheard of in neurology. Meanwhile, feedback control systems were creeping into factory assembly lines, because a mechanical system, too, can modify its own behavior. Feedback is the governor, the steersman.
So Cybernetics became the title of Wiener’s first book, published in the fall of 1948 in both the United States and France. Subtitle: Control and Communication in the Animal and the Machine. The book is a hodgepodge of notions and analysis, and, to the astonishment of its publishers, it became the year’s unexpected bestseller. The popular American news magazines, Time and Newsweek, both featured it. Wiener and cybernetics were identified with a phenomenon that was bursting into public consciousness just at that moment: computing machines. With the end of the war, a veil had been lifted from the first urgent projects in electronic calculation, particularly the ENIAC, a thirty-ton monster of vacuum tubes, relays, and hand-soldered wires stretching across eighty feet at the University of Pennsylvania’s electrical engineering school. It could store and multiply up to twenty numbers of ten decimal digits; the army used it to calculate artillery firing tables. The In
ternational Business Machines company, IBM, which provided punched card machines for the army projects, also built a giant calculating machine at Harvard, the Mark I. In Britain, still secret, the code breakers at Bletchley Park had gone on to build a vacuum-tube computing machine called the Colossus. Alan Turing was beginning work on another, at the University of Manchester. When the public learned about these machines, they were naturally thought of as “brains.” Everyone asked the same question: Can machines think?
“They are growing with fearful speed,” declared Time in its year-end issue. “They started by solving mathematical equations with flash-of-lightning rapidity. Now they are beginning to act like genuine mechanical brains.”♦ Wiener encouraged the speculation, if not the wild imagery:
Dr. Wiener sees no reason why they can’t learn from experience, like monstrous and precocious children racing through grammar school. One such mechanical brain, ripe with stored experience, might run a whole industry, replacing not only mechanics and clerks but many of the executives too.…
As men construct better calculating machines, explains Wiener, and as they explore their own brains, the two seem more & more alike. Man, he thinks, is recreating himself, monstrously magnified, in his own image.