The Kepler of Word Frequencies?
Why do I view that fateful metro ride as a Kepler moment? For Kepler, the role of toy had originally been played by the ellipse, an esoteric geometric curve with little known application. I dealt with an esoteric wrinkle in the study of language as it stood in 1950. That wrinkle—statistical thermodynamics—is one of the most sublime pillars of physics.
The key feature of the Zipf-Mandelbrot formula exponent was inherited from the statistical thermodynamics motivation: a “temperature of discourse.” It could measure differences from text to text, from speaker to speaker. It gave a numerical grade to the richness of someone’s vocabulary. Low temperature, limited vocabulary. High temperature, rich vocabulary. The original Zipf’s formula is a very close approximation—but misleading. Joyce’s Ulysses was welcomed by Zipf because it was long, but also because it was atypical. The temperature of discourse could become a powerful tool of social measurement by capturing erudition in a number.
So that long metro ride witnessed the first of many Kepler moments in my life. Soon after it, I examined Zipf’s book. His charts confirmed that the Zipf-Mandelbrot formula was a vast improvement. A difficulty: a well-defined probability may exist for common words, but what about rare words, especially in multiauthor works or composite files of newspaper articles? In due time, I identified many problems—which still remain open.
Those events taught me a fundamental lesson—that an applied mathematician’s relation to reality is fraught with problems. Worse, experimentalists try to help by simplifying what they see, and key facts are often unwittingly overlooked. They must be respected but never trusted without question.
A Case of Haste Rewarded?
Miraculously, the paper reprint of Walsh’s review of Zipf remained in my files and came to light as I was writing this memoir. I made a point of reading it again. It’s clear to me now that in my excitement I read it casually and rushed to work. Walsh’s review also contained these words I had either missed or forgotten:
It would be rash to prophesy that a new science of human behavior will now evolve along the lines of the history of mechanics, but it would be foolish to ignore the lessons of that history.… Tycho Brahe … made numerous observations of the motions of the planets [that were] used by … Kepler … to formulate … fundamental laws … and Newton … in turn [to found] the science of mechanics.… Opportunity is ripe for new Tycho Brahes, Keplers and Newtons!…
It might be fruitful to investigate speech as a natural phenomenon … a peculiar form of behavior … in the manner of the exact sciences.
Shame on me! I had forgotten that Walsh mentioned Kepler by name. My first Kepler moment concerned long tails, an uncanny fit to my wild dream. I had to be reminded of these words. Yet I recall being spellbound by the Keplerian possibilities and not bothered by the absence of geometry—which became central to my work.
Early on, a shadow was present—the example I worked on was devoid of important consequences. No one could predict that I was to be called the “Kepler of word frequencies,” then more generally the “father of long tails.” In a fifty-year time span, they went from an aberration hardly worth mentioning to the center of wide attention in the early 2000s. Had I approached it from a seemingly more “worthy” angle, I am convinced I would have failed. My luck was holding.
Questions rush in. Computer searches reveal that Zipf was reviewed favorably and—among the mathematically unsophisticated—had found a following. Had Walsh noticed that Zipf’s original formula was nonsense, he would have discouraged his friend. Why did that review fail to attract the attention of anyone else of at least adequate mathematical competence?
From Unruly Beginner to “Father of Long Tails”
I had in hand the key topic of my doctoral thesis: the very simple mathematics behind the unexpected distribution of word frequencies. On December 19, 1952, the die was cast. My Ph.D. dissertation loudly affirmed a Keplerian determination to become a solo scientist—the kind my world thought had vanished. Figuratively, I was choosing to be an apprentice-hermit at a time when science was rushing to adopt the ways of the more structured religious churches. Convinced that this direction could never be reversed, I stopped thinking of ever contributing to plain mathematics or physics. Well … I eventually did—very late in life and with a vengeance.
While carefully thought through, my dissertation was “technically” easy and imperfectly written. It barely matched my ambition—but I was in a rush and underestimated myself. Plain old-fashioned luck, and perhaps a learned skill of turning difficulties to assets, made me the first—and for a long time the only—mathematically competent person to face long tails squarely.
Determination but No Foresight
I simply loved being able to do everything all by myself. With my records at Normale and Carva still widely known, my thesis was approved when it was clear that nobody in Paris much cared about my topic or my career. Of course, my parents, Szolem, and many others pushed in conflicting ways. So—for better or worse—I did it in my own way.
I did it with determination but no foresight. How should I follow up? Szolem had warned me in no uncertain terms that—before rushing to Caltech—I should identify in Paris a suitable combination of a topic and an adviser who could protect me for a while. Otherwise, nobody would help me find a job. I was beginning to wonder about my chances for a proper academic career in any country.
Against the background of an early life of hard knocks of every kind, was I acting like a spoiled child? I had a free built-in insurance. Not only was I still a member of the CNRS, but the Ph.D. had earned me a promotion. More than a few of my contemporaries stayed at the CNRS, kept quiet, and pursued activities they carefully failed to report. So, no, I was not acting spoiled. I did not want to hide—I wanted to find the best conditions to fulfill my Keplerian dream. Dreams can be burdensome.
My political innocence would not be punished. In a few years, explosive growth overwhelmed the old French universities, and many new lifetime jobs had to be created. That Ph.D. certificate would come to matter greatly. But in 1952, this growth was but a distant hope. I knew that a hastily written doctoral thesis in French and devotion to a field that did not officially exist were unlikely to be enough. In any event, finding companions and putting down professional roots demanded some commitment.
Luckily, short-term jobs were plentiful—though mostly not in France. During the five years after my Ph.D., I sampled several thoroughly different ones, making my life extremely interesting and varied. But, like at Caltech, I didn’t accomplish much new. Eventually, inspiration did come from my Carva teachers Paul Lévy and Gaston Julia—but only through their work. Had I sought their advice, I am sure I would not have taken it.
15
Postdoctoral Grand Tour Begins at MIT, 1953
I RECALLED THOSE WORDS Gaudeamus igitur, juvenes dum sumus—“While we are young, let us rejoice.” In my case, rejoicing did not mean carousing. It first meant a highly unorthodox Ph.D. boldly asserting the kind of work I hoped to carry on. Later, it meant a modern form of a different medieval tradition: that of apprentice-scholars’ wandering years, which I think of as my “postdoctoral grand tour.” During that time, I worked near the two exalted living role models to whom my thesis was dedicated; mathematicians of the highest rank, they had repeatedly achieved the Keplerian dream I wanted to emulate.
The first was Norbert Wiener, a professor at MIT, the Massachusetts Institute of Technology, in Cambridge. He had authored an unusual book I greatly admired: Cybernetics, or Control and Communication in the Animal and the Machine. “Cybernetics” was a word Wiener had just coined, and the title defined that word as ranging from brains to telephone switchboards.
The second was John von Neumann, a professor at IAS, the Institute for Advanced Study, in Princeton. After MIT, I became von Neumann’s last postdoc there. He had written, with Oskar Morgenstern, Theory of Games and Economic Behavior. Both titles promised new frontiers and new topi
cs—or at least altogether new combinations of existing topics.
My Ph.D. dissertation’s title, Games of Communication, overstressed a bit my devotion to both men—whom I perceived as made of stardust. These two men were the only living proof that my Keplerian dream was not an idle one—that it was possible to put together and develop a new mathematical approach to a very old, very concrete problem that overlapped several disciplines. Matching the sterling quality of their accomplishments was far beyond my ambitions, and I couldn’t think of less exalted advisers.
Norbert Wiener of MIT
The towering Keplerian achievements of Norbert Wiener (1894–1964) were his mathematical theory of Brownian motion and cybernetics—the word and the book. Isaac Newton knew around 1700 that prisms decompose light into components of different colors. But the mathematical theory was given much later, by Wiener. A related achievement, his theory of Brownian motion, strongly affected me later in my life—as a miserable model of the variation of competitive prices, and as a wiggle with an interesting boundary that forms fractal islands. His own account of early motivations was thrilling. Having become interested in the motion of pollen as seen through a microscope, he decided that the solution must use something called Lebesgue integrals—at that time still novel and the epitome of an esoteric toy.
As Wiener’s follower, I never tried to further develop the technical problems he had raised. I preferred to either move sideways and open new technical problems or take conceptual new steps by going beyond the Brownian realm. Yet Wiener’s work has remained a shining beacon for me.
He was a mathematical genius—a widely celebrated establishment figure. He became the leader of a scientific avant-garde that he hoped would grow to cover communication and control in machines and living things. To denote this goal before it was even partially fulfilled, he drew on a Greek word to coin “cybernetics.” I heard this word early on. When Wiener was in Paris in 1947, Szolem invited him for lunch and asked me to join them for coffee.
He was a master in a field of mathematics very close to Szolem’s, and they had written joint papers. Szolem looked up to his barely older friend, but acknowledged that his and Wiener’s mathematics had been born in concrete contexts that had occurred centuries ago. Fresh inputs from science were, for Szolem, intolerable, and there was always an undercurrent of irritation.
To a mathematician, the term “function” often denotes something that varies in time. Wiener preferred to use “noise.” Szolem was bothered and wondered aloud. Was this a mannerism left over from consulting for the military, or a way of showing off? I argued that Wiener’s esoteric mathematics was genuine, part of a lifelong ambition to understand physical fluctuations. He wanted to “see over the fence” to engineering, biology, and social sciences—but not to narrowly defined economics.
Jerry Wiesner’s RLE, an Ideal Research Environment
Fired up by Norbert Wiener’s cybernetics … that unique scientific incubator, the Research Laboratory of Electronics (RLE), has for two decades provided an almost ideal research environment and has been a model for the structure of other research centers.… [Back in 1946,] We could hardly imagine the excitement and intellectual pleasure that lay ahead of us. In fact, as I look back, I have the impression of powerful personalities and even more powerful ideas drawing people together from all over the world. My memory is a great pleasant blur, not unlike my mental movie of the spontaneous creation of the universe.
These words were spoken by Jerry Wiesner at MIT during the twenty-fifth anniversary celebration of RLE, a most remarkable institution, where I went after my doctorate to continue my education. Jerome B. Wiesner was Professor Wiesner when I met him as head of RLE, later became Dr. Wiesner, and ended his career at MIT as the unforgettable President Wiesner. We knew him as Jerry.
In this quote, the words “from all over the world” are essential, and the word “spontaneous” toward the end is very important. All those who knew Jerry can testify that in his “mental movie” he did not view himself as a creator but as a facilitator. In fact, he was the rare manager who could make creation seem to occur spontaneously. RLE was a remarkable hybrid of solo scientists of the ancient academic tradition and the more modern group of academics inherited from MIT’s famed wartime Radiation Laboratory, where radar had been developed.
This was the heyday of RLE. Jerry was almost the complete opposite of Wiener, though the similar names (especially when enhanced by foreign accents) often led to—mostly innocuous—confusion.
For a big boss at the Radiation Laboratory during World War II, Jerry Wiesner had been astonishingly young. He was not himself an accomplished scientist, but was endowed in an unusual way: he had a keen eye for full personal commitment (a large part of scientific value), a highly developed sense of noblesse oblige, and the ability to interact with everyone and get things done. He simply knew how to run an organization without self-aggrandizement, with invisible bureaucracy, and with maximum respect for his charges—including many thoroughly spoiled brats. Next to him, I always felt like a child.
Early on, Jerry became close to Senator John F. Kennedy. After JFK ascended to the presidency, he took Jerry as his science adviser—more effective and visible than those who came before or after. Back at MIT, Jerry moved up by stages and became its president during a period when the New Left was riding high and the institute actually appeared endangered.
By chance, Jerry heard me lecture in London in 1952 and liked my talk and the fun discussion it provoked. You must believe it—I argued with the ethnologist Margaret Mead (1901–78), who had gained fame studying sex in the South Seas! That she attended my lecture illustrates the wonderful open mood of those distant years. As was his style, Jerry invited me over to MIT with practically no paperwork. He was then an associate professor and, like several other staff, sat in an open cubicle in a large room. This kept him close to the troops.
RLE was housed in the large and labyrinthine Building 20, a quickie wood, tar, and asbestos barracks that the Radiation Laboratory could perpetually adjust to changing needs. Like everything else, my chair was beat-up and shaky, but high-class—a sign affixed on its back read LEE DUBRIDGE, former big boss of Rad Lab and Caltech president in my student days. That chair’s survival testified that when freewheeling scientific research is properly managed, it is not a financial extravagance but a true bargain.
Northeast of what was Building 20, a neighborhood called East Cambridge is now filled with high-rise industrial labs and upscale housing—where I live. At that time, it was a low-rise mix of industries and tenements. Therefore, RLE was often filled with either the aroma of a traditional chocolate factory or the stench of a rendering plant that boiled carrion into pure white soap. I took all this as constant confirmation that the process of creation is intrinsically messy and suffers more from soulless order than from surrounding physical decay.
Controversial Balance Between Conjecture and Proof
Claude Shannon (1916–2001) was the intellectual leader whose wartime work, published in 1948, created information theory and provided RLE with an intellectual backbone. His work on noiseless channels was a point of departure for the theory of word frequencies presented in my Ph.D. thesis.
But far more impressive was his noisy channel theorem. Actually, it was not a theorem at all, only a brilliant conjecture—in a style that is controversial, and of which I eventually became a very active supplier. The point? Even an arbitrarily noisy channel may be programmed in such a way as to allow it to transmit messages with an accuracy as close to perfection as desired.
Shannon’s conjecture was plainly an important event, but his proofs were incomplete. Adding the years during which his work was classified, an increasingly clear and general proof was slow to come. The information theorists perceived this as a minor annoyance. But the mathematicians thumbed their noses, noting that Shannon’s noisy channel theorem was unproven.
On a later visit to MIT, I played a role in the first proof of this theorem when
Amiel Feinstein, a graduate student in physics, came to see me. He was seeking a new topic in electrical engineering that would promise a quick Ph.D. Momentarily irritated by his arrogance, I blurted out that he might try to prove Shannon’s bold claim. I explained the issue, mentioned several others who had tried very hard and failed miserably, and wished him good luck. He soon came back with a proof! Stylistically, it was relentlessly pure mathematics, written with no supervision, by a raw apprentice. His proof was checked, found to be correct, and—once touched up a bit—earned him a Ph.D. in physics. But he received little recognition and soon dropped out of scientific competition. The bulk of the credit stayed with Shannon. This was fair.
Noam Chomsky and László Tisza
My fondest recollections of RLE are of a field one would not expect to have found at the industrial MIT and gritty Building 20. Claude Lévi-Strauss, the illustrious anthropologist I had worked with in Paris, had recommended me to his close friend, the linguist Roman Jakobson. Next I met a Harvard junior fellow, Noam Chomsky, and learned about his project for the future of linguistics. In 1953, it was a wild dream, worlds away from existing mainstreams. Along with many others, I wondered whether and where the new linguistics could find a shelter to survive and develop. Chomsky’s extreme and often restated positions on broad political issues decreased the odds. To his credit, Jerry Wiesner arranged a home for linguistics at the least likely place, MIT. Chomsky stayed and rose to be Institute Professor. The time I spent playing with linguists was wonderful and educational, and left many lasting friends. Roman Jakobson wanted me to forsake seeking new Kepler thrills and make my home in linguistics. But the more I watched, the clearer it became that linguistics was to be dominated by Chomsky. I soon convinced him and his followers of one big thing: Zipf’s law was the basis of an important physics-like (thermodynamical) aspect of discourse, while grammar is like the chemistry or algebra of language. As planned—but not until I had received a first small serving of the combined effects of being admired, I found the field disappointing and moved on.
The Fractalist Page 17