by Jaron Lanier
For the first time, we can at least tell the outlines of a reasonable story about how your brain is recognizing things out in the world—such as smiles—even if we aren’t sure of how to tell if the story is true. Here is that story …
What the World Looks Like to a Statistical Algorithm
I’ll start with a childhood memory. When I was a boy growing up in the desert of southern New Mexico, I began to notice patterns on the dirt roads created by the tires of passing cars. The roads had wavy corduroylike rows that were a little like a naturally emerging, endless sequence of speed bumps. Their spacing was determined by the average speed of the drivers on the road.
When your speed matched that average, the ride would feel less bumpy. You couldn’t see the bumps with your eyes except right at sunset, when the horizontal red light rays highlighted every irregularity in the ground. At midday you had to drive carefully to avoid the hidden information in the road.
Digital algorithms must approach pattern recognition in a similarly indirect way, and they often have to make use of a common procedure that’s a little like running virtual tires over virtual bumps. It’s called the Fourier transform. A Fourier transform detects how much action there is at particular “speeds” (frequencies) in a block of digital information.
Think of the graphic equalizer display found on audio players, which shows the intensity of the music in different frequency bands. The Fourier transform is what does the work to separate the frequency bands.)
Unfortunately, the Fourier transform isn’t powerful enough to recognize a face, but there is a related but more sophisticated transform, the Gabor wavelet transform, that can get us halfway there. This mathematical process identifies individual blips of action at particular frequencies in particular places, while the Fourier transform just tells you what frequencies are present overall.
There are striking parallels between what works in engineering and what is observed in human brains, including a Platonic/Darwinian duality: a newborn infant can track a simple diagrammatic face, but a child needs to see people in order to learn how to recognize individuals.
I’m happy to report that Hartmut’s group earned some top scores in a government-sponsored competition in facial recognition. The National Institute of Standards and Technology tests facial recognition systems in the same spirit in which drugs and cars are tested: the public needs to know which ones are trustworthy.
From Images to Odors
So now we are starting to have theories—or at least are able to tell detailed stories—about how a brain might be able to recognize features of its world, such as a smile. But mouths do more than smile. Is there a way to extend our story to explain what a word is, and how a brain can know a word?
It turns out that the best way to consider that question might be to consider a completely different sensory domain. Instead of sights or sounds, we might best start by considering the odors detected by a human nose.
For twenty years or so I gave a lecture introducing the fundamentals of virtual reality. I’d review the basics of vision and hearing as well as of touch and taste. At the end, the questions would begin, and one of the first ones was usually about smell: Will we have smells in virtual reality machines anytime soon?
Maybe, but probably just a few. Odors are fundamentally different from images or sounds. The latter can be broken down into primary components that are relatively straightforward for computers—and the brain—to process. The visible colors are merely words for different wavelengths of light. Every sound wave is actually composed of numerous sine waves, each of which can be easily described mathematically. Each one is like a particular size of bump in the corduroy roads of my childhood.
In other words, both colors and sounds can be described with just a few numbers; a wide spectrum of colors and tones is described by the interpolations between those numbers. The human retina need be sensitive to only a few wavelengths, or colors, in order for our brains to process all the intermediate ones. Computer graphics work similarly: a screen of pixels, each capable of reproducing red, green, or blue, can produce approximately all the colors that the human eye can see.* A music synthesizer can be thought of as generating a lot of sine waves, then layering them to create an array of sounds.
Odors are completely different, as is the brain’s method of sensing them. Deep in the nasal passage, shrouded by a mucous membrane, sits a patch of tissue—the olfactory epithelium—studded with neurons that detect chemicals. Each of these neurons has cup-shaped proteins called olfactory receptors. When a particular molecule happens to fall into a matching receptor, a neural signal is triggered that is transmitted to the brain as an odor. A molecule too large to fit into one of the receptors has no odor. The number of distinct odors is limited only by the number of olfactory receptors capable of interacting with them. Linda Buck of the Fred Hutchinson Cancer Research Center and Richard Axel of Columbia University, winners of the 2004 Nobel Prize in Physiology or Medicine, have found that the human nose contains about one thousand different types of olfactory neurons, each type able to detect a particular set of chemicals.
This adds up to a profound difference in the underlying structure of the senses—a difference that gives rise to compelling questions about the way we think, and perhaps even about the origins of language. There is no way to interpolate between two smell molecules. True, odors can be mixed together to form millions of scents. But the world’s smells can’t be broken down into just a few numbers on a gradient; there is no “smell pixel.” Think of it this way: colors and sounds can be measured with rulers, but odors must be looked up in a dictionary.
That’s a shame, from the point of view of a virtual reality technologist. There are thousands of fundamental odors, far more than the handful of primary colors. Perhaps someday we will be able to wire up a person’s brain in order to create the illusion of smell. But it would take a lot of wires to address all those entries in the mental smell dictionary. Then again, the brain must have some way of organizing all those odors. Maybe at some level smells do fit into a pattern. Maybe there’s a smell pixel after all.
Were Odors the First Words?
I’ve long discussed this question with Jim Bower, a computational neuroscientist at the University of Texas at San Antonio, best known for making biologically accurate computer models of the brain. For some years now, Jim and his laboratory team have been working to understand the brain’s “smell dictionary.”
They suspect that the olfactory system is organized in a way that has little to do with how an organic chemist organizes molecules (for instance, by the number of carbon atoms on each molecule). Instead, it more closely resembles the complex way that chemicals are associated in the real world. For example, a lot of smelly chemicals—the chemicals that trigger olfactory neurons—are tied to the many stages of rotting or ripening of organic materials. As it turns out, there are three major, distinct chemical paths of rotting, each of which appears to define a different stream of entries in the brain’s dictionary of smells.
Keep in mind that smells are not patterns of energy, like images or sounds. To smell an apple, you physically bring hundreds or thousands of apple molecules into your body. You don’t smell the entire form; you steal a piece of it and look it up in your smell dictionary for the larger reference.
To solve the problem of olfaction—that is, to make the complex world of smells quickly identifiable—brains had to have evolved a specific type of neural circuitry, Jim believes. That circuitry, he hypothesizes, formed the basis for the cerebral cortex—the largest part of our brain, and perhaps the most critical in shaping the way we think. For this reason, Jim has proposed that the way we think is fundamentally based in the olfactory.
A smell is a synecdoche: a part standing in for the whole. Consequently, smell requires additional input from the other senses. Context is everything: if you are blindfolded in a bathroom and a good French cheese is placed under your nose, your interpretation of the odor will likely be very differen
t than it would be if you knew you were standing in a kitchen. Similarly, if you can see the cheese, you can be fairly confident that what you’re smelling is cheese, even if you’re in a restroom.
Recently, Jim and his students have been looking at the olfactory systems of different types of animals for evidence that the cerebral cortex as a whole grew out of the olfactory system. He often refers to the olfactory parts of the brain as the “Old Factory,” as they are remarkably similar across species, which suggests that the structure has ancient origins. Because smell recognition often requires input from other senses, Jim is particularly interested to know how that input makes its way into the olfactory system.
In fish and amphibians (the earliest vertebrates), the olfactory system sits right next to multimodal areas of the cerebral cortex, where the processing of the different senses overlaps. The same is true in reptiles, but in addition, their cortex has new regions in which the senses are separated. In mammals, incoming sights, sounds, and sensations undergo many processing steps before ending up in the region of overlap. Think of olfaction as a city center and the other sensory systems as sprawling suburbs, which grew as the brain evolved and eventually became larger than the old downtown.
All of which has led Jim and me to wonder: Is there a relationship between olfaction and language, that famous product of the human cerebral cortex? Maybe the dictionary analogy has a real physical basis.
Olfaction, like language, is built up from entries in a catalog, not from infinitely morphable patterns. Moreover, the grammar of language is primarily a way of fitting those dictionary words into a larger context. Perhaps the grammar of language is rooted in the grammar of smell. Perhaps the way we use words reflects the deep structure of the way our brain processes chemical information. Jim and I plan to test this hypothesis by studying the mathematical properties that emerge during computer simulations of the neurology of olfaction.
If that research pans out, it might shed light on some other connections we’ve noticed. As it happens, the olfactory system actually has two parts: one detects general odors, and the other, the pheremonic system, detects very specific, strong odors given off by other animals (usually of the same species), typically related to fear or mating. But the science of olfaction is far from settled, and there’s intense controversy about the importance of pheromones in humans.
Language offers an interesting parallel. In addition to the normal language we all use to describe objects and activities, we reserve a special language to express extreme emotion or displeasure, to warn others to watch out or get attention. This language is called swearing.
There are specific neural pathways associated with this type of speech; some Tourette’s patients, for instance, are known to swear uncontrollably. And it’s hard to overlook the many swear words that are related to orifices or activities that also emit pheremonic olfactory signals. Could there be a deeper connection between these two channels of “obscenity”?
Clouds Are Starting to Translate
Lngwidge iz a straynge thingee. You can probably read that sentence without much trouble. Sentence also not this time hard.
You can screw around quite a bit with both spelling and word order and still be understood. This shouldn’t be surprising: language is flexible enough to evolve into new slang, dialects, and entirely new tongues.
In the 1960s, many early computer scientists postulated that human language was a type of code that could be written down in a neat, compact way, so there was a race to crack that code. If it could be deciphered, then a computer ought to be able to speak with people! That end result turned out to be extremely difficult to achieve. Automatic language translation, for instance, never really took off.
In the first decade of the twenty-first century, computers have gotten so powerful that it has become possible to shift methods. A program can look for correlations in large amounts of text. Even if it isn’t possible to capture all the language variations that might appear in the real world (such as the above oddities I used as examples), a sufficiently huge number of correlations eventually yields results.
For instance, suppose you have a lot of text in two languages, such as Chinese and English. If you start searching for sequences of letters or characters that appear in each text under similar circumstances, you can start to build a dictionary of correlations. That can produce significant results, even if the correlations don’t always fit perfectly into a rigid organizing principle, such as a grammar.
Such brute-force approaches to language translation have been demonstrated by companies like Meaningful Machines, where I was an adviser for a while, and more recently by Google and others. They can be incredibly inefficient, often involving ten thousand times as much computation as older methods—but we have big enough computers in the clouds these days, so why not put them to work?
Set loose on the internet, such a project could begin to erase language barriers. Even though automatic language translation is unlikely to become as good as what a human translator can do anytime soon, it might get good enough—perhaps not too far in the future—to make countries and cultures more transparent to one another.
Editing Is Sexy; Creativity Is Natural
These experiments in linguistic variety could also inspire a better understanding of how language came about in the first place. One of Charles Darwin’s most compelling evolutionary speculations was that music might have preceded language. He was intrigued by the fact that many species use song for sexual display and wondered if human vocalizations might have started out that way too. It might follow, then, that vocalizations could have become varied and complex only later, perhaps when song came to represent actions beyond mating and such basics of survival.
Language might not have entirely escaped its origins. Since you can be understood even when you are not well-spoken, what is the point of being well-spoken at all? Perhaps speaking well is still, in part, a form of sexual display. By being well-spoken I show not only that I am an intelligent, clued-in member of the tribe but also that I am likely to be a successful partner and helpful mate.
Only a handful of species, including humans and certain birds, can make a huge and ever-changing variety of sounds. Most animals, including our great-ape relatives, tend to repeat the same patterns of sound over and over. It is reasonable to suppose that an increase in the variety of human sounds had to precede, or at least coincide with, the evolution of language. Which leads to another question: What makes the variety of sounds coming from a species increase?
As it happens, there is a well-documented case of song variety growing under controlled circumstances. Kazuo Okanoya of the Riken Institute in Tokyo compared songs between two populations of birds: the wild white-rump munia and its domesticated variant, the Bengalese finch. Over several centuries, bird fanciers bred Bengalese finches, selecting them for appearance only. Something odd happened during that time: domesticated finches started singing an extreme and evolving variety of songs, quite unlike the wild munia, which has only a limited number of calls. The wild birds do not expand their vocal range even if they are raised in captivity, so the change was at least in part genetic.
The traditional explanation for such a change is that it must provide an advantage in either survival or sexual selection. In this case, though, the finches were well fed and there were no predators. Meanwhile, breeders, who were influenced only by feather coloration, did the mate selection.
Enter Terry Deacon, a scientist who has made fundamental contributions in widely diverse areas of research. He is a professor of anthropology at the University of California at Berkeley and an expert on the evolution of the brain; he is also interested in the chemical origins of life and the mathematics behind the emergence of complicated structures like language.
Terry offered an unconventional solution to the mystery of Bengalese finch musicality. What if there are certain traits, including song style, that naturally tend to become less constrained from generation to generation but are normally held
in check by selection pressures? If the pressures go away, variation should increase rapidly. Terry suggested that the finches developed a wider song variety not because it provided an advantage but merely because in captivity it became possible.
In the wild, songs probably had to be rigid in order for mates to find each other. Birds born with a genetic predilection for musical innovation most likely would have had trouble mating. Once finches experienced the luxury of assured mating (provided they were visually attractive), their song variety exploded.
Brian Ritchie and Simon Kirby of the University of Edinburgh worked with Terry to simulate bird evolution in a computer model, and the idea worked well, at least in a virtual world. Here is yet another example of how science becomes more like storytelling as engineering becomes able to represent some of the machinery of formerly subjective human activities.
Realistic Computationalist Thinking Works Great for Coming Up with Evolutionary Hypotheses
Recent successes using computers to hunt for correlations in giant chunks of text offer a fresh hint that an explosion of variety in song might have been important in human evolution. To see why, compare two popular stories of the beginning of language.
In the first story, a protohuman says his first word for something—maybe ma for “mother”—and teaches it to the rest of the tribe. A few generations later, someone comes up with wa for “water.” Eventually the tribe has enough words to constitute a language.