This Will Make You Smarter

by John Brockman

  Just as red does not report the true color of a file, so hotness does not report the true attraction of a face: Files have no intrinsic color; faces have no intrinsic attraction. The color of an icon is an artificial convention to represent aspects of the utility of a colorless file. The initial sense of attractiveness is an artificial convention to represent mate utility.

  The phenomenon of synesthesia can help us to understand the conventional nature of our sensory experiences. In many cases of synesthesia, a stimulus that is normally experienced in one way (say, as a sound) is also automatically experienced in another way (say, as a color). Someone with sound-color synesthesia sees colors and simple shapes whenever they hear a sound. The same sound always occurs with the same colors and shapes. People with taste-touch synesthesia feel touch sensations in their hands every time they taste something. The same taste always occurs with the same feeling of touch in their hands. The particular connections between sound and color that one sound-color synesthete experiences typically differ from the connections experienced by another such synesthete. In this sense, the connections are an arbitrary convention. Now, imagine a sound-color synesthete who no longer has sound experiences from acoustic stimuli and instead has only the synesthetic color experiences. This synesthete would experience only as colors what the rest of us experience as sounds. In principle, they could get all the acoustic information the rest of us get, except in a color format rather than a sound format.

  This leads to the concept of a sensory desktop. Our sensory experiences—such as vision, sound, taste, and touch—can be thought of as sensory desktops that have evolved to guide adaptive behavior, not report objective truths. As a result, we should take our sensory experiences seriously. If something tastes putrid, we probably shouldn’t eat it. If it sounds like a rattlesnake, we probably should avoid it. Our sensory experiences have been shaped by natural selection to guide such adaptive behaviors.

  We must take our sensory experiences seriously but not literally. This is one place where the concept of a sensory desktop is helpful. We take the icons on a graphical desktop seriously; we won’t, for instance, carelessly drag an icon to the trash, for fear of losing a valuable file. But we don’t take the colors, shapes, or locations of the icons literally. They are not there to resemble the truth. They are there to facilitate useful behaviors.

  Sensory desktops differ across species. A face that could launch a thousand ships probably has no attraction to a macaque. The carrion that tastes putrid to me might taste like a delicacy to a vulture. My taste experience guides behaviors appropriate for me; eating carrion could kill me. The vulture’s taste experience guides behaviors appropriate to it; carrion is its primary food source.

  Much of evolution by natural selection can be understood as an arms race between competing sensory desktops. Mimicry and camouflage exploit limitations in the sensory desktops of predators and prey. A mutation that alters a sensory desktop to reduce such exploitation conveys a selective advantage. This cycle of exploiting and revising sensory desktops is a creative engine of evolution.

  On a personal level, the concept of a sensory desktop can enhance our cognitive toolkit by refining our attitude toward our own perceptions. It is common to assume that the way I see the world is, at least in part, the way it really is. Because, for instance, I experience a world of space and time and objects, it is common to assume that these experiences are, or at least resemble, objective truths. The concept of a sensory desktop reframes all this. It loosens the grip of sensory experiences on the imagination. Space, time, and objects might just be aspects of a sensory desktop specific to Homo sapiens. They might not be deep insights into objective truths, just convenient conventions that have evolved to allow us to survive in our niche. Our desktop is just a desktop.

  The Senses and the Multisensory

  Barry C. Smith

  Director, Institute of Philosophy, School of Advanced Study, University of London; writer and presenter, BBC World Service series The Mysteries of the Brain

  For far too long, we have labored under a faulty conception of the senses. Ask anyone you know how many senses we have and they will probably say five—unless they start talking to you about a sixth sense. But why pick five? What of the sense of balance provided by the vestibular system, telling you whether you are going up or down in a lift, forward or backward on a train, or side to side on a boat? What about proprioception that gives you a firm sense of where your limbs are when you close your eyes? What about feeling pain, or heat and cold? Are these just part of touch, like feeling velvet or silk? And why think of sensory experiences like seeing, hearing, tasting, touching, and smelling as being produced by a single sense?

  Contemporary neuroscientists have postulated two visual systems—one responsible for how things look to us, the other for controlling action—that operate independently of each other. The eye may fall for visual illusions, but the hand does not, reaching smoothly for a shape that looks larger than it is.

  And it doesn’t stop here. There is good reason to think that we have two senses of smell: (1) an external sense of smell—orthonasal olfaction, produced by inhaling—that enables us to detect such things in the environment as food, predators, and smoke; and (2) an internal sense—retronasal olfaction, produced by exhaling—that enables us to detect the quality of what we have just eaten, allowing us to decide whether we want any more or should expel it. Associated with each sense of smell is a distinct hedonic response. Orthonasal olfaction gives rise to the pleasure of anticipation. Retronasal olfaction gives rise to the pleasure of reward. Anticipation is not always matched by reward. Have you ever noticed how the enticing aromas of freshly brewed coffee are never quite matched by the taste? There is always a little disappointment. Interestingly, the one food where the intensity of orthonasally and retronasally judged aromas match perfectly is chocolate. We get just what we expected, which may explain why chocolate is such a powerful stimulus.

  Besides the proliferation of the senses in contemporary neuroscience, another major change is taking place. We used to study the senses in isolation, with the greatest majority of researchers focusing on vision. Things are rapidly changing. We now know that the senses do not operate in isolation but combine, both at early and late stages of processing, to produce our rich perceptual experiences of our surroundings. It is almost never the case that our experience presents us with just sights or sounds. We are always enjoying conscious experiences made up of sights and sounds, smells, the feel of our body, the taste in our mouths—and yet these are not presented as separate sensory parcels. We simply take in the rich and complex scene without giving much thought to how the different contributors produce the whole experience.

  We give little thought to how smell provides a background to every conscious waking moment. People who lose their sense of smell can be plunged into depression and, a year later, show less sign of recovery than people who lose their sight. This is because familiar places no longer smell the same and people no longer have their reassuring olfactory signature. Also, patients who lose their smell believe they have lost their sense of taste. When tested, they acknowledge that they can taste sweet, sour, salt, bitter, savory, and metallic. But everything else, missing from the taste of what they are eating, is due to retronasal smell.

  What we call “taste” is one of the most fascinating case studies for how inaccurate our view of our senses is: It is not produced by the tongue alone but is always an amalgam of taste, touch, and smell. Touch contributes to sauces tasting creamy and other foods tasting chewy, crisp, or stale. The only difference between potato chips that “taste” fresh or stale is a difference in texture. The largest part of what we call “taste” is in fact smell in the form of retronasal olfaction, which is why people who lose their ability to smell say they can no longer taste anything. Taste, touch, and smell are not merely combined to produce experiences of foods or liquids; rather, the information from the separate sensory channe
ls is fused into a unified experience of what we call taste and what food scientists call flavor.

  Flavor perception is the result of multisensory integration of gustatory, olfactory, and oral somatosenory information into a single experience whose components we are unable to distinguish. It is one of the most multisensory experiences we have, and it can be influenced by both sight and sound. The colors of wines and the sounds that food make when we bite or chew them can significantly affect our resulting appreciation and assessment; irritation of the trigeminal nerve in the face will make chilies feel “hot” and menthol feel “cool” in the mouth without any actual change in temperature.

  In sensory perception, multisensory integration is the rule, not the exception. In audition, we don’t just hear with our ears, we use our eyes to locate the apparent sources of sounds in the cinema where we “hear” the voices coming from the actors’ mouths on the screen, although the sounds are coming from the sides of the theater. This is known as the ventriloquism effect. Similarly, retronasal odors detected by olfactory receptors in the nose are experienced as tastes in the mouth. The sensations get relocated to the mouth because oral sensations of chewing or swallowing capture our attention, making us think these olfactory experiences are occurring in the same place.

  Other surprising collaborations among the senses are due to cross-modal effects, whereby stimulation of one sense boosts activity in another. Looking at someone’s lips across a crowded room can improve our ability to hear what they are saying, and the smell of vanilla can make a liquid we sip “taste” sweeter and less sour. This is why we say vanilla is sweet-smelling, although sweet is a taste and pure vanilla is not sweet at all. Industrial manufacturers know about these effects and exploit them. Certain aromas in shampoos, for example, can make the hair “feel” softer; red-colored drinks “taste” sweet, whereas drinks with a light green color “taste” sour. In many of these interactions vision will dominate, but not in every case.

  People unlucky enough to have a disturbance in their vestibular system will feel the world is spinning, although cues from the eyes and body should be telling them that everything is still. In people without this difficulty, the brain goes with the vision, and proprioception falls into line. Luckily, our senses cooperate, and we get ourselves around the world we inhabit—and that world is not a sensory but a multisensory world.

  The Umwelt

  David Eagleman

  Neuroscientist; director, Laboratory for Perception and Action, Initiative on Neuroscience and Law, Baylor College of Medicine; author, Incognito: The Secret Lives of the Brain

  In 1909, the biologist Jakob von Uexküll introduced the concept of the umwelt. He wanted a word to express a simple (but often overlooked) observation: Different animals in the same ecosystem pick up on different environmental signals. In the blind and deaf world of the tick, the important signals are temperature and the odor of butyric acid. For the black ghost knifefish, it’s electrical fields. For the echolocating bat, it’s air-compression waves. The small subset of the world that an animal is able to detect is its umwelt. The bigger reality, whatever that might mean, is called the umgebung.

  The interesting part is that each organism presumably assumes its umwelt to be the entire objective reality “out there.” Why would any of us stop to think that there is more beyond what we can sense? In the movie The Truman Show, the eponymous Truman lives in a world completely constructed around him by an intrepid television producer. At one point, an interviewer asks the producer, “Why do you think Truman has never come close to discovering the true nature of his world?” The producer replies, “We accept the reality of the world with which we’re presented.” We accept our umwelt and stop there.

  To appreciate the amount that goes undetected in our lives, imagine you’re a bloodhound. Your long nose houses 200 million scent receptors. On the outside, your wet nostrils attract and trap scent molecules. The slits at the corners of each nostril flare out to allow more air flow as you sniff. Even your floppy ears drag along the ground and kick up scent molecules. Your world is all about olfaction. One afternoon, as you’re following your master, you’re stopped in your tracks by a revelation. What is it like to have the pitiful, impoverished nose of a human being? What can humans possibly detect when they take in a feeble little noseful of air? Do they suffer a hole where smell is supposed to be?

  Obviously, we suffer no absence of smell, because we accept reality as it’s presented to us. Without the olfactory capabilities of a bloodhound, it rarely strikes us that things could be different. Similarly, until a child learns in school that honeybees enjoy ultraviolet signals and rattlesnakes employ infrared, it does not strike her that plenty of information is riding on channels to which we have no natural access. From my informal surveys, it is very uncommon knowledge that the part of the electromagnetic spectrum visible to us is less than a ten-trillionth of it.

  A good illustration of our unawareness of the limits of our umwelt is that of color-blind people: Until they learn that others can see hues they cannot, the thought of extra colors does not hit their radar screen. The same goes for the congenitally blind: Being sightless is not like experiencing “blackness” or “a dark hole” where vision should be. Like the human compared with the bloodhound, blind people do not miss vision; they do not conceive of it. The visible part of the spectrum is simply not part of their umwelt.

  The more science taps into these hidden channels, the more it becomes clear that our brains are tuned to detect a shockingly small fraction of the surrounding reality. Our sensorium is enough for us to get by in our ecosystem, but it does not approximate the larger picture.

  It would be useful if the concept of the umwelt were embedded in the public lexicon. It neatly captures the idea of limited knowledge, of unobtainable information, of unimagined possibilities. Consider the criticisms of policy, the assertions of dogma, the declarations of fact that you hear every day, and just imagine that all of these could be infused with the proper intellectual humility that comes from appreciating the amount unseen.

  The Rational Unconscious

  Alison Gopnik

  Psychologist, University of California–Berkeley; author, The Philosophical Baby: What Children’s Minds Tell Us About Truth, Love, and the Meaning of Life

  One of the greatest scientific insights of the twentieth century was that most psychological processes are not conscious. But the “unconscious” that made it into the popular imagination was Freud’s irrational unconscious—the unconscious as a roiling, passionate id, barely held in check by conscious reason and reflection. This picture is still widespread, even though Freud has been largely discredited scientifically.

  The “unconscious” that has actually led to the greatest scientific and technological advances might be called Turing’s rational unconscious. If the version of the “unconscious” you see in movies like Inception were scientifically accurate, it would include phalanxes of nerds with slide rules instead of women in negligees wielding revolvers amid Daliesque landscapes. At least that might lead the audience to develop a more useful view of the mind—though probably not to buy more tickets.

  Earlier thinkers like Locke and Hume anticipated many of the discoveries of psychological science but thought that the fundamental building blocks of the mind were conscious “ideas.” Alan Turing, the father of the modern computer, began by thinking about the highly conscious and deliberate step-by-step calculations performed by human “computers” like the women decoding German ciphers at Bletchley Park. His first great insight was that the same processes could be instantiated in an entirely unconscious machine, with the same results. A machine could rationally decode the German ciphers using the same steps that the conscious “computers” went through. And the unconscious relay-and-vacuum-tube computers could get to the right answers in the same way the flesh-and-blood ones could.

  Turing’s second great insight was that we could understand much of the hum
an mind and brain as an unconscious computer too. The women at Bletchley Park brilliantly performed conscious computations in their day jobs, but they were unconsciously performing equally powerful and accurate computations every time they spoke a word or looked across the room. Discovering the hidden messages about three-dimensional objects in the confusing mess of retinal images is just as difficult and important as discovering the hidden messages about submarines in the incomprehensible Nazi telegrams, and it turns out that the mind solves both mysteries in a similar way.

  More recently, cognitive scientists have added the idea of probability into the mix, so that we can describe an unconscious mind, and design a computer, that can perform feats of inductive as well as deductive inference. Using this sort of probabilistic logic, a system can accurately learn about the world in a gradual, probabilistic way, raising the probability of some hypotheses and lowering that of others, and revising hypotheses in the light of new evidence. This work relies on a kind of reverse engineering. First, work out how any rational system could best infer the truth from the evidence it has. Often enough, you will find that the unconscious human mind does just that.

  Some of the greatest advances in cognitive science have been the result of this strategy. But they have been largely invisible in popular culture, which has been understandably preoccupied with the sex and violence of much evolutionary psychology (like Freud, it makes for a better movie). Vision science studies how we are able to transform the chaos of stimulation at our retinas into a coherent and accurate perception of the outside world. It is, arguably, the most scientifically successful branch of both cognitive science and neuroscience. It takes off from the idea that our visual system is, entirely unconsciously, making rational inferences from retinal data to figure out what objects are like. Vision scientists began by figuring out the best way to solve the problem of vision and then discovered, in detail, just how the brain performs those computations.