The Long Tail

by Chris Anderson

  When you place an item in your wire shopping basket, the store knows nothing about it until you arrive at the checkout, at which time it’s too late to do anything but feebly give you a coupon for discounts on future purchases. Somewhere, retail scientists dream of smart shopping carts that detect their contents by radio frequency ID tags and then spit out recommendations on the fly. Yet even those scientists still can’t transport matter into reach and make acting on those recommendations easy. In the physical world, shoppers move; products don’t.

  THE WAL-MART EFFECT

  When I was a twenty-something slacker, I, like many of my twenty-something kin, worked in a record store. It was a pretty big record store in the business district of downtown Washington, D.C., part of a chain that no longer exists.

  Catering mostly to the lawyers, admins, and paralegals who worked around the area, the store was relatively mainstream. Nevertheless I can still remember the aisle of import records, mostly British new wave (after all, this was the mid-eighties) that stretched the length of the store near the stool where I sat watching the door and answering questions. The entire back wall was twelve-inch singles (Depeche Mode and Billy Idol were big), and classical had its own room, with excellent acoustics for more refined listening.

  This all came back to me recently as I wandered through the two aisles of the music department at a Wal-Mart in Oakland, California. (And really, is wandered even the right word for fifteen paces down one aisle and back the other?) Wal-Mart, which accounts for about one-fifth of all music sales in America, is by far the nation’s largest music retailer. Some 138 million Americans shop at Wal-Mart each week, making it perhaps the single most unifying cultural force in the country.

  Over the past decade, these types of big-box retailers, including stores like Best Buy, have changed the face of the music industry with their unmatchable economies of scale. Today, the number of large independent music stores like the one I once worked at has dropped dramatically; the classical listening room is now an endangered species. There are, needless to say, few import aisles left.

  In the place of specialist stores’ often eclectic collections, the superstores offer just a relatively small selection of hits. It’s ironic that such big stores carry so little in each category, but that’s what the economics of big-boxery dictates. Still, their prices are excellent and they are packed with eager shoppers. A triumph of supply-chain efficiencies and bulk pricing, these big retailers are the state of the art in bricks-and-mortar retail today.

  Welcome to the Short Head.

  How short is short? The average Wal-Mart now carries around 4,500 unique CD titles (as a point of comparison, Amazon lists about 800,000). More specifically, consider the music department at that Wal-Mart I strolled through in Oakland. Here’s the number of records I found for each of the store’s categories:

  “Rock/Pop/R&B”

  1,800

  “Latina”

  1,500

  “Christian/Gospel”

  360

  “Country”

  225

  “Classical/Easy Listening”

  225

  Again, there were just two main aisles. One was “Rock/Pop/R&B”; the other was “Latina.” All other categories were lumped into single four-by-five-foot racks. “Jazz,” “Classical,” “World Music,” “Easy Listening,” and “New Age” were all together along one rack.

  Of the estimated 30,000 new albums released each year, Wal-Mart carries just 750, according to David Gottlieb, a former label executive. That works out to only 2.5 percent of the new music released each year; and those 4,500 titles in the total inventory are less than half a percent of all the music available. Entire categories, from Dance to Spoken-Word, are either missing or buried deep in catch-all categories such as “Rock/Pop/R&B.” There are no copies of the Rolling Stones’ Exile on Main Street or Nirvana’s Nevermind.

  There you have it. Scarcity, bottlenecks, the distortion of distribution, and the tyranny of shelf space all wrapped up in one big store. Again, it’s ironic, this paradox of plenty: Walk into a Wal-Mart and you’re overwhelmed by the abundance and choice. Yet look closer and the utter thinness of this cornucopia is revealed. Wal-Mart’s shelves are a display case a mile wide and twenty-four inches deep. At first glance that may look like everything, but in a world that’s actually a mile wide and a mile deep, a veneer of variety just isn’t enough.

  IN THE LIBRARY OF MISSHELVED BOOKS

  One of the most vexing problems with physical goods is that they force us into crude categorization and static taxonomies, as we saw with Wal-Mart. That means that a windbreaker can be in the “Jackets” section or the “Sports” section, but not in a “Blue” or “Nylon” section. Generally, this isn’t seen as a big problem, since most of those categories would be silly for most people (the one-size-fits-all economics of retail must ignore the few shoppers those categories would be perfect for).

  As a store manager, you have to guess as to where most people would expect to find a windbreaker. So after constructing your store around a preconceived taxonomy, you can do nothing but hope that your setup corresponds to the way most people think. And those customers that don’t, in fact, think this way? One hopes they’ll ask for assistance.

  With the evolution of online retail, however, has come the revelation that being able to recategorize and rearrange products on the fly unlocks their real value. For one, online stores are free to list products in whichever, and however many, sections they choose. This captures the attention of potential buyers who wouldn’t have found the product in the default category, and it also stimulates demand in people who weren’t even looking for the product in the first place but were spurred to buy because of clever positioning.

  The efficiency and success of online retail have illuminated the cost of traditional retail’s inflexibility and taxonomical oversimplifications. It’s one thing to have high prices or limited selection; it’s quite another to simply be unable to help people find what they want.

  In the world of information science, the tricky question of where to put things is known as the “ontology problem.” Ontology is a word that means different things in different disciplines, but for librarians and computer scientists (and for store managers, whether they know it or not), it’s about ways to organize things. The Dewey Decimal System is one way to organize books; the Encyclopædia Britannica is one way to organize information; the Periodic Table of the Elements is one way to organize matter.

  All of these are successes, as far as they go. However, as the Google era has shown, we’re suddenly realizing how limited those fixed ways of making sense of the world really are.

  Let’s start with the Dewey Decimal System, which divides the world of knowledge into ten top-level categories:

  000 Computers, information, and general reference

  100 Philosophy and psychology

  200 Religion

  300 Social sciences

  400 Language

  500 Science and mathematics

  600 Technology

  700 Arts and recreation

  800 Literature

  900 History and geography

  Seems reasonable so far, right? Okay, let’s look at the next level of organization, the second digit. Here it is for Category 200, Religion:

  200 Religion

  210 Philosophy and theory of religion

  220 Bible

  230 Christianity

  240 Christian moral and devotional theology

  250 Christian orders and local churches

  260 Social and ecclesiastical theology

  270 History of Christianity and Christian sects

  280 Christian denominations

  290 Comparative religion and other religions

  See the problem? Islam, Judaism, Hinduism, Taoism, and all the world’s other religions, which account for most of the global population, are lumped into a subset of the “other” category. This taxonomy says more about the culture of nineteenth-century A
merica in which the system was developed (and probably something about Melvil Dewey himself) than it does about the world of faith.

  Truth be told, the Dewey Decimal taxonomy really isn’t about the world of knowledge at all; it’s about the world of books. Clay Shirky, a prominent thinker on the social and economic effects of Internet technologies, explains:

  What’s being optimized is the number of books on the shelf. The musculature of this scheme looks like it’s about concepts. It’s organized into non-overlapping categories that get more detailed at lower and lower levels—any concept is supposed to fit in one category and in no other categories. But every now and then the skeleton pokes through, and that skeleton, the supporting structure around which the system is really built, is designed to minimize seek times on shelves.

  We’ve come a long way since the nineteenth century, of course. Libraries built card catalogs that cross-indexed books by multiple categories: authors, titles, keywords, alternative subjects. Eventually came digital catalogs and keyword search, which at least made things findable. Regardless, the physical books were still stacked on the shelves according to the Dewey Decimal System. This meant that although you could now locate the book you wanted (even if you didn’t subscribe to Melvil Dewey’s Victorian worldview), you might not find much relevance in the books stacked around it.

  Even with the card catalog, books are still vulnerable to the physics of materiality. Consider what happens when one is stacked in the wrong place, orphaned in the wrong category. It’s as if it were vaporized. Unless someone stumbles upon it and reshelves it, that book will effectively be lost to the world (even though it’s still sitting somewhere). No wonder the semantics of shelves are often so negative. “Shelf life” refers to the mortality of expiring goods—whether literal (think: bananas just starting to brown) or figurative (think: Halloween-themed paper plates in March). In the realm of film and television, “shelved” means canceled or delayed. Shelves are places where things go to die.

  On the other hand, think about a world of ad-hoc organization, determined by whatever makes sense at the time. That’s more like a big pile of stuff on a desk instead of rows of items stringently arranged on shelves. Sure it may seem messy, but that’s just because it’s a different kind of organization: spontaneous, contextual order, easily reordered into a different context as need be. That image is a little bit like the Web itself, seen through Google’s lens: a world of infinite variety and little predetermined order; a world of dynamic structure, shaped differently for each observer.

  Recently, I toured the new Seattle Public Library, which was designed by famed architect Rem Koolhaas to be a model library for the twenty-first century. He faced the challenge of making stacks of books fit into a search-engine culture. Realizing that the relative balance between computers and books was changing and would probably continue to change, Koolhaas didn’t make too many assumptions about how books should be shelved. He arranged the stacks on rails in a spiral, which could expand or contract as demand dictated.

  Yet even within this commendably flexible system, he obviously needed to arrange the books in some order. Since it takes more than the turn of a century or two to change library culture, that order was our friend the Dewey Decimal System. However, in the Seattle Public Library the Dewey numbers are marked on the floor on rubber mats that slide into grooves in the concrete. As the stacks accordion and shift with the world’s changing information priorities, the rubber mats will change as well. And if, someday, the Dewey system has reached its own expiry date, those rubber mats can be turned over to provide nothing more than a good place to wipe your shoes. A future-proof library makes no assumptions about the information landscape of tomorrow.

  SHOPPING IN THE MISCELLANEOUS AISLE

  What’s true for libraries is doubly true for retail stores. In libraries, at least there is a standard categorization scheme—the card catalog is there to be searched, and librarians tend to know their stuff. However, good luck finding what you want quickly in an unfamiliar supermarket. The consequences of ad-hoc taxonomies and capricious shelving are frustrated customers, unsold products, and a flight to the best-known brands and products, simply because those are the ones that are easy to find. Likewise, for most other kinds of stores, from hardware to clothes.

  As an example, I recently went looking in my local Blockbuster for Akira, a Japanese anime classic. What section to look in? Science fiction? Animation? Foreign? Action? As it turned out, it didn’t matter—they didn’t have the film. Physical stores’ advantages of immediate gratification are of little meaning if you can’t find what you want.

  On Amazon, however, it was simply a matter of typing “akira” into the search bar (and just note how there’s no need to capitalize or even necessarily to spell it quite right). The film immediately came up, as did two other versions (as well as both new and used copies of all three). If I had wanted to browse by category, any of those above would have worked; the film was listed in all of them. A tempting package deal with Ghost in the Shell was offered, another virtue of dynamic marketing and positioning. Likewise, Amazon also recommended two other films it thought I might like: Princess Mononoke and Ghost in the Shell 2. And, of course, both of those were also in stock and cheaper than Blockbuster. The experiences I had with these two stores couldn’t have been more different.

  In a sense, an online retailer is to a bricks-and-mortar store what Google is to a library. Because of the constraints of physical shelves, the real-world outlets are forced to create taxonomies and assign everything to them. I tremble to think where the Dewey Decimal System will place the book you’re reading right now. Technology? Economics? Business? Culture? None of them are quite right by themselves. Sadly, there is no category for “all of the above.”

  Google, by contrast, will put it in no category at all. The book’s natural place (s) in the world will emerge spontaneously after the fact, measured in terms of incoming links. My publisher might call this a “business book,” but if the world decides it’s really more “popular economics” and links to it in that context, then that is what it is and what it will be, along with virtually any other description that someone may find relevant. In a Google world, meaning and ontology are entirely in the eyes and minds of the beholder. One thing can be many different things to many different people. As such, Google’s algorithms simply measure the wisdom of the crowd by calculating the most appropriate results for the keywords a searcher types in.

  Meanwhile, Amazon will start by giving this book five or six category designations. Customers will then have their say by “tagging” it, which means typing in any words they choose to make their own categories (“Internet,” “blogger,” “to read later,” “Pareto,” “good geek gift,” etc.). Others will be able to see what tags have been assigned, which is another useful piece of context that will help this book find its place in the world. This process of tagging creates what are known as “folksonomies”—after-the-fact categorizations based entirely on whatever people choose to say is meaningful about something. Interestingly, Amazon gives these tags so much weight that they appear before its own list of preset categories.

  Still, that is only the start of the multidimensional process of teasing out what something is in the infinite bookstore. Amazon’s software will digest every word of this book’s text and determine a list of “statistically improbable phrases,” which are word combinations that do not appear in many, if any, other books. In a sense, these will comprise a unique fingerprint of my book, but they’re also an indication of any unique ideas or subject areas, which is useful in itself. The software will also list unique capitalized words, which will help define the factual foundation of my book. Then, Amazon will deploy all its usual collaborative filtering recommendation tools to find books that other customers looked at or bought along with mine, which will help define the book through its peer set.

  THE TYRANNY OF GEOGRAPHY

  Shelves have another disadvantage: They are bound by geography
. Their contents are available only to people who happen to be in the same place as they are. That is, of course, also their virtue: The stores near you are convenient and offer the immediate gratification of sending your purchase home with you. For all the time we may spend online, we do, after all, live in the physical world.

  The main constraint of bricks-and-mortar retail is the need to find local audiences. Whether we’re talking about movies, CDs, or any number of products, bricks-and-mortar retailers will carry only the content that earns its keep, products that attract the greatest amount of interest (and dollars) from the limited local population.

  In America, 20 percent of the population live more than eight miles from the closest bookstore; 8 percent live more than twenty miles away. The numbers for music stores, movie theaters, and video rental shops aren’t much different. Even if everyone wanted to buy that way, many can’t.

  Remember, in the tyranny of physical space, an audience too thinly spread is the same as no audience at all. Thus, local demand must be at a high enough concentration to compensate for the high costs of physical distribution. In other, more obvious words, not enough local demand equals no store.

  This is true for goods of any sort. There is a reason why ski stores are not often found in hot climates and diving stores are not often found inland (despite the fact that lots of people fly from both sorts of places to ski and dive). There may be a local demand for the goods, but again, the question for any store owner is whether there’s enough local demand. The calculation goes a little like this:

  Sales =

  The percentage of the population who might buy

  Minus

  The percentage not within ten miles of the store