The Long Tail

by Chris Anderson

  These lists are, in other words, a semi-random collection of totally disparate things.

  To use an analogy, top-blog lists are akin to saying that the best-sellers in the supermarket today were:

  DairyFresh 2% vitamin D milk

  Hayseed Farms mixed grain bread

  Bananas, assorted bunches

  Crunchios cereal, large size

  DietWhoopsy, 12-pack, cans

  and so on…

  Which is pointless. Nobody cares if bananas outsell soft drinks. What they care about is which soft drink outsells which other soft drink. Lists make sense only in context, comparing like with like within a category.

  My take: This is another reminder that you have to treat niches as niches. When you look at a widely diverse three-dimensional marketplace through a one-dimensional lens, you get nonsense. It’s a list, but it’s a list without meaning. What matters is the rankings within a genre (or subgenre), not across genres.

  Let’s take this back to music. As I write, the top ten artists on Rhapsody overall are:

  Jack Johnson

  Eminem

  Coldplay

  Fall Out Boy

  Johnny Cash

  Nickelback

  James Blunt

  Green Day

  Death Cab for Cutie

  Kelly Clarkson

  Which is, as I count it, two “adult alternative,” one “crossover/hiphop,” one “Brit-rock,” one “emo,” one “outlaw country,” one “post-grunge,” one “punk-pop,” one “indie-rock,” and one “teen beat.” Does anybody care if outlaw country outsells teen beat this week or vice versa? Does this list help anyone who is drawn to any of these categories find more music they’ll like? Yet the Top 10 (or Top 40, or Top 100) list is the lens through which we’ve looked at music culture for nearly half a century. It’s mostly meaningless, but it was all we had.

  Let’s contrast that with a different kind of top ten list, that for the music subgenre Afro-Cuban jazz:

  Tito Puente

  Buena Vista Social Club

  Cal Tjader

  Arturo Sandoval

  Poncho Sanchez

  Dizzy Gillespie

  Perez Prado

  Ibrahim Ferrer

  Eddie Palmieri

  Michel Camilo

  Now that’s a top ten list. It’s apples-to-apples and thus useful from top to bottom. Such lists are possible because we have abundant information about consumer preference and enough space for an infinite number of top ten lists—there doesn’t have to be just one. In this case Tito Puente is number one in a very niche category—a big fish in a small pond. For people into this genre, this is a big deal indeed. For those who aren’t, he’s simply another obscure artist and safely ignored. Tito Puente’s albums don’t rise to the top of the overall music charts—they’re not blockbusters. But they do dominate their category, creating what writer Erick Schonfeld calls “nichebusters.” Filters and recommendations work best at this scale, bringing the mainstream discovery and marketing techniques to micromarkets.

  IS THE LONG TAIL FULL OF CRAP?

  Why are filters so important to a functioning Long Tail? Because without them, the Long Tail risks just being noise.

  The field of “information theory” was built around the problem of pulling coherent signals from random electrical noise, first in radio broadcasts and then in any sort of electronic transmission. The notion of a signal-to-noise ratio is now used more broadly to refer to any instance where clearing away distraction is a challenge. In a traditional “Short Head” market this isn’t much of a problem, because everything on the shelf has been prefiltered to remove outliers and other products far from the lowest common denominator. But in a Long Tail market, which includes nearly everything, noise can be a huge problem. Indeed, if left unchecked, noise—random content or products of poor quality—can kill a market. Too much noise and people don’t buy.

  The job of filters is to screen out that noise. Call it pulling wheat from chaff or diamonds from the rough, the role of a filter is to elevate the few products that are right for whoever is looking and suppress the many that aren’t. I’ll explain this by considering one commonly held misperception.

  One of the most frequent mistakes people make about the Long Tail is to assume that things that don’t sell well are “not as good” as things that do sell well. Or, to put it another way, they assume that the Long Tail is full of crap. After all, if that album/book/film/whatever were excellent, it would be a hit, right? Well, in a word, no.

  Niches operate by different economics than the mainstream. And the reason for that helps explain why so much about Long Tail content is counterintuitive, especially when we’re used to scarcity thinking.

  First, let’s get one thing straight: The Long Tail is indeed full of crap.

  Yet it’s also full of works of refined brilliance and depth—and an awful lot in between. Exactly the same can be said of the Web itself. Ten years ago, people complained that there was a lot of junk on the Internet, and sure enough, any casual surf quickly confirmed that. Then along came search engines to help pull some signal from the noise, and finally Google, which taps the wisdom of the crowd itself and turns a mass of incoherence into the closest thing to an oracle the world has ever seen.

  This is not unique to the Web—it’s true everywhere. Sturgeon’s Law (named after the science fiction writer Theodore Sturgeon) states that “ninety percent of everything is crud.” Just think about art, not from the perspective of a gallery but from a garage sale. Ninety percent (at least) is crud. And the same is true for music, books, and everything else. The reason we don’t think of it that way is that most of it is filtered away by the scarcity sieve of commercial retail distribution.

  On a store shelf or in any other limited means of distribution, the ratio of good to bad matters because it’s a zero sum game: Space for one eliminates space for the other. Prominence for one obscures the other. If there are ten crappy toys for each good one in the aisle, you’ll think poorly of the toy store and be discouraged from browsing. Likewise it’s no fun to flip through bin after bin of CDs if you haven’t heard of any of them.

  But where you have unlimited shelf space, it’s a non-zero sum game. The billions of crappy Web pages about whatever are not a problem in the way that billions of crappy CDs on the Tower Records shelves would be. Inventory is “non-rivalrous” on the Web and the ratio of good to bad is simply a signal-to-noise problem, solvable with information tools. Which is to say it’s not much of a problem at all. You just need better filters. In other words, the noise is still out there, but Google allows you to effectively ignore it. Filters rule!

  This leads to the key to what’s different about Long Tails. They are not prefiltered by the requirements of distribution bottlenecks and all those entail (editors, studio execs, talent scouts, and Wal-Mart purchasing managers). As a result their components vary wildly in quality, just like everything else in the world.

  One way to describe this (using the language of information theory again) would be to say that Long Tails have a wide dynamic range of quality: awful to great. By contrast, the average store shelf has a relatively narrow dynamic range of quality: mostly average to good. (There’s some really great stuff, but much of that is too expensive for the average retail shelf; niches exist at both ends of the quality spectrum.)

  So tails have a wide dynamic range and heads have a narrow dynamic range. Graphically, that looks like this:

  It’s crucial to note that there are high-quality goods in every part of the curve, from top to bottom. Yes, there are more low-quality goods in the tail and the average level of quality declines as you go down the curve. But with good filters, averages don’t matter. Diamonds can be found anywhere.

  To clarify, here are some examples of criteria people might use to evaluate content.

  Obviously, the terms “high quality” and “low quality” are entirely subjective, so all of these criteria are in the eye of the beholder. Thus, there a
re no absolute measures of content quality. One person’s “good” could easily be another’s “bad”; indeed, it almost always is.

  This is why niches are different. One person’s noise is another’s signal. If a producer intends something to be absolutely right for one audience, it will, by definition, be wrong for another. The compromises necessary to make something appeal to everyone mean that it will almost certainly not appeal perfectly to anyone—that’s why they call it the lowest common denominator.

  The remarkable consequence of the above graphic is that for many people, the best stuff is in the Tail. If you’re interested in audiophile stereo equipment, the finest gear is not going to be among the top-sellers at Best Buy. It will be too expensive, too complicated, and too hard to sell to the average customer. Instead, it’s going to be available at a specialist, and in overall sales ranking will be far down the Tail. Because this gear is so right for the audiophiles, it’s probably not right for people with less focused interests. Niche products are, by definition, not for everyone.

  Down there in the low-selling side of the curve, there are also products that just aren’t very good. The challenge of filtering is to be able to tell one from the other. If you’ve got help—smart search engines, recommendations, or other filters—your odds of finding something just right for you are actually greater in the Tail. Best-sellers tend to appeal, at least superficially, to a broad range of taste. Niche products are meant to appeal strongly to a narrow set of tastes. That’s why the filter technologies are so important. They not only drive demand down the Tail, but they can also increase satisfaction by connecting people with products that are more right for them than the broad-appeal products at the Head.

  THE TAIL THAT WAGS EVERYTHING ELSE

  Another way to look at the situation is the graph below. As the Tail gets longer, the signal-to-noise ratio gets worse. Thus, the only way a consumer can maintain a consistently good enough signal to find what he or she wants is if the filters get increasingly powerful:

  Why does the signal-to-noise ratio fall as you go down the Tail? Because there’s so much stuff there that what you’re looking for is overshadowed by all the things you aren’t looking for. The reason for this is simple: The vast majority of everything in the world is in the Tail.

  One of the consequences of living in a hit-driven culture is that we tend to assume that hits are a far bigger share of the market than they really are. Instead, they are the rare exception. This is what Nassim Taleb calls the “Black Swan Problem.”

  The phrase comes from David Hume, the eighteenth-century Scottish philosopher, who gave it as an example of the complications that lie in deriving general rules from observed facts. In what has now become known as Hume’s Problem of Induction, he asked how many white swans one need observe before inferring that all swans are white and that there are no black swans. Hundreds? Thousands? We don’t know. (The Black Swan is not just a hypothetical metaphor: Until the discovery of Australia, common belief held that all swans were white. That belief was shattered with the sighting of the first Cygnus atratus.)

  The problem is that we have a hard time putting rare events in context. In any given population there will be a few people who are tremendously rich. Some are smart and some are lucky and we really can’t tell which is which. In Fooled by Randomness, Taleb pokes fun at a bestseller called The Millionaire Next Door, which catalogs the investing tricks and work habits of multimillionaires, so that you can follow them and get rich, too. But as Taleb notes, random factors are just as likely to be responsible for that neighborly millionaire as investing strategies.

  He defines a Black Swan as:

  A random event satisfying the following three properties: large impact, incomputable probabilities, and surprise effect. First, it carries upon its occurrence a disproportionately large impact. Second, its incidence has a small but incomputable probability based on information available prior to its incidence. Third, a vicious property of a Black Swan is its surprise effect: at a given time of observation there is no convincing element pointing to an increased likelihood of the event.

  He could just as easily be describing a blockbuster hit.

  The reality is that the vast majority of content (from music to movies) is not hits. Indeed, the vast majority of content is about as far from a hit as it’s possible to be, counting its audience in hundreds rather than millions. Sometimes that’s because it’s not very good. Sometimes it’s because it wasn’t marketed well or made by people with the right connections. And sometimes it’s because of some random factor that got in the way, which is just as likely as the random factors that sometimes make a blockbuster out of the flimsiest novelty fare (“Who Let the Dogs Out” comes to mind).

  This is simply the natural consequence of what’s called a “powerlaw” distribution, a term for a curve where a small number of things occur with high amplitude (read: sales) and a large number of things occur with low amplitude. A few things sell a lot and a lot of things sell a little. (The phrase comes from the fact that the curve has a 1/x shape, which is the same as x raised to the–1 power.)

  Since most stuff doesn’t sell very well, the volume of the material available—and by extension the volume of stuff you don’t want—rises as the Long Tail falls. Here’s some actual data from the book industry, showing the number of titles in each sales category for 2004:

  The consequence of this is that whatever you are looking for, there’s more stuff you aren’t looking for the farther you go down the Tail. That’s why the signal-to-noise ratio gets worse, despite the fact that you’re often more likely (i.e., if you have access to good search and filters) to find what you want as you go down the Tail. It sounds like a paradox, but it isn’t. It’s just a problem for filters to solve.

  PRE-FILTERS AND POST-FILTERS

  When you think about it, the world is already full of a different kind of filter. In the scarcity-driven markets of limited shelves, screens, and channels that we’ve lived with for most of the past century, entire industries have been created around finding and promoting the good stuff. This is what the A&R talent scouts at the record labels do, along with the Hollywood studio executives and store purchasing managers (“buyers”). In boardrooms around the world, market research teams pore over data that predicts what’s likely to sell and thus deserves to win a valuable spot on the shelf, screen, or page…and what’s unlikely to sell and therefore doesn’t deserve a spot.

  The key word in the preceding paragraph is “predicts.” What’s different about those kinds of filters and the ones I’ve been focusing on is that they filter before things get to market. Indeed, their job is to decide what will make it to market and what won’t. I call them “pre-filters.”

  By contrast, the recommendations and search technologies that I’m writing about are “post-filters.” The post-filters find the best of what’s already out there in their area of interest, elevating the good (i.e., what is relevant, interesting, original, etc.) and downplaying, even ignoring, the bad. When I talk about throwing everything out there and letting the marketplace sort it out, these post-filters are the voice of the marketplace. They channel consumer behavior and amplify it, rather than trying to predict it.

  Here, in table form, are some examples of each:

  The fact that post-filters amplify, rather than predict, behavior is an important distinction. In the existing Short Head markets, where distribution is expensive and shelf space is at a premium, the supply side of the market has to be exceedingly discriminating in what it lets through. These producers, retailers, and marketers have made a science of trying to guess what people will want, to improve their odds of picking winners. Obviously they don’t always guess right. There are surely as many things that deserved to make it to market but were overlooked as there are things that made it to market and then flopped. Nevertheless, the survivors obtain a credible reputation for having some sort of mystical insight into the consumer psyche.

  However, in Long Tail markets—where
distribution is cheap and shelf space is plentiful—the safe bet is to assume that everything is eventually going to be available.

  As such, in Long Tail markets, the role of filter then shifts from gatekeeper to advisor. Rather than predicting taste, post-filters such as Google measure it. Rather than lumping consumers into predetermined demographic and psychographic categories, post-filters such as Netflix’s customer recommendations treat them like individuals who reveal their likes and dislikes through their behavior. Rather than keeping things off the market, post-filters such as MP3 blogs create a market for things that are already available by stimulating demand for them. Jeff Jarvis calls this the difference between “first-person and third-person markets.”

  In general, blogs are shaping up to be a powerful source of influential recommendations. There are independent enthusiast sites such as PVRblog and Horticultural (an organic gardening blog), commercial blogs such as Gizmodo and Joystiq, and then the random recommendations of whichever blogger you happen to read for any reason. (There does seem to be a natural connection between mavens, who know a lot and like to share their knowledge, and blogging.) What they may lack in polish and scope, they more than make up in credibility: Their readers know that there is a real person there that they can trust.

  Of course, just as pre-filters aren’t perfect—e.g., the talent scouts don’t always pick artists that sell records—the same is true of post-filters. Because post-filters tend to be amateurs, oftentimes that means less critical independence and more random malice. Moreover, the problem with post-filtering is that feedback comes after publication, not before. As a result, errors that would have been caught by editors and other wise eyes can sneak through, and even though the collective post-filter feedback can eventually correct them, they may never disappear entirely.