by Kevin Kelly
According to the most recent count I could find, the total number of songs that have been recorded on the planet is 180 million. Using standard MP3 compression, the total volume of recorded music for humans would fit into 720 terabytes. Today 720 terabytes sells for $72,000 and fills a closet. In ten years it will sell for $700 and fit into your pocket. Very soon you’ll be able to carry around all the music of humankind in your pants. On the other hand, if this library is so minuscule, why even bother to carry it around when you could get all music of the world in the cloud streamed to you on demand?
What goes for music also goes for anything and everything that can be rendered in bits. In our lifetime, the entire library of all books, all games, all movies, every text ever printed will be available 24/7 on that same screen thingy or in the same cloud thread. And every day, the library swells. The number of possibilities we confront has been expanded by a growing population, then expanded further by technology that eases creation. There are three times as many people alive today as when I was born (1952). Another billion are due in the next 10 years. An increasing proportion of those extra 5 billion to 6 billion people since my birth have been liberated by the surplus and leisure of modern development to generate new ideas, create new art, make new things. It is 10 times easier today to make a simple video than 10 years ago. It is a hundred times easier to create a small mechanical part and make it real than a century ago. It is a thousand times easier today to write and publish a book than a thousand years ago.
The result is an infinite hall of options. In every direction, countless choices pile up. Despite obsolete occupations like buggy whip maker, the variety of careers to choose from expands. Possible places to vacation, to eat, or even kinds of food all stack up each year. Opportunities to invest explode. Courses to take, things to learn, ways to be entertained explode to astronomical proportions. There is simply not enough time in any lifetime to review the potential of each choice, one by one. It would consume more than a year’s worth of our attention to merely preview all the new things that have been invented or created in the previous 24 hours.
The vastness of the Library of Everything quickly overwhelms the very narrow ruts of our own consuming habits. We’ll need help to navigate through its wilds. Life is short, and there are too many books to read. Someone, or something, has to choose, or whisper in our ear to help us decide. We need a way to triage. Our only choice is to get assistance in making choices. We employ all manner of filtering to winnow the bewildering spread of options. Many of these filters are traditional and still serve well:
We filter by gatekeepers: Authorities, parents, priests, and teachers shield the bad and selectively pass on “the good stuff.”
We filter by intermediates: Sky high is the reject pile in the offices of book publishers, music labels, and movie studios. They say no much more often than yes, performing a filtering function for what gets wide distribution. Every headline in a newspaper is a filter that says yes to this information and ignores the rest.
We filter by curators: Retail stores don’t carry everything, museums don’t show everything, public libraries don’t buy every book. All these curators select their wares and act as filters.
We filter by brands: Faced with a shelf of similar goods, the first-time buyer retreats to a familiar brand because it is a low-effort way to reduce the risk of the purchase. Brands filter through the clutter.
We filter by government: Taboos are prohibited. Hate speech or criticism of leaders or of religion is removed. Nationalistic matters are promoted.
We filter by our cultural environment: Children are fed different messages, different content, different choices depending on the expectations of the schools, family, and society around them.
We filter by our friends: Peers have great sway over our choices. We are very likely to choose what our friends choose.
We filter by ourselves: We make choices based on our own preferences, by our own judgment. Traditionally this is the rarest filter.
None of these methods disappear in the rising superabundance. But to deal with the escalation of options in the coming decades, we’ll invent many more types of filtering.
What if you lived in a world where every great movie, book, and song ever produced was at your fingertips as if “for free,” and your elaborate system of filters had weeded out the crap, the trash, and anything that would remotely bore you. Forget about all the critically acclaimed creations that mean nothing to you personally. Focus instead on just the things that would truly excite you. Your only choices would be the absolute cream of the cream, the things your best friends would recommend, including a few “random” choices to keep you surprised. In other words, you would encounter only things perfectly matched to you at that moment. You still don’t have enough time in your life.
For instance, you could filter your selection of books by reading only the greatest ones. Just focus on the books chosen by experts who have read a lot of them and let them guide you to the 60 volumes considered the best of the very best in Western civilization—the canonical collection known as the Great Books of the Western World. It would take you, or the average reader, some 2,000 hours to completely read all 29 million words. And that’s just the Western world. Most of us are going to need further filtering.
The problem is that we start with so many candidates that, even after filtering out all but one in a million, you still have too many. There are more super great five-stars-for-you movies than you can ever watch in your lifetime. There are more useful tools ideally suited to you than you have time to master. There are more cool websites to linger on than you have attention to spare. There are, in fact, more great bands, and books, and gizmos aimed right at you, customized to your unique desires, than you can absorb, even if it was your full-time job.
Nonetheless, we’ll try to reduce this abundance to a scale that is satisfying. Let’s start with the ideal path. And I’ll make it personal. How would I like to choose what I give my attention to next?
First I’d like to be delivered more of what I know I like. This personal filter already exists. It’s called a recommendation engine. It is in wide use at Amazon, Netflix, Twitter, LinkedIn, Spotify, Beats, and Pandora, among other aggregators. Twitter uses a recommendation system to suggest who I should follow based on whom I already follow. Pandora uses a similar system to recommend what new music I’ll like based on what I already like. Over half of the connections made on LinkedIn arise from their follower recommender. Amazon’s recommendation engine is responsible for the well-known banner that “others who like this item also liked this next item.” Netflix uses the same to recommend movies for me. Clever algorithms churn through a massive history of everyone’s behavior in order to closely predict my own behavior. Their guess is partly based on my own past behavior, so Amazon’s banner should really say, “Based on your own history and the history of others similar to you, you should like this.” The suggestions are highly tuned to what I have bought and even thought about buying before (they track how long I dwell on a page deliberating, even if I don’t choose it). Computing the similarities among a billion past purchases enables their predictions to be remarkably prescient.
These recommendation filters are one of my chief discovery mechanisms. I find them far more reliable, on average, than recommendations from experts or friends. In fact, so many people find these filtered recommendations useful that these kinds of “more like this” offers are responsible for a third of Amazon sales—a difference amounting to about $30 billion in 2014. They are so valuable to Netflix that it has 300 people working on its recommendation system, with a budget of $150 million. There are of course no humans involved in guiding these filters once they are operational. The cognification is based on subtle details of my (and others’) behavior that only a sleepless obsessive machine might notice.
The danger of being rewarded with only what you already like, however, is that you can spin into an
egotistical spiral, becoming blind to anything slightly different, even if you’d love it. This is called a filter bubble. The technical term is “overfitting.” You get stuck at a lower than optimal peak because you behave as if you have arrived at the top, ignoring the adjacent environment. There’s a lot of evidence this occurs in the political realm as well: Readers of one political stripe who depend only on a simple filter of “more like this” rarely if ever read books outside their stripe. This overfitting tends to harden their minds. This kind of filter-induced self-reinforcement also occurs in science, the arts, and culture at large. The more effective the “more good stuff like this” filter is, the more important it becomes to alloy it with other types of filters. For instance, some researchers from Yahoo! engineered a way to automatically map one’s position in the field of choices visually, to make the bubble visible, which made it easier for someone to climb out of their filter bubble by making small tweaks in certain directions.
Second in the ideal approach, I’d like to know what my friends like that I don’t know about. In many ways, Twitter and Facebook serve up this filter. By following your friends, you get effortless updates on the things they find cool enough to share. The ease of shouting out a recommendation via a text or photo is so easy from a phone that we are surprised when someone loves something new but doesn’t share it. But friends can also act like a filter bubble if they are too much like you. Close friends can make an echo chamber, amplifying the same choices. Studies show that going to the next circle, to friends of friends, is sometimes enough to enlarge the range of options away from the expected.
A third component in the ideal filter would be a stream that suggested stuff that I don’t like but would like to like. It’s a bit similar to me trying a least favorite cheese or vegetable every now and then just to see if my tastes have changed. I am sure I don’t like opera, but a few years ago I again tried one—Carmen at the Met—teleprojected real time in a cinema with prominent subtitles on the huge screen, and I was glad I went. A filter dedicated to probing one’s dislikes would have to be delicate, but could also build on the powers of large collaborative databases in the spirit of “people who disliked those, learned to like this one.” In somewhat the same vein I also, occasionally, want a bit of stuff I dislike but should learn to like. For me that might be anything related to nutritional supplements, details of political legislation, or hip-hop music. Great teachers have a knack for conveying unsavory packages to the unwilling in a way that does not scare them off; great filters can too. But would anyone sign up for such a filter?
Right now, no one signs up for any of these filters because filters are primarily installed by platforms. The 200 average friends of your average Facebook member already post such a torrent of updates that Facebook feels it must cut, edit, clip, and filter your news to a more manageable stream. You do not see all the posts your friends make. Which ones have been filtered out? By what criteria? Only Facebook knows, and it considers the formulas trade secrets. What it is optimizing for is not even communicated. The company talks about increasing the satisfaction of members, but a fair guess is that it is filtering your news stream to optimize the amount of time you spend on Facebook—a much easier thing to measure than your happiness. But that may not be what you want to optimize Facebook for.
Amazon uses filters to optimize for maximum sales, and that includes filtering the content on the pages you see. Not just what items are recommended, but the other material that appears on the page, including bargains, offers, messages, and suggestions. Like Facebook, Amazon performs thousands of experiments a day, altering their filters to test A over B, trying to personalize the content in response to actual use by millions of customers. They fine-tune the small things, but at such a scale (a hundred thousand subjects at a time) that their results are extremely useful. As a customer I keep returning to Amazon because it is trying to maximize the same thing I am: cheap access to things I will like. That alignment is not always present, but when it is, we return.
Google is the foremost filterer in the world, making all kinds of sophisticated judgments about what search results you see. In addition to filtering the web, it processes 35 billion emails a day, filtering out spam very effectively, assigning labels and priorities. Google is the world’s largest collaborative filter, with thousands of interdependent dynamic sieves. If you opt in, it personalizes search results for you and will customize them for your exact location at the time you ask. It uses the now proven principles of collaborative filtering: People who found this answer valuable also found this next one good too (although they don’t label it that way). Google filters the content of 60 trillion pages about 2 million times every minute, but we don’t often question how it recommends. When I ask it a query, should it show me the most popular, or the most trusted, or the most unique, or the options most likely to please me? I don’t know. I say to myself I’d probably like to have the choice to rank results each of those four different ways, but Google knows that all I’d do is look at the first few results and then click. So they say, “Here’s the top few we think are the best based on our deep experience in answering 3 billion questions a day.” So I click. Google is trying to optimize the chance I’ll return to ask it again.
As they mature, filtering systems will be extended to other decentralized systems beyond media, to services like Uber and Airbnb. Your personal preferences in hotel style, status, and service can easily be ported to another system in order to increase your satisfaction when you are matched to a room in Venice. Heavily cognified, incredibly smart filters can be applied to any realm with a lot of choices—which will be more and more realms. Anywhere we want personalization, filtering will follow.
Twenty years ago many pundits anticipated the immediate arrival of large-scale personalization. A 1992 book called Mass Customization by Joseph Pine laid out the plan. It seemed reasonable that custom-made work—which was once the purview of the rich—could be widened to the middle class with the right technology. For instance, an ingenious system of digital scans and robotic flexible manufacturing could provide personally tailored shirts for the middle class, instead of just bespoke shirts for the gentry. A few startups tried to execute “mass customization” for jeans, shirts, and baby dolls in the late 1990s, but they failed to catch on. The main hurdle was that, except in trivial ways (choosing a color or length), it was very difficult to capture or produce significant uniqueness without raising prices to the luxury level. The vision was too far ahead of the technology. But now the technology is catching up. The latest generation of robots are capable of agile manufacturing, and advanced 3-D printers can rapidly produce units of one. Ubiquitous tracking, interacting, and filtering means that we can cheaply assemble a multidimensional profile of ourselves, which can guide any custom services we desire.
Here is a picture of where this force is taking us. My day in the near future will entail routines like this: I have a pill-making machine in my kitchen, a bit smaller than a toaster. It stores dozens of tiny bottles inside, each containing a prescribed medicine or supplement in powdered form. Every day the machine mixes the right doses of all the powders and stuffs them all into a single personalized pill (or two), which I take. During the day my biological vitals are tracked with wearable sensors so that the effect of the medicine is measured hourly and then sent to the cloud for analysis. The next day the dosage of the medicines is adjusted based on the past 24-hour results and a new personalized pill produced. Repeat every day thereafter. This appliance, manufactured in the millions, produces mass personalized medicine.
My personal avatar is stored online, accessible to any retailer. It holds the exact measurements of every part and curve of my body. Even if I go to a physical retail store, I still try on each item in a virtual dressing room before I go because stores carry only the most basic colors and designs. With the virtual mirror I get a surprisingly realistic preview of what the clothes will look like on me; in fact, because I can spin my simulated dr
essed self around, it is more revealing than a real mirror in a dressing room. (It could be better in predicting how comfortable the new clothes feel, though.) My clothing is custom fit based on the specifications (tweaked over time) from my avatar. My clothing service generates new variations of styles based on what I’ve worn in the past, or on what I spend the most time wishfully gazing at, or on what my closest friends have worn. It is filtering styles. Over years I have trained an in-depth profile of my behavior, which I can apply to anything I desire.
My profile, like my avatar, is managed by Universal You. It knows that I like to book inexpensive hostels when I travel on vacation, but with a private bath, maximum bandwidth, and always in the oldest part of the town, except if it is near a bus station. It works with an AI to match, schedule, and reserve the best rates. It is more than a mere stored profile; rather it is an ongoing filter that is constantly adapting to wherever I have already gone, what kind of snapshots and tweets I made about past visits, and it weighs my new interests in reading and movies since books and movies are often a source for travel desires. It pays a lot of attention to the travels of my best friends and their friends, and from that large pool of data often suggests specific restaurants and hostels to visit. I generally am delighted by its recommendations.
Because my friends let Universal You track their shopping, eating out, club attendance, movie streaming, news screening, exercise routines, and weekend excursions, it can make very detailed recommendations for me—with minimal effort on their part. When I wake in the morning, Universal filters through my update stream to deliver the most vital news of the type I like in the morning. It filters based on the kinds of things I usually forward to others, or bookmark, or reply to. In my cupboard I find a new kind of cereal with saturated nutrition that my friends are trying this week, so Universal ordered it for me yesterday. It’s not bad. My car service notices where the traffic jams are this morning, so it schedules my car later than normal and it will try an unconventional route to the place I’ll work today, based on several colleagues’ commutes earlier. I never know for sure where my office will be since our startup meets in whatever coworking space is available that day. My personal device turns the space’s screens into my screen. My work during the day entails tweaking several AIs that match doctoring and health styles with clients. My job is to help the AIs understand some of the outlier cases (such as folks with faith-healing tendencies) in order to increase the effectiveness of the AIs’ diagnoses and recommendations.