Big Data: A Revolution That Will Transform How We Live, Work, and Think

by Viktor Mayer-Schonberger

  [>] Anderson’s backpedal—National Public Radio, “Search and Destroy,” July 18, 2008 (http://www.onthemedia.org/2008/jul/18/search-and-destroy/transcript/).

  [>] On choices influencing our analysis—danah boyd and Kate Crawford. “Six Provocations for Big Data,” paper presented at Oxford Internet Institute’s “A Decade in Internet Time: Symposium on the Dynamics of the Internet and Society,” September 21, 2011 (http://ssrn.com/abstract=1926431).

  5. Datafication

  [>] Details of Maury’s life compiled from numerous works by and about him. They include Chester G. Hearn, Tracks in the Sea: Matthew Fontaine Maury and the Mapping of the Oceans (International Marine/McGraw-Hill, 2002); Janice Beaty, Seeker of Seaways: A Life of Matthew Fontaine Maury, Pioneer Oceanographer (Pantheon Books, 1966); Charles Lee Lewis, Matthew Fontaine Maury: The Pathfinder of the Seas (U.S. Naval Institute, 1927) (http://archive.org/details/matthewfontainem00lewi); and Matthew Fontaine Maury, The Physical Geography of the Sea (Harper, 1855).

  [>] Maury quotations—From Maury, Physical Geography of the Sea, “Introduction,” pp. xii, vi.

  [>] Car seat data—Nikkei, “Car Seat of Near Future IDs Driver’s Backside,” December 14, 2011.

  [>] Quantifying the world—Much of the authors’ thinking on the history of datafication has been inspired by Crosby, The Measure of Reality.

  [>] Europeans were never exposed to abacuses—Ibid., 112.

  Calculating faster using Arabic numerals—Alexander Murray, Reason and Society in the Middle Ages (Oxford University Press, 1978), p. 166.

  [>] Total number of books published and Harvard study on Google book-scanning project—Jean-Baptiste Michel et al., “Quantitative Analysis of Culture Using Millions of Digitized Books,” Science 331 (January 14, 2011), pp. 176–182 (http://www.sciencemag.org/content/331/6014/176.abstract). For a video lecture on the paper, see Erez Lieberman Aiden and Jean-Baptiste Michel, “What We Learned from 5 Million Books,” TEDx, Cambridge, MA, 2011 (http://www.ted.com/talks/what_we_learned_from_5_million_books.html).

  [>] On wireless modules in cars and insurance—See Cukier, “Data, Data Everywhere.”

  UPS’s Jack Levis—Interview with Cukier, April 2012.

  Data on UPS’s savings—Institute for Operations Research and the Management Sciences (INFORMS), “UPS Wins Gartner BI Excellence Award,” 2011 (http://www.informs.org/Announcements/UPS-wins-Gartner-BI-Excellence-Award).

  [>] Pentland research—Robert Lee Hotz, “The Really Smart Phone,” Wall Street Journal, April 22, 2011 (http://online.wsj.com/article/SB10001424052748704547604576263261679848814.html).

  [>] Eagle’s study of slums—Nathan Eagle, “Big Data, Global Development, and Complex Systems,” Santa Fe Institute, May 5, 2010 (http://www.youtube.com/watch?v=yaivtqlu7iM). Also, interview with Cukier, October 2012.

  [>] Facebook data—From Facebook IPO Prospectus, 2012.

  Twitter data—Alexia Tsotsis, “Twitter Is at 250 Million Tweets per Day, iOS 5 Integration Made Signups Increase 3x,” TechCrunch, October 17, 2011, http://techcrunch.com/2011/10/17/twitter-is-at-250-million-tweets-per-day/.

  Hedge funds using twitter—Kenneth Cukier, “Tracking Social Media: The Mood of the Market,” Economist.com, June 28, 2012 (http://www.economist.com/blogs/graphicdetail/2012/06/tracking-social-media).

  [>] Twitter and forecasting Hollywood box-office revenue—Sitaram Asur and Bernardo A. Huberman, “Predicting the Future with Social Media,” Proceedings of the 2010 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology, pp. 492–499; online at http://www.hpl.hp.com/research/scl/papers/socialmedia/socialmedia.pdf. Twitter and global moods—Scott A. Golder and Michael W. Macy, “Diurnal and Seasonal Mood Vary with Work, Sleep, and Daylength Across Diverse Cultures,” Science 333 (September 30, 2011), pp. 1878–81.

  Twitter and flu shots—Marcel Salathé and Shashank Khandelwal, “Assessing Vaccination Sentiments with Online Social Media: Implications for Infectious Disease Dynamics and Control,” PLoS Computational Biology, October 2011.

  [>] IBM’s “smart floor” patent—Lydia Mai Do, Travis M. Grigsby, Pamela Ann Nesbitt, and Lisa Anne Seacat. “Securing premises using surfaced-based computing technology,” U.S. Patent number: 8138882. Issue date: March 20, 2012.

  The quantified-self movement—“Counting Every Moment,” The Economist, March 3, 2012.

  Apple earbuds for bio-measurements—Jesse Lee Dorogusker, Anthony Fadell, Donald J. Novotney, and Nicholas R Kalayjian, “Integrated Sensors for Tracking Performance Metrics,” U.S. Patent Application 20090287067. Assignee: Apple. Application Date: 2009-07-23. Publication Date: 2009-11-19.

  Derawi Biometrics, “Your Walk Is Your PIN-Code,” press release, February 21, 2011 (http://biometrics.derawi.com/?p=175).

  iTrem information—See the iTrem project page of the Landmarc Research Center at Georgia Tech (http://eosl.gtri.gatech.edu/Capabilities/LandmarcResearchCenter/LandmarcProjects/iTrem/tabid/798/Default.aspx) and email exchange.

  Kyoto researchers on tri-axial accelerometers—iMedicalApps Team, “Gait Analysis Accuracy: Android App Comparable to Standard Accelerometer Methodology,” mHealth, March 23, 2012.

  [>] Newspapers gave rise to the nation state—Benedict Anderson, Imagined Communities: Reflections on the Origin and Spread of Nationalism (Verso, 2006).

  Physicists suggest information is the basis of everything—Hans Christian von Baeyer, Information: The New Language of Science (Harvard University Press, 2005).

  6. Value

  [>] Story of Luis von Ahn—Based on Cukier interviews with von Ahn from 2010. See also Clive Thompson, “For Certain Tasks, the Cortex Still Beats the CPU,” Wired, June 25, 2007 (http://www.wired.com/techbiz/it/magazine/15-07/ff_humancomp?currentPage=all); Jessie Scanlon, “Luis von Ahn: The Pioneer of ‘Human Computation,’” Businessweek, November 3, 2008 (http://www.businessweek.com/stories/2008-11-03/luis-von-ahn-the-pioneer-of-human-computation-businessweek-business-news-stock-market-and-financial-advice). His technical description of reCaptchas is at Luis von Ahn et al., “reCAPTCHA: Human-Based Character Recognition via Web Security Measures,” Science 321 (September 12, 2008), pp. 1465–68 (http://www.sciencemag.org/content/321/5895/1465.abstract).

  [>] Smith’s pin factory—Adam Smith, The Wealth of Nations (reprint, Bantam Classics, 2003), book I, chapter one. (A free electronic version is at http://www2.hn.psu.edu/faculty/jmanis/adam-smith/Wealth-Nations.pdf).

  [>] Storage—Viktor Mayer-Schönberger, Delete: The Virtue of Forgetting in the Digital Age (Princeton University Press, 2011), p. 63.

  [>] On electrical cars’ power usage—IBM, “IBM, Honda, and PG&E Enable Smarter Charging for Electric Vehicles,” press release, April 12, 2012 (http://www-03.ibm.com/press/us/en/pressrelease/37398.wss). Also see Clay Luthy, “Guest Perspective: IBM Working with PG&E to Maximize the EV Potential” PGE Currents Magazine, April 13, 2012 (http://www.pgecurrents.com/2012/04/13/ibm-working-with-pge-to-maximize-the-ev-potential).

  [>] Amazon and AOL’s data—Cukier interview with Andreas Weigend, 2010 and 2012.

  Nuance software and Google—Cukier, “Data, Data Everywhere.”

  [>] Logistics company—Brad Brown, Michael Chui, and James Manyika, “Are You Ready for the Era of ‘Big Data’?” McKinsey Quarterly, October 2011, p. 10.

  [>] Telefonica monetizes mobile information—“Telefonica Hopes ‘Big Data’ Arm Will Revive Fortunes,” BBC Online, October 9, 2012. (http://www.bbc.co.uk/news/technology-19882647).

  Danish Cancer Society study—Patrizia Frei et al., “Use of Mobile Phones and Risk of Brain Tumours: Update of Danish Cohort Study,” BMJ 343 (2011) (http://www.bmj.com/content/343/bmj.d6387), and interview with Cukier, October 2012.

  [>] Google’s Street View’s GPS records and self-driving car—Peter Kirwan, “This Car Drives Itself,” Wired UK, January 2012 (http://www.wired.co.uk/magazine/archive/2012/01/features/this-car-drives-itself?page=all).

  [>] Google’s spell check and quotation—Interv
iew with Cukier at the Googleplex in Mountain View, California, December 2009; some material also appeared in Cukier, “Data, Data Everywhere.”

  [>] Hammerbacher’s insight—Interview with Cukier, October 2012.

  Barnes & Noble e-book data—Alexandra Alter, “Your E-Book Is Reading You,” Wall Street Journal, June 29, 2012 (http://online.wsj.com/article/SB10001424052702304870304577490950051438304.html).

  [>] Andrew Ng’s Coursera class and data—Interview with Cukier, June 2012.

  [>] Obama’s open government policy—Barack Obama, “Presidential memorandum,” White House, January 21, 2009.

  [>] On Facebook’s data’s worth—For an excellent examination of the discrepancy between market and book value for Facebook’s IPO, see Doug Laney, “To Facebook You’re Worth $80.95,” Wall Street Journal, May 3, 2012 (http://blogs.wsj.com/cio/2012/05/03/to-facebook-youre-worth-80-95/). For valuing Facebook’s discrete items, Laney extrapolated from Facebook’s growth to estimate the 2.1 trillion pieces of content. In his WSJ article he valued the items at three cents each since he was using an earlier Facebook market valuation estimate of $75 billion. In the end, it was over $100 billion, or five cents, as we extrapolated ourselves based on his calculation. Value gap of physical assets and intangible ones—Steve M. Samek, “Prepared Testimony: Hearing on Adapting a 1930’s Financial Reporting Model to the 21st Century,” U.S. Senate Committee on Banking, Housing and Urban Affairs, Subcommittee on Securities, July 19, 2000.

  Value of intangibles—Robert S. Kaplan and David P. Norton, Strategy Maps: Converting Intangible Assets into Tangible Outcomes (Harvard Business Review Press, 2004), pp. 4–5.

  [>] Tim O’Reilly quotation—Interview with Cukier, February 2011.

  7. Implications

  [>] Info on Decide.com—Cukier email exchange with Etzioni, May 2012.

  [>] McKinsey report—James Manyika et al., “Big Data: The Next Frontier for Innovation, Competition, and Productivity,” McKinsey Global Institute, May 2011 (http://www.mckinsey.com/insights/mgi/research/technology_and_innovation/big_data_the_next_frontier_for_innovation), p. 10.

  Hal Varian quotation—Interview with Cukier, December 2009.

  [>] Carl de Marcken quotation—Email exchange with Cukier, May 2012.

  [>] On MasterCard Advisors—Cukier interview with Gary Kearns, a MasterCard Advisors executive, at The Economist’s “The Ideas Economy: Information” conference, Santa Clara, California, June 8, 2011.

  [>] Accenture and city of St. Louis, Missouri—Cukier interview with municipal employees, February 2007.

  Microsoft Amalga Unified Intelligence System—“Microsoft Expands Presence in Healthcare IT Industry with Acquisition of Health Intelligence Software Azyxxi,” Microsoft press release, July 26, 2006 (http://www.microsoft.com/en-us/news/press/2006/jul06/07-26azyxxiacquisitionpr.aspx). The Amalga service is now a part of Microsoft’s joint venture with General Electric called Caradigm.

  [>] Bradford Cross—Interviews with Cukier, March-October 2012.

  [>] Amazon and “collaborative filtering”—IPO Prospectus, May 1997 (http://www.sec.gov/Archives/edgar/data/1018724/0000891020-97-000868.txt).

  [>] Car’s microprocessors—Nick Valery, “Tech.View: Cars and Software Bugs,” Economist.com, May 16, 2010 (http://www.economist.com/blogs/babbage/2010/05/techview_cars_and_software_bugs).

  Maury called ships “floating observatories”—Maury, The Physical Geography of the Sea.

  [>] Inrix—Cukier interview with executives, May and September, 2012.

  [>] On Health Care Cost Institute—Sarah Kliff, “A Database That Could Revolutionize Health Care,” Washington Post, May 21, 2012.

  Decide.com’s data-usage agreement—Cukier email exchange with Etzioni, May 2012.

  Google and ITA deal—Claire Cain Miller, “U.S. Clears Google Acquisition of Travel Software,” New York Times, April 8, 2011 (http://www.nytimes.com/2011/04/09/technology/09google.html?_r=0).

  [>] Inrix and ABS—Cukier interview with Inrix executives, May 2012.

  [>] Roadnet story and Len Kennedy quotation—Interview with Cukier, May 2012.

  Dialogue from film Moneyball, directed by Bennett Miller, Columbia Pictures, 2011.

  [>] McGregor’s data amounting to more than a decade of patient-years—Interview with Cukier, May 2012. Goldbloom quotation—Interview with Cukier, March 2012.

  [>] On Hollywood box office versus video game sales—For movies, see Brooks Barnes, “A Year of Disappointment at the Movie Box Office,” New York Times, December 25, 2011 (http://www.nytimes.com/2011/12/26/business/media/a-year-of-disappointment-for-hollywood.html). For video games, see “Factbox: A Look at the $65 billion Video Games Industry,” Reuters, June 6, 2011 (http://uk.reuters.com/article/2011/06/06/us-videogames-factbox-idUKTRE75552I20110606).

  Zynga data analytics—Nick Wingfield, “Virtual Products, Real Profits: Players Spend on Zynga’s Games, but Quality Turns Some Off,” Wall Street Journal, September 9, 2011 (http://online.wsj.com/article/SB10001424053111904823804576502442835413446.html).

  [>] Ken Rudin quotation—From interview of Rudin by Niko Waesche, cited in Erik Schlie, Jörg Rheinboldt, and Niko Waesche, Simply Seven: Seven Ways to Create a Sustainable Internet Business (Palgrave Macmillan, 2011). p. 7.

  Auden quotation—W. H. Auden, “For the Time Being,” 1944. Thomas Davenport quotation—Cukier interview with Davenport, December 2009.

  The-Numbers.com—Cukier interviews with Bruce Nash, October 2011 and July 2012.

  [>] Brynjolfsson study—Erik Brynjolfsson, Lorin Hitt, and Heekyung Kim, “Strength in Numbers: How Does Data-Driven Decisionmaking Affect Firm Performance?” working paper, April 2011 (http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1819486).

  [>] On Rolls-Royce—See “Rolls-Royce: Britain’s Lonely High-Flier,” The Economist, January 8, 2009 (http://www.economist.com/node/12887368). Figures updated from press office, November 2012.

  Erik Brynjolfsson, Andrew McAfee, Michael Sorell, and Feng Zhu, “Scale Without Mass: Business Process Replication and Industry Dynamics,” Harvard Business School working paper, September 2006 (http://www.hbs.edu/research/pdf/07-016.pdf also http://hbswk.hbs.edu/item/5532.html).

  [>] On the movement toward increasingly large data holders—See also Yannis Bakos and Erik Brynjolfsson, “Bundling Information Goods: Pricing, Profits, and Efficiency,” Management Science 45 (December 1999), pp. 1613–30.

  [>] Philip Evans—Interviews with the authors, 2011 and 2012.

  8. Risks

  [>] On the Stasi—Much of the literature unfortunately is in German, but one well researched exception is Kristie Macrakis, Seduced by Secrets: Inside the Stasi’s Spy-Tech World (Cambridge University Press, 2008); a very personal story is shared in Timothy Garton Ash, The File (Atlantic Books, 2008). We also recommend the Academy Award–winning movie The Lives of Others, directed by Florian Henckel von Donnersmark, Buena Vista/Sony Pictures, 2006.

  Surveillance cameras near Orwell’s home—“George Orwell, Big Brother Is Watching Your House,” The Evening Standard, March 31, 2007 (http://www.thisislondon.co.uk/news/george-orwell-big-brother-is-watching-your-house-7086271.html).

  On Equifax and Experian—Daniel J. Solove, The Digital Person: Technology and Privacy in the Information Age (NYU Press, 2004), pp. 20–21.

  [>] On block addresses of Japanese in Washington handed over to U.S. authorities—J. R. Minkel, “The U.S. Census Bureau Gave Up Names of Japanese-Americans in WW II,” Scientific American, March 30, 2007 (http://www.scientificamerican.com/article.cfm?id=confirmed-the-us-census-b).

  On data used by Nazis in the Netherlands—William Seltzer and Margo Anderson, “The Dark Side of Numbers: The Role of Population Data Systems in Human Rights Abuses,” Social Research 68 (2001), pp. 481–513.

  [>] On IBM and the Holocaust—Edwin Black, IBM and the Holocaust (Crown, 2003).

  On the amount of data smart meters collect—See Elias Leake Quinn, “Smart Metering and Privacy: Existing Law and Competing Policies;
A Report for the Colorado Public Utility Commission,” Spring 2009 (http://www.w4ar.com/Danger_of_Smart_Meters_Colorado_Report.pdf). See also Joel M. Margolis, “When Smart Grids Grow Smart Enough to Solve Crimes,” Neustar, March 18, 2010 (http://energy.gov/sites/prod/files/gc prod/documents/Neustar_Comments_DataExhibitA.pdf)

  [>] Fred Cate on notice and consent—Fred H. Cate, “The Failure of Fair Information Practice Principles,” in Jane K. Winn, ed., Consumer Protection in the Age of the “Information Economy” (Ashgate, 2006), p. 341 et seq.

  [>] On the AOL data release—Michael Barbaro and Tom Zeller Jr., “A Face Is Exposed for AOL Searcher No. 4417749,” New York Times, August 9, 2006.

  Also see Matthew Karnitschnig and Mylene Mangalindan, “AOL Fires Technology Chief After Web-Search Data Scandal,” Wall Street Journal, August 21, 2006.

  [>] Netflix identified individual—Ryan Singel, “Netflix Spilled Your Brokeback Mountain Secret, Lawsuit Claims,” Wired, December 17, 2009 (http://www.wired.com/threatlevel/2009/12/netflix-privacy-lawsuit/).

  On the Netflix data release—Arvind Narayanan and Vitaly Shmatikov, “Robust De-Anonymization of Large Sparse Datasets,” Proceedings of the 2008 IEEE Symposium on Security and Privacy, p. 111 et seq. (http://www.cs.utexas.edu/~shmat/shmat_oak08netflix.pdf); Arvind Narayanan and Vitaly Shmatikov, “How to Break the Anonymity of the Netflix Prize Dataset,” October 18, 2006, arXiv:cs/0610105 [cs.CR] (http://arxiv.org/abs/cs/0610105).

  On identifying people from three characteristics—Philippe Golle, “Revisiting the Uniqueness of Simple Demographics in the US Population,” Association for Computing Machinery Workshop on Privacy in Electronic Society 5 (2006), p. 77.

  On the structural weakness of anonymization—Paul Ohm, “Broken Promises of Privacy: Responding to the Surprising Failure of Anonymization,” 57 UCLA Law Review 1701 (2010).