by Amy Webb
Ultimately, with the Big Nine’s blessing and encouragement, GAIA determines that with human safety and security in the balance, new restrictions must be built into all AGIs to limit their rate of self-improvement and to ensure that no unwanted mutations can be implemented. Soon, GAIA will deploy a series of guardian AIs that will act as an early warning system for any AGI that’s gained too much cognitive power. Even the guardians won’t necessarily prevent a rogue actor from trying to create ASIs on their own, but GAIA is writing scenarios to prepare for that eventuality. In GAIA, and in the Big Nine, we place our unwavering affection and trust.
CHAPTER SIX
LEARNING TO LIVE WITH MILLIONS OF PAPER CUTS: THE PRAGMATIC SCENARIO
By 2023, we’ve acknowledged AI’s problems but along the way decided to make only minor tweaks in the developmental track of artificial intelligence, a system that we can all see is clearly fractured. We pursue only tweaks because AI’s stakeholders aren’t willing to get uncomfortable: to sacrifice financial gains, make politically unpopular choices, and curb our wild expectations in the short-term, even if it means improving our long-term odds of living alongside AI. Worse, we ignore China and its plans for the future.
Leaders in Congress, our various federal agencies, and the White House continue to deprioritize artificial intelligence and advanced scientific research in general, preferring to invest in industries that are politically appealing but nearing obsolescence. A strategic plan for the future of AI published by the Obama administration in 2016—a document that heavily influenced China’s own 2025 strategic plan—is shelved, along with the federally funded AI R&D program it recommended. America has no long-term vision or strategy on AI, and it disavows any economic, educational, and national security impacts. US government leaders, on both sides of the aisle, focus on how to stifle China when they should be strategizing on how to establish a coalition made up of the G-MAFIA and government.
The absence of a coalition and coherent national AI strategy foment paper cuts—millions and millions of them—which over time start to bleed. We don’t notice at first. Because popular culture, evocative stories by tech journalists, and social media posts by influencers have trained us to be on the lookout for big, obvious signposts—like killer robots—we miss the real signposts, small and scattershot as they may seem, as AI evolves. The Big Nine are forced to prioritize speed over safety, so AI’s developmental track—from ANI to AGI and beyond—pushes ahead without first resolving serious technical vulnerabilities. Here are a few of the less obvious paper cuts—many self-inflicted—that we are not treating as the serious wounds they are in the present.
As consumers of technology, our expectation is that AI’s tribes will have already imagined and solved every problem before any new apps, products, or services leave the R&D labs. We have been habituated to adopting technology that works right out of the box. When we purchase new smartphones and TVs, we plug them in and they function as promised. When we download new software, whether it’s for word processing or data analytics, it behaves as anticipated. We forget that AI is not technology that works out of the box, because in order for it to function as we want it to, an AI system needs vast amounts of data and an opportunity to learn in real time.
None of us—not individual consumers, journalists, or analysts—give the Big Nine any room for error. We demand new products, services, patents, and research breakthroughs on a regular cycle, or we register our complaints publicly. It doesn’t matter to us that our demands are distracting AI’s tribes from doing better work.
AI models and frameworks, regardless of how large or small, need lots of data in order to learn, improve, and get deployed. Data is analogous to our world’s oceans. It surrounds us, is an endless resource, and remains totally useless to us unless we desalinate it, treating and processing it for consumption. At the moment, there are just a few companies that can effectively desalinate data at a scale that matters. That’s why the most challenging part of building a new AI system isn’t the algorithms or the models but rather collecting the right data and labeling it correctly so that a machine can begin training with and learning from it. Relative to the various products and services the Big Nine are breathlessly working to build, there are very few data sets ready to be used. A few of these are ImageNet (the enormous data set of images that’s used widely), WikiText (a language modeling data set using Wikipedia articles), 2000 HUB5 English (an English-only data set used for speech), and LibriSpeech (about 500 hours of audiobook snippets). If you wanted to build a health AI to spot anomalies in blood work and oncology scans, the problem isn’t the AI, it’s data—humans are complicated, our bodies have tons of possible variants, and there isn’t a big enough data set ready to be deployed.
A decade ago, in the early 2010s, the IBM Watson Health team partnered with different hospitals to see if its AI could supplement the work of doctors. Watson Health had some stunning early wins, including a case involving a very sick nine-year-old boy. After specialists weren’t able to diagnose and treat him, Watson assigned a probability to possible health issues—the list included common ailments as well as outliers, including a rare childhood illness called Kawasaki disease. Once word got out that Watson was performing miracle diagnoses and saving peoples’ lives, the Watson team was under pressure to commercialize and sell the platform, and incomprehensibly unrealistic targets were set. IBM projected that Watson Health would grow from a $244 million business in 2015 to a $5 billion business by 2020.1 That was an anticipated 1,949% growth in under five years.
Before Watson Health could reproduce the same magic it had shown earlier—following a whiplash-inducing development timeline, no less—it would need significantly more training data and time to learn. But there wasn’t enough real-world health data available, and what was available to train the system wasn’t nearly comprehensive enough. That’s because patient data was locked up in electronic health-record software systems managed by another company, which saw IBM as a competitor.
As a result, the IBM team used a workaround common among AI’s tribes. It had fed Watson Health what’s called “synthetic data,” which is data that represents hypothetical information. Since researchers can’t just scrape and load “ocean data” into a machine-learning system for training, they will buy a synthetic data set from a third party or build one themselves. This is often problematic because composing that data set—what goes into it and how it’s labeled—is rife with decisions made by a small number of people who often aren’t aware of their professional, political, gender, and many other cognitive biases.
Outsized expectations for Watson Health’s immediate profitability, combined with a reliance on synthetic data sets, is what led to a serious problem. IBM had partnered with Memorial Sloan Kettering Cancer Center to apply Watson Health’s skills to cancer treatment. Not long after, a few medical experts working on the project reported examples of unsafe and incorrect treatment recommendations. In one example, Watson Health recommended a bizarre treatment protocol for a patient diagnosed with lung cancer who also showed signs of bleeding: chemotherapy and a drug called bevacizumab, a contraindicated drug because it can cause severe or fatal hemorrhaging.2 The story of Watson’s ineptitude made the rounds in medical and hospital industry publications and on techie blogs, often with sensational headlines. Yet the problem wasn’t that Watson Health had it out for humans—but rather that market forces had pressured IBM to rush its AI research to make good on projections.
Here’s another paper cut: Some AIs have figured out how to hack and game their own systems. If an AI is specifically programmed to learn a game, play it, and do whatever necessary to win, researchers have discovered cases of “reward hacking,” where a system will exploit evolutionary and machine-learning algorithms to win using trickery and deception. For example, an AI learning to play Tetris figured out that it could simply pause the game forever so that it could never lose. Ever since you first read about reward hacking—it made headlines recently when two financial
AI systems predicted a precipitous drop in stock market values and attempted to autonomously close markets indefinitely—you’ve been wondering what might happen if your data got caught up in a reward-hacking system. That winter vacation you have coming up—what if the air traffic control wound up locked?
Another paper cut: malicious actors can inject poisonous data into AI’s training programs. Neural networks are vulnerable to “adversarial examples,” which are fake or intentionally designed with wrong information to cause an AI system to make a mistake. An AI system might label a picture as a panda, with 60% confidence; but add just a tiny bit of noise to the image, like a few pixels out of place that would be imperceptible to a human, and the system will relabel the image a gibbon with 99% confidence. It’s possible to train a car’s computer vision to think that a stop sign actually means “speed limit 100” and send its passengers careening at top speed through an intersection. Adversarial inputs could retrain a military AI system to interpret all of the visual data found outside a typical hospital—such as ambulances or the words “emergency” and “hospital” on signs—as terrorist markers. The problem is that the Big Nine haven’t figured out how to safeguard their systems from adversarial examples, either in the digital or physical worlds.
A deeper cut: the Big Nine know that adversarial information can actually be used to reprogram machine-learning systems and neural networks. A team within Google’s Brain division published a paper in 2018 on how a bad actor could inject adversarial information into a computer vision database and effectively reprogram all the AI systems that learn from it.3 Hackers could someday embed poisonous data in your smart earphones and reprogram them with someone else’s identity simply by playing adversarial noise while sitting next to you on the train.
What complicates things is that sometimes adversarial information can be useful. A different Google Brain team discovered that adversarial information could also be used to generate new information that can be put to good use in what’s called a “generative adversarial network,” or GAN. In essence, it’s the Turing test but without any humans involved. Two AIs are trained on the same data—such as images of people. The first one creates photos of, say, North Korean dictator Kim Jong-un that seem realistic, while the second AI compares the generated photos with real ones of him. Based on the judgment of the second AI, the first one goes back and makes tweaks to its process. This happens again and again, until the first AI is automatically generating all kinds of images of Kim Jong-un that look entirely realistic, but never actually happened in the real world. Pictures that show Kim Jong-un having dinner with Vladimir Putin, playing golf with Bernie Sanders, or sipping cocktails with Kendrick Lamar. Google Brain’s goal isn’t subterfuge. It’s to solve the problem created by synthetic data. GANs would empower AI systems to work with raw, real-world data that hasn’t been cleaned and without the direct supervision of a human programmer. And while it’s a wonderfully creative approach to solve a problem—it could someday be a serious threat to our safety.
Still another paper cut: when complex algorithms work together, sometimes they compete against each other to accomplish a goal, and that can poison an entire system. We witnessed system-wide problems when the price of a developmental biology textbook started escalating quickly. The book was out of print, but Amazon showed that there were 15 used copies available from resellers, starting at $35.54—and two brand-new copies starting at $1.7 million. Hidden from view, Amazon’s algorithms had engaged in an autonomous price war, choosing to lift the price further and further until it reached $23,698,655 (plus $3.99 for shipping). The system of learning algorithms had made real-time adjustments in response to each auction, which is what they were designed to do. Put another way: we may have inadvertently taught AI that bubbles are a good thing. It isn’t difficult to image competing algorithms illogically inflating real estate assets, stock prices, or even something as simple as digital advertising.
These are just a tiny fraction of the paper cuts AI’s tribes have decided we can all live with in pursuit of the goals set by market forces in the United States and the CCP in Beijing. Rather than curbing expectations of speed and profitability, AI’s tribes are continually pressured to get products to market. Safety is an afterthought. Employees and leadership within the G-MAFIA are worried, but we don’t afford them any time to make changes. And we haven’t yet talked about China.
Between 2019 and 2023 we effectively ignore Xi Jinping’s proclamations about the future: China’s comprehensive national AI strategy, his plans to dominate the global economy, and China’s goal to become a singular force driving geopolitical decisions. We fail to connect the dots between the future of AI, its surveillance infrastructure and social credit system, and China’s person-to-person diplomacy in various African, Asian, and European countries. So when Xi speaks publicly and often about the need for a global governance reform and follows up by launching multinational bodies like the Asian Infrastructure Investment Bank, we give him the side eye rather than our full attention. It’s a mistake we don’t immediately acknowledge.
Within China, the path toward AI domination hasn’t been exactly smooth. China has its own paper cuts to contend with as the BAT struggles to innovate like Silicon Valley under the heavy-handed rule of Beijing. The BAT repeatedly skirts bureaucratic rules. All those earlier scandals—when China’s State Administration of Foreign Exchange fined Alipay 600,000 yuan (about $88,000) for misrepresenting international payments from 2014 to 2016, and Tenpay was punished for failing to file proper registration paperwork for cross-border payments between 2015 and 2017—turned out not to be anomalies.4 It becomes apparent that these aren’t isolated incidents as Chinese state officials experience the tension between socialist sensibilities and the realities of capitalism.
Already we are seeing the downstream implications of all these political, strategic, and technical vulnerabilities. To placate Wall Street, the G-MAFIA chase lucrative government contracts rather than strategic partnerships. This seeds competition rather than collaboration. It leads to restricted interoperability across AI frameworks, services, and devices. In the early 2020s, the market nudged the G-MAFIA to divvy up certain functionality and features: Amazon now owns e-commerce and our homes, while Google owns search, location, personal communications, and the workplace. Microsoft owns enterprise cloud computing, while IBM owns enterprise-level AI applications and applied health systems. Facebook owns social media, and Apple makes hardware (phones, computers, and wearables).
None of the G-MAFIA agrees to a single set of core values that prioritize transparency, inclusivity, and safety. While leadership within the G-MAFIA agrees that there should probably be widely adopted and implemented standards governing AI, there’s just no way to divert resources or time to work on them.
Your personal data record is built, maintained, and owned initially by four of the G-MAFIA: Google, Amazon, Apple, and Facebook. But here’s the rub: you’re not even aware that PDRs exist or that they’re being used by the G-MAFIA and by AI’s tribes. It’s not intentional but rather an oversight due to speed. It’s all explained in the terms of service we all agree to but never, ever read.
The formatting used by each PDR provider isn’t complementary, so there’s both duplicative data being spread around and, paradoxically, big holes with important data missing. It’s as if four different photographers took your photo: one with light stands and reflective umbrellas, one with a fisheye lens, one using an instant camera, and one shooting with an MRI machine. Technically what results are four pictures of your head, but the data embedded within them are vastly different.
In an effort to make a more complete picture, AI’s tribes release “digital emissaries”—little programs that act as go-betweens and negotiate on behalf of the G-MAFIA. The digital emissaries from both Google and Amazon work for a time, but they aren’t realistic long-term solutions. They’re too difficult to keep updated, especially since so many different third-party products and services link into them.
Rather than releasing new emissary versions daily, Google makes a big change.
In the early 2020s, Google releases its penultimate operating system, one mega-OS that can run on smartphones, smart speakers, laptops, tablets, and connected appliances. That’s just to start. Eventually, Google plans to grow and enrich this OS so that it becomes the invisible infrastructure powering our everyday lives, running our spoken interfaces, our smart earbuds and smart glasses, our cars, and even parts of our cities. That system is fully intertwined with our PDRs, and it’s a dramatic improvement for those who use it.
Google’s mega-OS comes at a bad time for Apple, which may have become America’s first trillion-dollar company but whose iPhone sales saw steady declines in the wake of newer connected devices like smart earbuds and wristbands. For its many successes, Amazon (America’s second trillion-dollar company) hasn’t had a big consumer hardware hit since its Echo smart speaker. In a surprising twist, Apple and Amazon partner exclusively in 2025 to build out a comprehensive OS that will power hardware made by both companies. The resulting OS—Applezon—poses a formidable threat to Google. In the consumer space, this cements a two-operating-system model and sets the stage for massive, fast consolidation within the AI ecosystem.
Facebook decides it must seek out a similar partnership; it’s bleeding active monthly users, who no longer view the social network as invaluable. It tries to friend Applezon, which isn’t interested. Microsoft and IBM stay focused on the enterprise.
China and its new diplomatic partners all use BAT technologies, while the rest of the world now uses either Google’s mega-OS or Applezon, both of which power and are powered by our PDRs. This limits our choices in the marketplace. There are a just a few options for smartphone models (and soon, the smart glasses and wristbands that will replace mobile phones) and for all of the devices in our homes: speakers, computers, TVs, major appliances, and printers. It’s easier to align ourselves with just one brand—so we are Google households or Applezon households. Technically, we can move our PDRs to other providers; however, we do not own the data in our PDRs, nor do we own the PDRs themselves. We are not afforded total transparency—what Google and Applezon do with our PDRs is, to a large extent, invisible by design to protect IP.