by Amy Webb
China hasn’t waged physical war on any country since the 1979 Sino-Vietnamese War. And it wouldn’t appear as though China has any serious military adversaries—it hasn’t suffered terrorist attacks, it doesn’t have antagonistic relationships with the usual suspects (e.g., Russia, North Korea), and it hasn’t made enemies of other nations. So why the military push?
Because in the future, wars will be fought by code. Not hand-to-hand combat. Using AI techniques, a military can “win” by destabilizing an economy rather than demolishing countrysides and city centers. From that perspective, and given China’s unified march advancing artificial intelligence, China is dangerously far ahead of the West.
In my view, we’ve come to this realization too late. In my own meetings at the Pentagon with Department of Defense officials, an alternative view on the future of warfare (code vs. combat) has taken a long time to find widespread alignment. For example, in 2017, the DoD established an Algorithmic Warfare Cross-Functional Team to work on something called Project Maven—a computer vision and deep-learning system that autonomously recognizes objects from still images and videos. The team didn’t have the necessary AI capabilities, so the DoD contracted with Google for help training AI systems to analyze drone footage. But no one told the Google employees assigned to the project that they’d actually been working on a military project, and that resulted in high-profile backlash. Four thousand Google employees signed a petition objecting to Project Maven. They took a full-page ad out in the New York Times, and ultimately dozens of employees resigned.54 Eventually, Google said that it wouldn’t renew its contract with the DoD.
Amazon, too, came under fire because of Pentagon contract worth $10 billion. In October 2018, House Appropriations committee members Tom Cole, a Republican from Oklahoma, and Steve Womack, a Republican from Arkansas, accused the DoD of conspiring with Amazon to tailor the contract so that no other tech giant would qualify. But that wasn’t the only complaint. There was a small wave of dissent at Amazon. Some Amazon workers were outraged that the company would do any work at all with the US military, while others didn’t like that Amazon’s facial recognition technology was being used by law enforcement. In response, Jeff Bezos told a conference audience, “If big tech companies are going to turn their back on the US Department of Defense, this country is going to be in trouble.”55
While in the US, our tech giants are navigating a tricky path between national security and full transparency, the relationships the BAT has with China’s government are exactly opposite. But here’s a chilling example: the current posture of the US military is that a human must be kept in the loop, regardless of how advanced AI, unmanned systems, and robots become. This will ensure that we don’t someday cede lethal authority to software. That is not the case in China.56 PLA Lieutenant General Liu Guozhi, who directs the Chinese military’s Science and Technology Commission, is quoted as warning, “(We) must… seize the opportunity to change paradigms.”57 It was an indirect way of announcing China’s intent to rebuild its military might.
Third, if economic and military advantages aren’t concerning, China’s views on privacy will be. Again, why would this matter to you if you’re not a Chinese citizen? Because authoritarian governments form all the time, and they tend to emulate the playbooks of established regimes. With nationalism on the rise worldwide, the way China is using AI could become a model in other countries in years to come. This could destabilize markets, trade, and the geopolitical balance.
In what will later be viewed as one of the most pervasive and insidious social experiments on humankind, China is using AI in an effort to create an obedient populace. The State Council’s AI 2030 plan explains that AI will “significantly elevate the capability and level of social governance” and will be relied on to play “an irreplaceable role in effectively maintaining social stability.”58 This is being accomplished through China’s national Social Credit Score system, which according to the State Council’s founding charter will “allow the trustworthy to roam everywhere under heaven while making it hard for the discredited to take a single step.”59 It’s an idea that goes back to 1949, when the Communist Party first took power and began experimenting with various social control schemes. During Mao Zedong’s rule in the 1950s, social policing became the norm: workers were forced into communal farm groups, and they were assigned rankings based on their output. Individuals policed each other as members of farm groups, and that ranking determined how much access someone had to pubic goods. The system broke down under Mao, and it collapsed again in the 1980s because, as it turns out, humans aren’t accurate judges of each other—they’re motivated by their own individual needs, insecurities, and biases.
In 1995, then President Jiang Zemin envisioned a social policing system that leveraged technology—and by the mid-2000s, the Chinese government was working to build and implement a scoring system that functioned automatically.60 It partnered with Peking University to establish the China Credit Research Center to research how to build and implement an AI-powered national social credit score system. This partially explains the current president’s insistence on AI. It promises not only to make good on that idea proposed at the dawn of the Communist Party; importantly, it promises to keep the Communist Party in power.
In the city of Rongcheng, an algorithmic social credit scoring system has already proven that AI works. Its 740,000 adult citizens are each assigned 1,000 points to start, and depending on behavior, points are added and deducted. Performing a “heroic act” might earn a resident 30 points, while blowing through a traffic light would automatically deduct 5 points. Citizens are labeled and sorted into different brackets, ranging from A+++ to D, and their choices and ability to move around freely are dictated by their grade. The C bracket might discover that they must first pay a deposit to rent a public bike, while the A group gets to rent them for free for 90 minutes. It isn’t just individuals getting scored. In Rongcheng, companies are also scored for behavior—and their ability to do business depends very much on their bracket standings.61
AI-powered directional microphones and smart cameras now dot the highways and streets of Shanghai. Drivers who honk excessively are automatically issued a ticket via Tencent’s WeChat, while their names, photographs, and national identity card numbers are displayed on nearby LED billboards. If a driver pulls over on the side of the road for more than seven minutes, they will trigger another instant traffic ticket.62 It isn’t just the ticket and the fine—points are deducted in the driver’s social credit score. When enough points are deducted, they will find it hard to book airline tickets or land a new job. There was a popular episode of Black Mirror portending a dystopian future like this. In Shanghai, that future has already arrived.
State-level surveillance is enabled by the BAT, who are in turn emboldened through China’s various institutional and industrial policies. Alibaba’s Zhima Credit service hasn’t publicly disclosed that it is part of the national credit system; however, it is calculating a person’s available credit line based on things like what that person is buying and who his or her friends are on Alipay’s social network. In 2015, Zhima Credit’s technology director publicly said that buying diapers would be considered “responsible behavior,” while playing video games for too long would be counted as a demerit.63
Recall our earlier discussion in the introduction of China’s Police Cloud, which was built to monitor and track people with mental health issues, who have publicly criticized the government, and who are ethnic minorities. The Integrated Joint Operations Program uses AI to detect pattern deviations, such as jaywalking. China’s social credit scores rate and rank citizens based on their behavior; decision-making AI systems use those scores to determine who’s allowed to secure a loan, who can travel, and even where their children are allowed to go to school.
Robin Li, one of Baidu’s founders, argued that to the Chinese, privacy isn’t a core value as it is in the West. “The Chinese people are more open or less sensitive about the privacy iss
ue,” Li told an audience at the China Development Forum in Beijing. “If they are able to trade privacy for convenience, safety, and efficiency, in a lot of cases, they are willing to do that.”64 Or maybe it has more to do with repercussions.
I’d argue that China’s national social credit score isn’t about strengthening the Communist Party or a complicated way of achieving strategic advantage over those working on AI in the West. Rather, it’s about exerting total control to shape our global economy. Early in 2018, President Xi told the state news agency Xinhua that “by tightening our belts and gritting our teeth, we built ‘two bombs and one satellite,’” which was a reference to a military weapons program developed under Mao. “This was because we made best use of the socialist system. We concentrated our efforts to get great things done. The next step is to do the same with science and technology. We must cast away false hopes and rely on ourselves.”65
Xi rejects the notions of market economies, a free internet, and a diverse ecosystem of competing and complementary ideas. China’s tightly controlled domestic economy walls itself off from competition. It is enabling “splinternets” in which the rules of the internet depend on a user’s physical location. It is centralizing cyberpolicy, clamping down on free speech, and asserting itself into every aspect of the third era of computing via regulatory control: the internet’s infrastructure, the global flow of data, and the hardware are increasingly subject to Beijing’s approval. Speaking at an event in 2016, Xi said that the government would henceforth have total discretion to determine how it would protect networks, devices, and data.66
It will exert this considerable control by enticing its Belt and Road Initiative partners with infrastructure and tech pilots. Tanzania was selected as an early pilot partner—and, perhaps not coincidentally, the country has now adopted many of China’s data and cyberpolicies. Tanzania’s government was given technical assistance by Chinese counterparts, and a senior Tanzanian official said that “our Chinese friends have managed to block such media in their country and replaced them with their homegrown sites that are safe, constructive, and popular.”67 The same is happening elsewhere in Africa. Vietnam has now adopted China’s stringent cybersecurity laws. As of June 2018, India was considering legislation that would mirror China’s requirements for housing domestic data and sourcing domestic cybersecurity technologies.68
What if China starts influencing its Belt and Road Initiative partners such that one of its primary exports is its national social credit score system? It’s easy to see how the world’s autocracies, like Turkey and Rwanda, could become a buyer of China’s surveillance technology. But what about in other countries, such as Brazil and Austria, that have given in to populism and as of this writing have nationalistic leaders? What if a government agency in your country is inspired or strong-armed into adopting a social credit score system, one that begins monitoring you without your explicit consent? Would you ever know that you had a score, perhaps one that landed you on a watch list?
What if foreign companies are assessed brackets and either given preferential treatment or prevented from doing business with China—or even with each other? As China’s economy grows, what happens if this power and influence propagates throughout the internet, our gadgets and devices, and AI itself?
What if China builds a social credit score for people outside its borders, using data it mines on the free and open web and the West’s social networks? What if it’s scraping all the ambient data you’re leaving behind after your trips visiting the Great Wall and Forbidden City? What about all the hacking operations we hear about periodically, where big data breaches appear to be coming from networks based in China?
There is another reason we should be concerned about China’s plans, and that brings us back to that place where AI’s tribes form: education. China is actively draining professors and researchers away from AI’s hubs in Canada and the United States, offering them attractive repatriation packages. There’s already a shortage of trained data scientists and machine-learning specialists. Siphoning off people will soon create a talent vacuum in the West. By far, this is China’s smartest long-term play—because it deprives the West of its ability to compete in the future.
China’s talent pipeline is draining researchers back into the mainland as part of its Thousand Talents Plan. The rapid expansion of the BAT has created demand for talented people—most of whom trained in the United States and are currently working in American universities and companies. This government scheme targets chief technologists and tenured academics, offering them a golden ticket of sorts: providing them with compelling financial incentives (both personal and for research projects) and a chance to join an R&D environment free from the regulatory and administrative constraints common in the US. More than 7,000 people have been accepted into the program so far, and they’ve received a signing bonus from the Chinese government: 1 million yuan (roughly $151,000), an initial personal research budget of 3–5 million yuan ($467,000–$778,000), subsidies for housing and education, meal allowances, relocation compensation, assistance helping spouses land new jobs, and even all-expenses-paid trips to visit home.69 All of the returnees—in some way, even if a few steps removed—end up using their talents on behalf of the BAT.
America’s Tribes:
The G-MAFIA
If AI is China’s space race, it’s currently positioned to win, and to win big. During the past two years, as AI has passed critically important milestones, the Trump administration siphoned money away from basic science and technology research, spread false information about AI’s impact on our workforce, alienated our strategic global allies, and repeatedly taunted China with tariffs.
We will soon grapple with the realization that our lawmakers have no grand strategy for AI nor for our longer-term futures. Filling the void is opportunism and the drive for commercial success. America’s Big Nine companies may be individually successful, but they are not part of a coordinated effort to amass and centralize economic and military power in the United States. Not that they would—or should—agree to such a scheme.
The origin of America’s part of the Big Nine is now a familiar story, but less well known are the significant changes about to take place in the relationship between America’s Big Nine members, your data, and the devices you use.
The US-based portion of the Big Nine—Google, Microsoft, Amazon, Facebook, IBM, and Apple—are inventive, innovative, and largely responsible for the biggest advancements in AI. They do function as a mafia in the purest (but not pejorative) sense: it’s a closed supernetwork of people with similar interests and backgrounds working within one field who have a controlling influence over our futures. At this particular moment in time, Google wields the most of that influence over the field of AI, our businesses, our government, and our daily lives, so we’ll refer to America’s companies as the G-MAFIA. It’s no wonder they inspired so much imitation in China or that they’ve largely found themselves blocked from doing business there. They didn’t start out as AI companies, but in the past three years, all six have shifted their center of gravity to focus on the commercial viability of AI, though R&D, partnerships, and new products and services.
In China, the government exerts control over the BAT. In the United States, the G-MAFIA wield significant power and influence over government in part because of America’s market economy system and because we have a strong cultural aversion toward strong government control of business. But there’s another reason the G-MAFIA are so influential—they have been ignored by DC lawmakers. While Xi was consolidating domestic power and publicly launching his 2030 plan for global AI dominance, Trump’s deputy assistant for technology policy, Michael Kratsios, told a group of industry leaders convened at the White House that the best way forward for America was for Silicon Valley to chart its own course independently without government intervention.70
There is an imbalance of power because the US government hasn’t been able to create the networks, databases, and infrastructure it nee
ds to operate. So it needs the G-MAFIA. For example, Amazon’s government cloud-computing business will likely hit $4.6 billion in 2019—while Jeff Bezos’s private space company, Blue Origin, is expected to start supporting NASA and the Pentagon on various missions. In America, the government relies on the G-MAFIA, and since we’re a market-driven economy with laws and regulations in place to protect businesses, the Valley has a significant amount of leverage. Let me be very clear: I do not begrudge the G-MAFIA’s role as successful, profitable companies. Nor do I believe that earning lots of money is in any way negative. The G-MAFIA should not be constrained or regulated in their pursuit of profit as long as they aren’t violating other laws.
But all this opportunity comes at a cost. There is tremendous pressure for the G-MAFIA to build practical and commercial applications for AI as quickly as possible. In the digital space, investors have grown accustomed to quick wins and windfalls. Dropbox, a file-sharing platform, reached a $10 billion valuation just six years after it launched. Silicon Valley venture capital firm Sequoia Capital owned a 20% stake when Dropbox filed its IPO, making its shares worth $1.7 billion.71 In Silicon Valley, startups that are valued over $1 billion are called “unicorns,” and with a valuation ten times that amount, Dropbox is what’s known as a “decacorn.” By 2018, there were enough unicorns and decacorns to fill a Silicon Valley zoo, and several of them were partners with the G-MAFIA, including SpaceX, Coinbase, Peloton, Credit Karma, Airbnb, Palantir, and Uber. With fast money comes heightened expectations that the product or service will start earning back its investment, either through widespread adoption, acquisition, or hype in the market.
You have a personal relationship with the G-MAFIA, even if you don’t use their well-known products. The “six degrees of separation” theory is a mathematical way to explain how we’re all connected—you are one degree of separation away from anyone you know, and two degrees from people they know, and so on. There are shockingly few degrees of separation between you and the G-MAFIA, even if you’re offline.