by Alec Ross
For companies that build facial recognition and other technologies that can empower authoritarian regimes, that also means being responsible about their customer base. Claiming ignorance is no longer a valid excuse, said R. David Edelman, the former White House senior official. “It is no longer an acceptable excuse for a tech CEO to say, ‘well, I didn’t know what use they were going to put it to.’ The sort of near-criminal negligence that we heard from tech CEOs of even three years ago is simply no longer plausible in today’s era.”
Technology companies have the expertise in which technology applications are possible and reliable. But it is also important for technology companies to have the autonomy to decide how their relationships with the national security community will proceed, and to develop clear principles for how what they build can be used. If they have an objection to specific applications of AI, it is worth voicing those broadly, in a way that other companies and policy makers alike can weigh.
“Now more than ever, we need to bring technologists into a place where they can help shape and craft the policies and the direction of not only how these technologies will be built, but how they will be used,” said Rebellion Defense CEO Chris Lynch. “If the conversation is only happening in the Department of Defense … that is not a long-term strategy. If the conversation is only being had in a coffee shop in San Francisco with a bunch of people who have never spent a moment thinking about the mission of defense, those people are failing just as much. If you don’t bring those two sides together, there is one thing I am one hundred percent certain of, and that is that nobody will be happy with the outcome. If you don’t have that discussion, and if you don’t participate in that discussion, we end up in complete and total failure.”
* * *
TO GRAPPLE WITH exactly why, we need to look across the Pacific to a wholly different model—that of China. Even the ability to debate the ethics of technology is a luxury of Western democracies.
While Western democracies work to define the expectations and responsibilities of their technology companies, this debate is nowhere to be found in China. That is because in China, the government and private sector act as one and the same.
To draw a line between state and capital in China is to make a distinction without meaning. Politicians and business leaders operate in tandem, with each doing their part to execute national strategies dictated by Beijing. Over the last forty years, the Chinese Communist Party used its command-and-control system to transform China from a poor agrarian backwater into an industrial powerhouse. And today, it is using it to advance a model of techno-authoritarianism that threatens the very foundation of liberal democratic society.
Modern China was built on the wave of globalization that began near the end of the Cold War. Beginning in the late 1970s, the Communist government implemented a series of market-based reforms to modernize China’s economy. Local entrepreneurs began launching their own businesses, and foreign multinationals flocked to the country’s coastal cities to take advantage of their cheap, lightly regulated labor markets. Within a few decades, China became the world’s factory, producing everything from German cars to American flags.
In 1990, when the Fortune Global 500 list was first published, it included zero Chinese companies. In 2020, the number of companies on the list from China and Hong Kong surpassed those from the United States, 124 to 121.
Over the last fifteen years, China has also emerged as a global leader in the industries of the future. The Chinese technology industry began as something of a counterfeit Silicon Valley, populated by copycat companies producing cheaper, less functional knockoffs of American technology, often with stolen intellectual property. But with close supervision and substantial assistance from Beijing, the industry came into its own. Today, companies like Alibaba, Baidu, Tencent, and Huawei are competing with Western technology giants for global market share. The US and China are near peers in technology, and in some areas the Chinese technology industry is outpacing Silicon Valley.
Like the manufacturing boom of prior decades, the ascendance of the Chinese technology industry was engineered by government policy makers. Unlike the United States government, the Chinese Communist Party does not need to bother with the messiness of the democratic process. Party leaders sit at the helm of a massive state apparatus, steering the country’s economic and political system free from the constraints of political parties and public consent. When party leaders want something done, the government and private sector immediately fall in line. There is no kludgeocracy or vetocracy in China.
This centralized control of the private sector gives China certain advantages. The Chinese government can mobilize investment in strategic areas in a way that Western democracies cannot. Hundreds of billions of dollars flow as a result of the national strategies set out by Beijing. Between 2007 and 2017, the government marshaled the construction of more than twenty-five thousand kilometers of high-speed railway, double the combined length of high-speed track laid by every other country up to that point.
In recent years, the Chinese government has devoted similar attention to building the country’s domestic artificial intelligence industry.
In 2017, President Xi Jinping unveiled a national strategy to transform China into the global leader in artificial intelligence by the year 2030, with a domestic industry worth more than $150 billion. The plan included a detailed roadmap of the policies that would lead the country to AI dominance, as well as billions of dollars in government funds to inject into promising start-ups and moonshot AI projects.
As soon as the plan was released, the government and private sector kicked into gear. Local governments started pouring funds into AI start-ups, and industry partnerships began to form. The following month, the government drafted a “national AI team,” selecting four domestic companies to take the lead in strategic AI fields including autonomous vehicles (Baidu), medical imaging (Tencent), natural language processing (iFLYTEK), and smart city technology (Alibaba). By August 2019, the team had expanded to fifteen members, each with its own area of expertise. These national champions are granted special access to government funds and databases. They collaborate with one another in a manner that does not exist in a serious way in Silicon Valley, sharing research insights and setting standards for the Chinese AI ecosystem.
China’s foray into artificial intelligence and other emerging technologies has also been bolstered by the aggressive acquisition of technology and intellectual property from foreign competitors, particularly in the United States. In some cases, Chinese firms can gain access to an American tech company’s crown jewels of intellectual property by offering capital investments, entering joint ventures, poaching employees, or acquiring companies outright. Foreign companies that want to do business in China are often required to share technical data and intellectual property directly with government agencies, which can then offer it up to domestic companies. But Chinese companies also steal competitors’ trade secrets, internal communications, and other sensitive data through cyberattacks. This economic espionage is often committed at the request—and with the direct assistance—of the Chinese government.
Through overt and covert technology transfer, China can effectively co-opt other countries’ research and development efforts for its own gain. The theft of foreign assets has enabled the Chinese economy—as well as its military—to keep pace with the rest of the world at a fraction of the cost.
As it bolsters its own economy through direct investment and state-sponsored tech transfer, the Chinese state and capital apparatus also works behind the scenes to wall off the country from foreign competition. With its 1.4 billion citizens, China is one of the most attractive global markets for multinational companies. Over the years, numerous technology companies have tried to gain a foothold in China to little avail. By way of illustration, when Uber launched in China in 2014, its CEO Travis Kalanick hoped his ride-sharing app would become one of the first American consumer technology firms to succeed in China. Having watched
US tech giants like Google, Amazon, and Facebook fail to expand into China, Kalanick sought to avoid making the same mistakes.
He set up a Chinese subsidiary—Uber China—which he hoped would avoid government restrictions that had kneecapped other Western tech companies. He made frequent visits to the country to court investors and study the local landscape—in 2015, Kalanick reportedly spent one in five days on the ground in China. The company catered its service to local consumers, allowing riders to pay for trips through Alipay, a popular third-party payment system. It entered into a joint venture partnership with the Chinese tech giant Baidu. It based its servers in China in an attempt to prevent any interference from the country’s Great Firewall.
“Travis structured something that worked within the construct of China … he was willing to make those compromises,” said Shervin Pishevar, an early Uber investor who introduced Kalanick to Baidu CEO Robin Li on a trip to China in 2013.
However, Uber faced a powerful local competitor—Didi Chuxing—and even with the support of local partners like Baidu, it could not overcome the realities of the Chinese market.
“Didi tried to destroy them,” Pishevar told me. The two companies entered “a massive war,” with each pouring billions of dollars into discounts and incentives to attract new drivers.
By 2016, Uber China had expanded to sixty cities, but Didi had set up operations in some four hundred. Both companies were burning through capital, but Didi had deeper pockets. The company received billions of dollars from China’s sovereign wealth fund and state-controlled banks. At one point, the Chinese tech giant Tencent blocked Uber on its massively popular app WeChat, but Didi remained embedded in the platform. In July 2016, the Chinese government issued regulations that prohibited driver subsidies that both Didi and Uber were using to expand their business, effectively freezing their existing market share. The regulations also applied more stringent government oversight to foreign companies. It was death by a thousand cuts.
Uber China was sold to Didi in August 2016. Kalanick and his partners retreated from the country with an equity stake in their victorious competitor and a $1 billion investment in its US business.
“Uber saw the writing on the wall,” Pishevar said. “They were like, ‘this is a war of attrition … we’ve got to make some kind of peace.’ The peace was only ‘twenty percent of Didi and get the fuck out.’ That’s what they did, and it was smart.”
By all accounts, Uber fought a good fight. It played by the rules and avoided the mistakes that hobbled other US tech companies in their pursuit of Chinese customers. But it occupied a strategic industry, and its success would have encroached on the technological ambitions of the Chinese government. China’s public-private capital apparatus kicked into gear, and the foreign rival was run out of town. After beating out Uber, Didi Chuxing opened AI research labs in Beijing and Silicon Valley. In July 2020, the company announced it would partner with the Chinese central bank to test a new digital currency.
It was a familiar story that has played out with countless Western technology companies: Western company enters China; company cuts promising deals with local partners; company slowly loses market share to a homegrown rival; company retreats from China; homegrown rival entrenches its dominant position.
China is proving to be as impenetrable to great Western technology companies as Afghanistan has been to great geopolitical powers like the Soviet Union and United States.
By keeping Western technology companies from gaining a foothold inside its borders, and by bolstering domestic companies through direct investment, economic espionage, and the state-sponsored acquisition of foreign technology assets, China is constructing a technological universe entirely removed from the Western sphere of influence. It is a world with its own companies, its own trends, and its own norms. And its values are dictated by the nationalistic ambitions of the Chinese government.
Over the last two decades, the Chinese government has used technology to construct an apparatus of surveillance and social control that would surpass the wildest dreams of despots from any prior century.
As of 2019, China has deployed approximately one surveillance camera for every four people, with that number expected to double by 2022. Most of the footage they collect is fed into AI systems trained to identify “antisocial” behaviors, ranging from murder to jaywalking to reading the Koran. Through facial recognition, the government can track every citizen throughout their daily lives, taking note of every place they go, every person they interact with, and every protest or meeting they attend.
Though controlled by the state, the Chinese surveillance apparatus is constructed and operated by the country’s myriad AI companies. It embodies the symbiosis between government officials and Chinese business leaders.
Building effective artificial intelligence requires lots and lots of training data, and you would be hard-pressed to find a government with fewer qualms about general public data collection than China’s. Companies share data with the government, which then shares data back with other companies, which then refine their algorithms and continue collecting more data. The CEO of the Chinese computer vision company SenseTime, which helped construct the Xinjiang surveillance apparatus, referred to the government as the company’s “largest data source.”
More data beget better algorithms, which beget better data. The surveillance state feeds itself and becomes more effective as it goes. As fifth-generation broadband networks enable China to embed more sensors on its streets, in its vehicles, and around its offices, homes, and public spaces, the panopticon will become more total.
In a real sense, China reveals the hazard that comes with perfect alignment of the business sector with the state. In the digital age, this arrangement puts all of our data in one set of hands, which can lead to frightening levels of control.
In most countries around the world, as we have seen the internet rise and spread, citizens have come to rely on companies as a buffer between ourselves and the state when it comes to our data. The private sector certainly is not a perfect buffer, and technology companies undoubtedly collect more data than most of us would like. But even so, the data habits of a Google or an Apple are a far cry from what can happen when there is no daylight between the government and the private sector. China’s example shows how that can hurtle a society into a real-world 1984.
China is also packaging its surveillance state for export. Thanks to state subsidies, Chinese technology companies can sell their products at artificially low prices, which has attracted foreign governments that want to construct their own apparatuses of social control. One of their customers is Zimbabwe.
Zimbabwe is no stranger to repressive regimes. The landlocked southern African nation was controlled by the British South Africa Company before becoming a British colony. It gained independence in 1980 after a fifteen-year civil war, only to fall under the thirty-seven-year rule of strongman Robert Mugabe. When Mugabe resigned in 2017 under pressure from the military, Zimbabweans rejoiced in the streets, hoping his ouster would set the country on a new trajectory toward democracy and political freedom. But less than six months later, the government of Zimbabwe signed a contract with the Chinese AI start-up CloudWalk Technology to install facial recognition systems in the country’s airports, bus stations, financial institutions, and other facilities. The government also planned to use the technology to build a national database of citizens’ faces. As the state integrates biometrics into its election system, the technology could become a tool for voter intimidation and control.
The biometric data from Zimbabwe can also be sent back to China, where it can help the Chinese private sector better train its algorithms to recognize African faces. China has engaged in similar partnerships with political leaders in the Philippines, Malaysia, Sri Lanka, Singapore, Mongolia, Serbia, Kenya, and other countries, and it is beginning to make inroads in Latin America. Once they hone their algorithms to identify people of additional races and ethnicities, Chinese companies will have a
n easier time selling their surveillance technology to authoritarians anywhere in the world and their own intelligence agencies will grow stronger.
It is not difficult to see the harm that persistent surveillance can inflict on democratic societies. A dynamic social contract hinges on citizens being able to express dissent through collective action. They mobilize against the government through opposition political movements and protest. They check the power of corporations through organized labor. But when the state and the private sector have such intimate insights into every person’s life, this sort of collective action becomes impossible. The group that controls the surveillance state can nip dissent in the bud before it ever blossoms. With enough data and processing power, they can even anticipate dissent and respond, like something out of a Philip K. Dick novel.
As China exports its techno-authoritarian model abroad and continues to refine it back home, it presents a fundamental threat to the future of democracy, particularly in the developing world. Fledgling democratic governments in Asia, Africa, Latin America, and even Europe are more likely to give in to authoritarian tendencies if they are equipped with the technologies of control.
In addition to software, China is the world’s top manufacturer of computer chips, telecommunications equipment, and other high-tech hardware. Using its position in the global technology supply chain, China can project its own power and kneecap its competitors. The US dependence on Chinese hardware for military equipment, medical supplies, and digital infrastructure remains a significant national security threat. American companies would have few places to turn if the Chinese government decided to cut them off from its factories.