Intelligence in War: The Value--And Limitations--Of What the Military Can Learn About the Enemy

by John Keegan

  Although the outcome of the First World War made Japan a Pacific oceanic power, both its domestic and external affairs after 1919 and until the late thirties were concerned almost exclusively with China. For centuries, even millennia, China’s cultural subordinate, Japan by the twentieth century had determined that its future lay in a reverse subordination, economic but also political and military, of China to its imperial needs. In 1915 Japan had issued a set of “twenty-one demands” which required China to concede rights and privileges to Japan, according it overlord status. The Chinese prevaricated and resisted, as far as they were able. In 1931 they were forced, however, to submit to effective Japanese annexation of Manchuria and then in 1937 to a full-scale Japanese invasion of the south. The nationalist government of Chiang Kai-shek withdrew inland, first to the city of Nanking, then to Chungking. Its capacity to resist was hampered by the attacks of the Chinese Communist Party armies under Mao Tse-tung.

  Japan’s imperial policy was strengthened and furthered during the 1930s by the rise of an intense nationalist spirit within its military class, particularly in the army. The “Manchuria Incident” of 1931 was largely the work of nationalist officers, in the Manchuria garrison. The “China Incident,” so-called by American observers, of 1937 in Shanghai was equally an outburst of ill-discipline by the Japanese occupying troops. By that date, however, the army led the government, which had escaped from the control of constitutional statesmen. By the outbreak of the Second World War, Japan, then in alliance with Germany and Italy, was a totalitarian state, committed to an imperialist programme of territorial expansion directed against China, with which it was in full-scale war, the Asian possessions of the European empires, principally Britain and the Netherlands, and the United States.

  The war on the mainland of China consumed most of the strength of the Japanese army, which fielded about twenty-five divisions. Militarily it was far superior to that of the Republic of China, which survived total defeat only by its ability to use space as a means of defence. The Japanese were not able to penetrate far beyond the coastal provinces, though as those contained China’s larger cities and main rice-growing areas, they had little strategic reason for mounting deeper offensives.

  The Japanese navy was scarcely involved in the China war, which had no maritime dimension. It was, nevertheless, much concerned with the strategic future since Japan’s attack on China had provoked the wrath of the United States, manifested in a series of increasingly constrictive trade embargoes. Japan, like Britain, lacked the domestic resources necessary to support an imperial policy. Its home islands did not produce enough food to support its population, which relied heavily on imports of rice, while its industries and infrastructure required large imports of metal ores, scrap and oil. By 1941, after Japan’s deployment of troops into French Indo-China, enforced on the defeated Vichy government, an initiative which directly threatened British Malaya, the American oil and metal embargoes were seriously hampering Japan’s ability to sustain its manufacturing output. America’s intention was to restrain Japan’s military ambitions. The effect was to drive Japan towards aggressive war.

  The Japanese army and navy operated, to a degree unusual even in the arena of naval and military rivalry, as separate entities. The army, which dominated government, only reluctantly accepted the navy’s right to speak on strategy. On the other hand, the navy justifiably argued that since the United States dominated the Pacific, the success of national strategy depended on its plans to overcome American naval power. In 1936 the army and navy had agreed on a statement of Fundamental Principles of National Strategy. The statement committed the army to achieve a strength sufficient to contain the Soviet Union—the old Russian enemy—in the Far East and the navy to acquire both a dominance in the South Seas, meaning the islands and peninsulas possessed by the British and Dutch, and the ability to “secure command of the Western Pacific against the U.S. Navy.”2

  By the summer of 1941 Japan was in a strategic quandary. Though the army had suffered in border conflicts with the Soviet Union in 1936 and 1939, it was in the ascendant in China, controlled Manchuria and had assumed a forward position in French Indo-China. Its leaders recognised, however, that the American embargo policy threatened to terminate its ability to sustain its aggressive strategy within one or two years. The navy lay under an even more exigent threat. American restrictions on the export of aviation fuel promised to end its capacity to conduct carrier operations even sooner. To maintain parity of status with the army, it needed access to a supply of oil outside American control, and the only sources available within its strategic zone lay in the Dutch East Indies and Burma. By the middle of 1941 the Japanese navy was psychologically committed to a war of Pacific conquest.

  Although Japan had benefited from the peace settlement of 1919 by its acquisition of the German Pacific islands, it had suffered under the post-war disarmament treaties. The Washington Naval Treaty of 1922, designed to avert another naval arms race equivalent to that between Britain and Germany, widely held to have helped precipitate the First World War, imposed a subordinate naval status on Japan. The United States and Britain, arguing that both their navies had two-ocean commitments, in the Atlantic and the Pacific, succeeded in bringing their wartime Japanese ally to accept that, as a Pacific power alone, it needed only 60 per cent of their naval strength. This 5:5:3 ratio, as it became known, applied to battleships, cruisers, destroyers and aircraft carriers. All signatories—they also included France and Italy—were required as a result to scrap some of their larger naval units and to limit the size of ships planned or under construction.

  The Japanese, who bitterly resented what they saw as Anglo-American condescension towards their status as a world naval power, had no recourse but to agree. They proceeded, nevertheless, to exploit any loophole in the treaty that was open to them. The Americans and British did likewise, converting half-constructed dreadnoughts into large aircraft carriers. The Japanese went further. By 1941 they had assembled seven carriers, a larger naval aviation fleet than that possessed by either Britain or the United States. More important, the Japanese carrier air groups were equipped with aircraft respectively superior and greatly superior to those embarked by their American and British counterparts. The torpedo bomber code-named Kate by the Americans was a better aircraft than their Avenger, while the Zero remained unchallengeably the best carrier-borne fighter in the world until the appearance of the American Hellcat in 1942. Japan’s naval air arm pilots were also of the first class; those who took part in the attack on Pearl Harbor had 800 hours of flying experience. Yet there were defects in the Japanese naval aviation system: the Zero was essentially a racing sports aircraft, faster both flat out and on the turn than contemporary American fighters, but fragile and combustible; the Japanese flight schools were not adapted to produce pilots en masse, so that any heavy loss of trained aircrew threatened the efficiency of the air groups. After Japan’s stunning initial victories, its weaknesses in aircraft design and pilot output would lead quite quickly to a decline in its ability to wage carrier warfare on equal terms with the United States.3

  By the summer of 1941 the Japanese army and navy were confronting the necessity to go to war and considering how best to deliver the opening stroke. The economic objectives were easily defined: the oil fields of the Dutch East Indies and Burma, the tin mines and rubber plantations of Malaya. The politics of the operation were far more complex. War with Britain could not be avoided, for her colonies were to be directly attacked; but, as her forces were already overstretched in the fight against Hitler, the consequences were manageable. War with the United States could equally not be avoided; the question was how long the outbreak could or should be postponed. Four timetables were considered: to seize the Dutch East Indies first, then the Philippines and Malaya, a sequence that would bring on early a war with the United States, which could not be allowed to retain a base in the Philippines; to attack the Philippines immediately, then the Dutch islands, then Malaya; to begin with Mala
ya and work backwards towards the Philippines, thus delaying a confrontation with the United States; to attack Malaya and the Philippines together, then the Dutch East Indies.

  Since the last sequence was the only one on which the army and navy could agree, it was adopted; but since it entailed provoking war with the United States from the outset, it also required the making of a subordinate plan about how to neutralise American naval power in the Pacific. Since early in the century, the Japanese navy had planned to defeat the American Pacific Fleet by drawing it into Japanese home waters, wearing down its strength meanwhile by attritional attacks as it made its long voyage across the ocean; the logic of the strategy was enhanced after 1919 by the acquisition of the Central Pacific barrier formed by the ex-German islands. An early war with the United States, however, demanded a quicker means of reducing American naval power.

  Admiral Isoroku Yamamoto, Commander of the Combined Fleet, which included the main carrier force, had been considering the problem since early 1941. In principle he opposed making war on the United States, which he knew well as a former English-language student at Harvard and naval attaché in Washington; he did not believe that Japan’s small industrial base could ever effectively support a war against the United States’ vastly larger economy. His well-known views had made him unpopular both with nationalist politicians and their supporters and within the armed forces; he had been sent to sea in 1939 largely to save him from assassination. The threats were not hollow; in 1936, a group of super-nationalist army officers had killed several moderate politicians, including the finance minister and a former prime minister, occupied central Tokyo and been overcome only after three days of street fighting. Yamamoto undoubtedly had reason on his side, as other naval officers saw. Confronted, however, by the reality of the army-dominated government’s determination to solve Japan’s economic problems by aggressive measures, Yamamoto stifled his objections and proposed an alternative attack strategy. He suggested using the carrier force to destroy the American Pacific Fleet at its moorings in Pearl Harbor, Hawaii, its Central Pacific base.

  Yamamoto’s thinking was greatly influenced by the results of the British attack on the Italian battle fleet in the harbour of Taranto on 11 November 1940. Aircraft launched from the carrier Illustrious had then sunk three Italian battleships with torpedoes, losing only two out of twenty-one aircraft engaged. While the Japanese naval air arm experimented with adapting torpedoes—the principal Japanese torpedo was faster, of longer range and fitted with a larger warhead than any in use in other navies—to run level in shallow water, Yamamoto and his staff considered how to bring the Combined Fleet undetected to within striking distance of Pearl Harbor. By October 1941 the planners—including Commander Minoru Genda, who would lead a major part of the First Air Fleet in the attack—had outlined a scheme which would take the Combined Fleet from the stormy waters of the northern Kurile Islands, above the isolated American possession of Midway Island and then due south, off frequented shipping lanes, to within 200 miles of Hawaii. The fleet would move in complete radio silence and, if possible, within the leading edge of one of the turbulent weather fronts common in the North Pacific, which impeded visual reconnaissance and interfered with radio transmissions. During October the liner Taiyo Maru was sailed down the chosen route without sighting a single ship.

  The Japanese and American governments meanwhile continued reasoned diplomatic exchanges. Japan asked for a relaxation of American trade embargoes—a joint army–navy committee estimated in June 1941 that oil reserves were being depleted at a rate a third greater than they were being replaced, a disastrous situation, since it implied inexorably that Japan would run out of oil (stocks plus replacement 33 million barrels, consumption 41 million barrels) in 1942–43.4 In return Japan would undertake to cease her military intrusion into Southeast Asia and eventually leave French Indo-China. The United States demurred, proposing instead a scheme to settle affairs all over mainland Asia, which would have required Japan to withdraw from China as well as Indo-China, and to leave Manchuria also. The Japanese, as was to be expected, rejected the proposal. On 4 December an imperial conference took the decision to go to war against the United States, beginning with an attack on Pearl Harbor, on the 7th. The Combined Fleet was already en route.

  AMERICAN PENETRATION OF JAPANESE CIPHERS

  The attack on Pearl Harbor on 7 December has given rise to one of the largest conspiracy theories in history. There are many versions, most alleging American foreknowledge, and scarcely one agrees with another. Allegations of culpable incompetence apart, which deserve attention, the two most important theories allege, first, that the British had foreknowledge of Japanese intentions but chose to conceal what they knew from the United States in order to bring America into the war; second, that President Roosevelt knew independently what the Japanese intended but took no preventative action, since he currently sought a pretext to bring his country into the war on Britain’s side. The two theories, in some of their versions, overlap.

  The subject is so large that it has stimulated the production of a library of books. Almost the only matter on which they agree is that American cryptanalysts, like their British counterparts at Bletchley, were freely reading Japanese ciphers before December 1941. Exactly what was read and when, what the interpreters made of the decrypts, and how the decrypts influenced the decisions taken by President Roosevelt, his cabinet officers, the chiefs of staff and the commanders in the zone of operations form the substance of the great Pearl Harbor mystery story.

  It does not connect with the story of Midway, which began to unroll only six months later. There is, however, this caveat: the cryptanalytic organisation which may have failed before Pearl Harbor was also the organisation that helped to deliver the victory of Midway. It worked in this way.

  American cryptanalysis differed from its British equivalent, in organisation, recruitment and ethos. Bletchley was a joint-service civil-military body, in which little distinction of rank was observed, built up on a word-of-mouth basis in the period immediately before the outbreak of war in 1939 and recruited largely among young Oxford and Cambridge dons. Proven mathematical ability was the principal qualification. The atmosphere at Bletchley was creatively amateur, informal and high-spirited; women formed a high proportion of the staff, some in senior positions, and romance flourished. There were many Bletchley marriages.

  The American cryptanalytic organisation, by contrast, was sharply divided into naval and military branches, which co-operated uneasily, was highly bureaucratic and almost completely male-dominated. Most of the cryptanalytic personnel were uniformed servicemen and the principal qualification for selection was language skill, particularly in Japanese. Unlike Bletchley, which had a high opinion of itself and cultivated a genial university common room atmosphere, the American intelligence branches were regarded by the rest of the army and navy as backwaters, staffed by officers unsuitable for operational appointments, an opinion of which their members were aware. It is remarkable, in the circumstances, how well they maintained their professional morale. A key indication of the difference between the British and American systems is that, while Bletchley has entered into British national legend and found a popular place in fiction and film, its American equivalents enjoy no such acclaim. Quite wrongly, for what the Americans achieved was equally remarkable, indeed perhaps more so, as the story of Midway indicates.

  The origins of the American cryptanalytic service belong, as do those of the British, in the First World War. Major Joseph Mauborgne, head of the army’s cipher research section in 1918, was a cryptographer far ahead of his time: he perceived the concept of the random key—one not retrievable by frequency analysis or, indeed, any mathematical or linguistic logic—and devised the one-time pad, still the only intrinsically unbreakable cipher. He would eventually become a general and the U.S. Army Chief Signal Officer.5 Of the same vintage but of even great importance to the American cryptanalytic effort was a civilian, William Friedman (who coined the term “c
ryptanalysis”). The son of a Russian Jewish immigrant, who entered the United States at the age of one, Friedman resembled in character both Dilly Knox and Alan Turing. As eccentric as either, he displayed a mathematical ability almost equivalent to Turing’s but unfortunately also Knox’s psychological fragility. He had suicidal tendencies and just before the outbreak of the Pacific War suffered a nervous collapse brought on by overwork.6

  Yet Friedman was largely responsible for the most important of America’s cryptanalytic successes, the breaking of Purple. In October 1940, the army and navy agreed on a division of labour, not in any spirit of fraternal co-operation but because each lacked the numbers to do much more than concentrate on a single task; in 1938 Friedman had a staff of only eight in the Secret Intelligence Service (SIS), in December 1940 the navy in Washington only thirty-six in its equivalent OP-20-G. There were other personnel at outstations in both the continental United States and the Pacific but most were intercept operators and technicians.7 The arrangement was that the army would work on foreign diplomatic intercepts on even days of the month, the navy on odd. The navy was meanwhile, naturally, working on Japanese naval intercepts; the army was not particularly interested in foreign army intercepts, since they were too faint to yield text.

  The Americans, few though they were in number, had had considerable success in breaking into both Japanese naval and diplomatic traffic in the 1930s, assisted by a succession of night-time burglaries of the Japanese consulate in New York. By 1933 the naval cryptanalysts had solved the main Japanese naval Blue Code, a book code with a cipher additive. When it was replaced in June 1939 by JN-25 (Japanese Naval Code 25, as the Americans denoted it), another book code with a more complex additive, the Americans took time to recover from the setback, but by December 1940, with the help of recently acquired IBM card-sorting machines, they had reconstructed the system of additives, the first thousand groups of the code and two of the keys used to work the system.