MAGIC

MAGIC is most commonly known as the American codeword to identify deciphered Japanese diplomatic communications. During the war the word was also used for deciphered Japanese military communications, and, to add to the confusion, all these deciphered messages were classified TOP SECRET ULTRA. While there is overlap in substance as well as designation between the diplomatic and military categories, it is important to preserve the distinction between messages exchanged by the Japanese foreign ministry and its diplomatic posts abroad and those of the Japanese army and navy. Accordingly MAGIC is restricted here to its more generally understood meaning of diplomatic communications, and Japanese military decipherments are dealt with elsewhere (see ULTRA, 2).

MAGIC included all decrypted messages in Japanese diplomatic codes and ciphers. The most valuable by far were those encrypted by the cipher machine known to the Americans as PURPLE. The cryptanalytical feat of breaking into PURPLE's ciphers was extraordinary. Once in, the Americans were able to read the most secret Japanese diplomatic communications from before Pearl Harbor to the end of the war. MAGIC supplied no specific warning of the attack on Pearl Harbor or on British and American possessions in South-East Asia, but the cumulative effect of the 1941 messages was the impression of an expansionist Japan ever nearing a decision for war. With MAGIC American officials could in effect peer over the shoulder of the Japanese ambassador in Washington as he sought a diplomatic formula to avoid war in the spring and autumn of 1941 (see also USA, 1).

By way of the Japanese ambassador in Berlin, MAGIC intelligence also provided vital information throughout the war about German plans and operations against the USSR and the western Allies, as well as about Japan's relations with the USSR and its attempts to secure Soviet mediation in ending the war.

The PURPLE cipher machine consisted of two typewriter keyboards connected by a maze of circuits, plugs, and switches. Machine encipherment was an important form of communications security in the Second World War, the most famous example being the German ENIGMA machine, but it was unusual for Japan and the PURPLE itself was unique. No ‘J’ machine, called the Type 97 Alphabetical Typewriter (97-shiki O-bun In-ji-Ki) by the Japanese, survived the war, nor are any of the American analogues known to exist. Eyewitness descriptions are few and sparse but offer a rough idea of how the machine worked

In place of rotors, which supplied a sequence of letter substitutions in most enciphering machines, the PURPLE machine used switching gear—stepping switches, to be precise—from the dial telephones of the day. To encipher, the operator pressed the appropriate typewriter key for the plain text letter. This carried current through a plugboard (like an old telephone switchboard), which provided changeable letter substitutions, which served as ‘keys’ for use on any given day. Thence the current ran through a series of stepping switches. Each of these consisted of a semicircular array, or matrix, of electrical contacts facing a shaft from which projected finger-like conductors, which rotated across the matrix. Each contact and finger stood for a letter of the alphabet. Since the fingers and the matrix were wired differently, each time a typewriter key was pressed a letter substitution occurred. At set intervals the shaft and fingers rotated ahead one or more steps to a new set of contacts, new wiring, and deeper encryption. Then the current passed on to another stepping switch (the PURPLE machine had four) and finally depressed a cipher text key in the second typewriter. Obviously these successive substitutions provided a formidable challenge to cryptanalysts.

Theoretically the possible substitutions by machine cipher were almost endless, millions upon millions. Practically the task was somewhat less daunting. In most cases, including MAGIC, the cryptanalysts found beachheads into the cipher from bureaucratic words and phrases regularly used and available in plain text. Certain forms of address and key words related to the events of the day could be anticipated. In the case of MAGIC, the Japanese foreign ministry made a critical mistake in repeating messages sent in previous encipherment (by the so-called Red machine), so old decrypts could be used to solve PURPLE. Easing the task was the division of the alphabet into two subsets, each group enciphering separately. Keys changed every ten days but within the month varied only slightly and predictably.

In many cases in the Second World War cryptanalysts were assisted by the capture of the other side's code or cipher material or machines but this was not the case with PURPLE. However, American cryptanalysts could get the gist of some enciphered texts from Japanese diplomatic messages delivered to the state department.

Leading the attack on PURPLE was the best American mind in codes and ciphers, William F. Friedman (1891–1969), chief cryptanalyst of the US Army Signal Intelligence Service. A child of Russian-Jewish immigrants, he followed the more prosaic career interests of electrical engineering and plant genetics until 1916 when at the Riverbank Laboratories in Geneva, Illinois, he became intrigued by a project aiming to prove by means of a hidden cipher that Sir Francis Bacon was the real author of Shakespeare's plays. During the First World War Friedman served in the code room of American headquarters in France and in 1921 returned to the army to spend the next 34 years at the heart of American cryptography.

Modern cryptology, as the authority David Kahn says, is ‘saturated with mathematical operations, mathematical methods, mathematical thinking’ (see Kahn [below], p. 410). Friedman, though not trained specifically as a mathematician, was expert in statistics and probability, and an authority in applying these to cryptanalysis. In addition he brought to the task of breaking PURPLE mastery of the whole field of cryptology, exceptional intuition, and dogged perseverance. Although his contribution was vital (and so intensive it put him in hospital with a nervous breakdown), this was not a one-man show, for Friedman engaged talented PhDs in mathematics to help him; and the US Army and Navy had made a prolonged effort to gather and train staffs of cryptanalysts, pooling resources and sharing results in the attack on PURPLE.

Beginning in early 1939, the breaking of PURPLE took eighteen months. One hypothesis after another was tried and discarded. A critical breakthrough occurred when a cryptanalyst from naval intelligence, Harry L. Clark, suggested that the Japanese might be using ordinary telephone stepping switches instead of rotors. Laboriously they separated cipher text into segments representing different key settings and then tackled texts in the same key, building out from known letters and words, looking for symmetries in the position of letters, and trying out letters according to the known frequency of their use. Translators filled in missing letters and completed words.

With agonizing slowness at first and then gradually more swiftly the plain text messages emerged. The first message was completed on 25 September 1940, two days before the signature of the Axis Tripartite Pact. So impressed with the feat was one authority that he referred to the team as ‘magicians’, hence the codeword MAGIC.

Once the Friedman group understood what kind of a machine enciphered PURPLE and how it must be wired, they constructed a machine to duplicate its functions. By the spring of 1941 four of these machines were at work, one in the Philippines, two in Washington, and one in the UK at Bletchley Park.

The American gift of a PURPLE machine to the UK—revealing to a foreign power a vital state secret—was the first big step in establishing British–American co-operation and co-ordination in signals intelligence and cryptanalysis. That path, as Bradley F. Smith has shown in his book The Ultra-Magic Deals (Novato, Ca., 1993), was painfully slow and tortuous but in the end extremely successful and important in the prosecution of the war. Upon receipt of the PURPLE machine the British began intercepting and decrypting Tokyo's messages to and from its embassies and consulates in Europe and the Middle East. By June 1941 the British had received a second machine for Singapore.

One example to which these machines were put to use was to reveal the treachery of Burma's prime minister, U Saw. On his way back home from London, after unsuccessful talks about Burma's independence, he visited the Japanese consulate in Lisbon. He assured the consul-general that if the Japanese invaded Burma his people would rise against the British and help the Japanese drive them out (see also Thakin). The next day Tokyo was apprised of this conversation; the encoded signal was decrypted; and U Saw was arrested further along his journey and spent the rest of the war in internment.

The flow of decrypts continued until Japan's defeat. The Japanese never suspected that their most secret diplomatic cipher had been compromised. Some messages were deciphered and translated the same day and most within a week; a few in cases of key change took longer, one as long as 59 days. Shortage of translators, in particular those familiar with the forms used in official, telegraphic transmissions, caused delay.

MAGIC was, as the chief of staff of the US Army, Marshall, said, a ‘priceless asset’ for the USA and UK and extraordinary measures were taken to keep it secret. Indeed, these precautions were so protective, at least before Pearl Harbor, that they hampered effective use of the information. Horrified to find a copy of a MAGIC message in a White House wastebin, the army for a time struck the president off its list of recipients. MAGIC was treated with such secrecy that it was almost impossible to integrate it with other forms of intelligence. In fact before Pearl Harbor there was no national system for correlating and evaluating intelligence from different sources. By the end of the war the distribution system was systematic and comprehensive, the president and high officials receiving the daily ‘Black Book’, a digest of important MAGIC and ULTRA intelligence from British and American sources.

Although it revealed the imminence of war, MAGIC did not pinpoint Pearl Harbor or other objectives since Japanese diplomats were kept in the dark about military plans. However, a better organized American intelligence system might have been alerted by a message of 24 September 1941, not in PURPLE but a lesser cipher, asking the precise location of warships in Pearl Harbor. But distinguishing Japanese intelligence-gathering for an attack on Pearl Harbor from the mass of information sought by the Japanese on American naval activities throughout the Pacific would have been difficult at best. In the last hours before war, MAGIC did disclose the Japanese intention of breaking off negotiations in Washington and the particular hour this was to occur, 1300 in Washington, dawn in the Hawaiian Islands. Washington officials anticipated an attack somewhere and issued warnings, but missed the Hawaiian connection.

While MAGIC had limited operational value during the war, it was important in reinforcing American and British perceptions of Japanese aggressiveness. Intercepts of June and July 1941 gave an inside view of Japan's coercive diplomacy to secure military bases in southern French Indo-China. They also plainly indicated Japan's interest in further penetration of South-East Asia: Tokyo directed its consuls to find Japanese who knew Malaya, secure maps of the region, and gain information about the Netherlands East Indies beach defences and the camouflage markings of American planes at Manila. Typical as a preliminary to war were messages ordering Japanese consulates to destroy back files and cipher material. They indicated, too, plans for anti-American propaganda and espionage networks in Latin America and the recruitment of African Americans as spies.

MAGIC was also a rich source of intelligence on the European war. While the Japanese foreign ministry had limited access to information about its own military forces, and shared it sparingly with its missions abroad, traffic the other way was heavy: Japanese embassies and legations in German-occupied Europe, Berne, Lisbon, Stockholm, and Moscow provided a stream of information on the tide of battle and German capabilities and intentions. The military and naval attachés in these posts used their own codes but even these in time were decrypted.

The most valuable reports were those of Lt-General Ōshima the Japanese ambassador in Berlin. As principal representative of Germany's Axis partner, Ōshima had access to the highest German sources including Hitler himself, as well as to leaders of the Wehrmacht. The Germans were not overly generous in sharing secrets with their Japanese ally (see Axis strategy and co-operation), but neither could they leave Tokyo entirely in the dark, so the ambassador's reports were of vital interest to Washington and London.

In early June 1941 Ōshima reported that conversations with Hitler and Foreign Minister von Ribbentrop indicated, in all probability, an imminent German attack on the USSR (see BARBAROSSA). Although the massing of German forces in the east was impossible to conceal or ignore, British and American intelligence found it difficult to believe that Hitler would actually strike. Rather it was suspected that he was seeking to intimidate Stalin into making large territorial concessions. Ōshima's message was important, though by no means singular, in convincing the doubters of Hitler's real intention.

The course of the swaying tide of battle in the German–Soviet war was of intense interest to Japan, and Ōshima followed it closely. By way of his deciphered messages the western powers gained confirmation from Berlin that the German drive on Moscow in the autumn of 1941 was slowing down, and the following spring that the Soviet counter-offensive was ebbing. During the crucial battle for Stalingrad, word that Japan had rejected a German appeal to attack the USSR encouraged the view in London that the Soviets would hold out. Further examples are: a decrypt of August 1943 reflecting German pessimism during the great battle of Kursk; and another in January 1944, as the Allied invasion of France approached (see OVERLORD), dwelling on the difficulty Hitler saw in waging war on more than one front.

One of the most valuable contributions of MAGIC was the information Japan's Berlin and Vichy embassies provided about German defences and troop dispositions against the Allied invasion of France in June 1944. Reports by Ōshima and his naval attaché of tours of the defences in France gave details of the German command structure in the west, the number of divisions in each sector, the composition of the mobile reserve, the nature of the Atlantic Wall, and warning of underwater obstacles erected against landing craft. The Germans planned to defend at the beachline, said Ōshima, and smash any beachhead with their panzer reserve. As to where the Allied forces would land, and whether the landings in Normandy were to be the only ones, the Japanese confirmed German uncertainty right down to D-Day and beyond (see also XX-committee).

Also of great value were decrypts concerning German production, morale, and weaponry. By way of the Japanese, the Allies learned the characteristics of the new German Schnorchel-type U-boats, and specifications of their radio-guided, air-launched, rocket-propelled, anti-ship bombs (see guided weapons). The growing weight of the Allied strategic air offensive against Germany was also reflected in MAGIC decrypts. In August 1943 Ōshima told of German plans for increased fighter plane production to counter the raids. By June 1944 the embassy was describing daylight attacks as overwhelming. It reported more than once on severe and possibly fatal bomb damage to oil refineries and synthetic oil plants. The Berlin embassy also correctly predicted, more than once, a German counter-offensive in the west in late 1944, but not where, when, and in what strength (see Ardennes campaign).

MAGIC was only one of many intelligence sources available to the western Allies. It was not always respected or heeded: to some, the Japanese seemed gullible, taken in by German claims. Nevertheless, it provided an extraordinarily valuable supply of operational intelligence both in Europe and in waging the Pacific war. Its final gift was the revelation to the Americans of Japan's desperate and futile effort to secure Soviet mediation in ending the Pacific war, which clearly indicated Japan's insistence on maintaining the institution of Emperor Hirohito as a condition for peace.

Waldo Heinrichs

Bibliography

Heinrichs, W. , Threshold of War: Franklin D. Roosevelt and American Entry Into World War II (New York, 1988).
Komatsu, K. , Origins of the Pacific War and the Importance of ‘Magic’ (Richmond: Japan Library, 1999).
Lewin, R. , The American Magic: Codes, Ciphers and the Defeat of Japan (New York, 1982). Published in UK as The Other Ultra.
Smith, M. , The Emperor's Codes (London, 2000).