The Secret Sentry Read online

  The critical importance of the initial SIGINT effort was underlined by the events that unfolded in the next few years— the Berlin Crisis and subsequent Berlin Airlift (June 1948 through July 1949) in response to Russia’s attempt to cut off West Berlin from access by its former allies, the detonation of the first Soviet atomic bomb in August 1949, and the outbreak of the Korean War in June 1950. What Anglo-American code breakers could learn about Russian capabilities and intentions was frightening enough; what they could not learn about because too many Soviet codes proved resistant to solution was an even greater cause for worry. Clarke, Rowlett, their colleagues, and their successors found themselves on the front line of a secret and increasingly desperate struggle. And the U.S. military, which soon began drawing up plans for war with the Soviet Union, would find SIGINT even more vital than it was in World War II, largely because Russia (as well as its satellite nations and China) was highly resistant to penetration by human intelligence operations.

  * The designation of the Soviet intelligence and security service changed on numerous occasions. After the postrevolutionary Cheka, it became the State Political Directorate, or GPU (1922–1923); the United State Political Directorate, or OGPU (1923–1934); the Main Directorate for State Security, or GUGB (1934–1943); the People’s Commissariat for State Security, or NKGB (1943–1946); and the Ministry for State Security, or MGB (1946–1953). From 1953 to 1954, all intelligence and internal security functions were merged into the Ministry for Internal Affairs (MVD). Between March 1954 and October 1991, the principal Soviet intelligence and security service was the Committee for State Security (KGB). In October 1991, the KGB was dissolved following the collapse of the USSR and the abortive coup d’état against Mikhail Gorbachev.

  CHAPTER 1

  Roller-Coaster Ride The Travails of American Communications Intelligence: 1945–1950

  When troubles come, they come not as single spies but in battalions.

  —WILLIAM SHAKESPEARE HAMLET

  On August 14, 1945, the day Japan formally surrendered, the American signals intelligence empire stood at the zenith of its power and prestige. The U.S. Army and Navy cryptologic organizations, the Signal Security Agency (SSA) and the Naval Communications Intelligence Organization (OP-20-G) respectively, together consisted of more than thirty-seven thousand military and civilian personnel manning thirty-seven listening posts and dozens of tactical radio intelligence units around the world. The reach of America’s code breakers was extraordinarily deep, with the army alone able to read 350 diplomatic code and cipher systems belonging to sixty countries. Needless to say, the two American SIGINT organizations seemed to be in much better shape, both quantitatively and qualitatively, than the poorly funded three-hundred-man American cryptologic establishment that had existed when Japan bombed Pearl Harbor on December 7, 1941. 1

  Structural changes within army and navy COMINT organizations came quickly after the end of the war. On September 15, 1945, the SSA was redesignated as the Army Security Agency (ASA), which was given complete control over all U.S. Army COMINT activities.2On July 10, 1946, the U.S. Navy COMINT organization OP-20-G was deactivated and all navy COMINT intercept and processing units were merged into a new and much smaller organization called the Communications Supplementary Activities (CSA).3

  The Terrible Peace

  Within hours of Japan’s surrender, the thousands of American radio intercept operators and intelligence analysts around the world suddenly found themselves unemployed as the few remaining Japanese radio transmitters went off the air. Listening posts around the world were given “make-work” projects until the intercept operators could be discharged and sent home.4 The same was true at the army and navy SIGINT analysis centers in Washington, D.C.5

  President Harry Truman’s order for rapid demobilization after Japan’s surrender took its toll on America’s SIGINT capability. General Corderman was forced to dismantle the unit he had personally spent so much time and effort building, and he did so amid intense opposition from Army G-2 and his own top deputies, such as his operations chief, Frank Rowlett, who urged him to fight the demobilization order. Decades later, a still-angry Rowlett recalled that his boss “made a speech to them, and in essence what he said was, we’d like you to stay but here’s your hat.”6

  Over the next 120 days, the army and navy COMINT organizations lost 80 percent of their personnel.7Desperate last-minute efforts to convince the best and the brightest of the departing staff to stay on were to no avail. America’s SIGINT establishment would need many years to make up for the loss of so much talent and intellectual firepower.

  The same evisceration was taking place at all of the army’s and navy’s listening posts. By December 1945, the army’s and navy’s radio intercept efforts had shrunk to skeleton crews whose operational accomplishments were deteriorating rapidly. Even more worrisome, the radio traffic that the two U.S. COMINT organizations could access plummeted, since most of the foreign military communications traffic that the United States had been listening to was shifted from radio to landlines, and the volume of foreign diplomatic message traffic dropped back to normal peacetime levels.8

  There was now much less raw material for the few remaining American cryptanalysts to work on, which in turn led to a dramatic decline in the number of foreign code and cipher systems that were being exploited. In particular, work on South American, Balkan, and Chinese diplomatic codes and ciphers fell off sharply because of a lack of intercepts. Without the assistance of the British, U.S. efforts to maintain continuity coverage of Middle Eastern and Near Eastern communications traffic would have collapsed. By the end of 1945, the supply of radio intercepts had fallen to a point where code-breaking work had almost come to a completestandstill, including the joint Anglo-American operation code-named Bourbon, the intercepting and decoding of Soviet communications.9

  The Customers Complain

  During the months after the end of the war, the U.S. Army and Navy COMINT organizations were not producing much in the way of useful political intelligence. Among the few sensitive materials produced during this troubled time were decrypted telegrams concerning foreign work on atomic energy, such as a September 27, 1945, French message mentioning Norwegian heavy water supplies and a November 27, 1945, Chinese diplomatic message concerning Russian nuclear weapons research efforts; decrypted French foreign intelligence service message traffic; and messages that revealed secret U.S. diplomatic activities around the world that the British and other allies were not meant to be privy to, such as a December 2, 1945, Chinese diplomatic message concerning the planned construction of an American air base in Saudi Arabia.10

  Then there was the super-secret intercept program known as Operation Gold. In May 1946, two years before the creation of the state of Israel, the U.S. Navy COMINT organization began intercepting the international telephone calls and international cable traffic of Jewish agents in the United States and elsewhere who were engaged in raising money and buying arms for the Jewish underground in Palestine. According to a former army intelligence official, the Gold intercepts proved to be highly informative. “We knew who was shipping the arms, who was paying for them, who was being paid in this country, every illegal thing that was going on in this country.” But the official added, “Because of politics, very little was ever done with [this intelligence].” 11

  COMINT was also producing very little meaningful intelligence on foreign military targets. As of 1946, the Army Security Agency (ASA) was reading the encrypted military communications of Argentina, Czechoslovaki a, France, Romania, Spain, and Yugo slavia. Decrypts of Soviet military traffic were notable by their absence.12

  By January 1946, the quantity and quality of the intelligence reporting coming from COMINT had fallen to such a low level that the director of naval intelligence, Rear Admiral Thomas Inglis, wrote that “we have been getting disappointingly little of real value from [communications intelligence] since VJ day.”13

  Complaints from intelligence consumers about the d
earth of intelligence coming from COMINT were rampant. For example, on December 22, 1945, former U.S. Army chief of staff General George Marshall went to China in a foredoomed effort to broker some sort of deal between Chiang Kaishek and Mao Tse-tung. No useful decrypts were available to offer any insight into the thorny problems confronting Marshall, and only months later did the army begin producing the first useful translations of intercepted Chinese Nationalist and Chinese Communist communications.14

  Yet the harshest criticism coming from customers was over the paucity of intelligence about what was going on inside the Soviet Union. A Senior U.S. Army officer who visited Eu rope in the spring of 1946 was told that it was unlikely that Washington would get any kind of meaningful advance warning of a Soviet attack on Western Eu rope because of a near total lack of reliable intelligence about “the main enemy.”15

  The BRUSA Agreement

  Thus the American COMINT establishment desperately needed help from somewhere in order to remain a viable intelligence provider. As it turned out, relief for the battered U.S. COMINT community was to come from across the Atlantic.

  On March 5, 1946, former prime minister Winston Churchill, at Truman’s invitation, delivered his famous speech in Fulton, Missouri, in which he warned, “From Stettin in the Baltic to Trieste in the Adriatic, an Iron Curtain has descended across the continent.” The “informal” war time arrangements for cooperation between American and British COMINT organizations were formalized on the same day. At almost the exact same time that Churchill was delivering his memorable speech, in a heavily guarded conference room in downtown Washington, D.C., a group of Senior American and British intelligence officials were signing a seven-page Top Secret intelligence-sharing agreement called the British–United States Communication Intelligence Agreement, which was referred to within the U.S. intelligence community as the BRUSA Agreement. This may be one of the most important and longest-lasting agreements among foreign intelligence services ever conceived. The product of six months of intense and often acrimonious negotiations, the agreement recognized that given the “disturbed” condition of the world, the American and British COMINT organizations needed to continue to work together in order to monitor the broad array of new threats, especially the Soviet Union.16

  In its final form, rather than being a blueprint for action, BRUSA was a general statement of principles meant to “govern the relations” of the United States, Britain, and the British Dominions “in communication intelligence matters only.”17Contrary to what has previously been written about it, it was strictly a bilateral agreement between the United States and Great Britain thatstandardized the day-to-day collaboration between the two countries’ SIGINT organizations. There was to be a complete and free exchange of all forms of communications intelligence “product” between the U.S. organizations and the British cryptologic organization, the Government Communications Headquarters (GCHQ). Both the U.S. Army and Navy COMINT organizations were required under the terms of the BRUSA Agreement to send one copy of every finished COMINT report (excepting those deemed to be specifically exempt from the intelligence-sharing agreement) to GCHQ, and vice versa. There was also a sidebar agreement between the Americans and the British for cryptanalytic cooperation on selected intelligence problems, such as the continuation of the joint efforts involving Russian and French ciphers. Other key provisions of the BRUSA Agreement established procedures governing the two nations’ handling, safekeeping, and exchange of COMINT.18

  America’s other English-speaking war time SIGINT allies— Canada, Australia, and New Zealand—were referenced, but not included as signatories. BRUSA recognized that thesenations, as British Dominions, would continue to operate under the overall direction of the British SIGINT agency GCHQ. Were the United States to make arrangements with the SIGINT organizations of these countries, BRUSA required that Britain be informed ahead of time, which in effect meant that London had to agree to the arrangements and could nix them at any point. It was to take eight more years and thousands of hours of further negotiations before BRUSA would finally morph, in 1954, into what is now known as the United Kingdom–United States (UKUSA) Agreement.19

  The first of the Dominion countries that the United States sought to establish bilateral SIGINT relations with was Canada. During World War II, the U.S. Army and Navy COMINT organizations had maintained close relations with their Canadian counterparts, although the level of cooperation between the two countries never came close to approaching the intimacy that characterized the Anglo-American COMINT relationship. After the end of the war, U.S. and Canadian officials held some preliminary discussions about continuing their war time COMINT collaborative relationship. But on September 5, 1945, a twenty-six-year-old Russian cipher clerk by the name of Igor Gouzenko walked out the door of the Russian embassy in Ottawa and after many adventures succeeded in defecting to Canada. Information provided by Gouzenko helped the Royal Canadian Mounted Police identify seventeen spies working for the Soviet military intelligence service, the GRU, in Canada and Britain.20The sensational revelations stemming from the Gouzenko spy scandal— that the Russians had an agent network inside the Canadian government—naturally made U.S. intelligence officials extremely wary about restoring their cryptologic relationship with the Canadians. The result was that in October 1945 U.S. intelligence officials broke off their talks with their Canadian counterparts, with the head of the U.S. Navy COMINT organization, Captain Joseph Wenger, telling his Canadian counterpart, “The whole matter is awaiting a high policy decision so, of course, nothing can be done until this is settled.”21

  The talks resumed in mid-1946 but essentially went nowhere until a series of compromises were reached that permitted the Canadian government to agree to the terms of the CANUSA COMINT Agreement, signed in November 1949.22

  Reaching an agreement that included the rather small Australian SIGINT organization was complicated because of mounting evidence emanating from the Venona intercepts (to be discussed later in this chapter), which strongly indicated that Soviet intelligence had spies inside the Australian government who were feeding Moscow highly classified documents concerning Anglo-American defense matters. In January 1948, the U.S. government cut off the Australian government from access to all American classified information, and the American COMINT organizations were specifically barred from cooperating with their Australian counterparts in any way. Only after a new conservative Australian government headed by Robert Menzies was elected in December 1949 did the U.S. government relent and resume SIGINT collaboration with Australia on a limited basis, in 1950, after it was clear that the Soviet spies inside the Australian government had been removed. Australia was not admitted to BRUSA until three years later, in September 1953. In May 1954, the BRUSA Agreement was renamed the UKUSA Agreement so as to reflect the addition of Australia and New Zealand as full members of the global Anglo-American SIGINT enterprise.23

  A Brief Shining Moment: The Break Into the Soviet Ciphers

  Almost immediately after the signing of the BRUSA Agreement, the U.S. intelligence community’s knowledge about what was transpiring inside the USSR began to improve, as the joint Anglo-American code-breaking enterprise—Bourbon—made dramatic progress solving a number of Soviet cipher systems.24

  The British end of Bourbon was run from a motley, drab collection of buildings hidden behind high walls in the nondescript London suburb of Eastcote, which was the new home of the GCHQ. (Better quarters would later be established in the somewhat more balmy climate of Cheltenham.)25

  The man who ran the British end of the Bourbon project was the head of the 140-man GCHQ Russian Cryptographic Section, Richard Pritchard.26 Pritchard, who had managed the secret British cryptanalytic attack on Russian codes and ciphers during World War II, was one of those rare people blessed with multiple gifts. He had extraordinary mathematical talent and a genius for music, and he was a natural cryptanalyst to boot. F. W. Winterbotham, author of The Ultra Secret, described Pritchard as “young, tall, clean-shaven, rathe
r round of face, with a quiet voice, could talk on any subject with witty penetration. He, too, was deeply musical.”27

  Pritchard assembled a small but remarkably talented group of veteran code breakers to work on Bourbon, the two most important of whom were Conel Hugh O’Donel Alexander, an extraordinarily gifted cryptanalyst and former British chess grand master, and Major Gerry Morgan, a brilliant machine cryptanalyst and the head of GCHQ’s Crypto Research Section, which contained the best of the British cryptanalysts who had chosen to remain on in government service after the war.28

  The level of “customer satisfaction” would soon begin to rise rapidly. In the span of only a year, teams of code breakers on both sides of the Atlantic accomplished an astounding series of cryptanalytic breakthroughs that, for an all-too-brief moment in time, gave the leaders of the United States and Great Britain unparalleled access to what was going on inside the Soviet Union, especially within the Russian military.

  In February 1946, less than a month before the signing of the BRUSA Agreement, ASA cryptanalysts at Arlington Hall Station in Virginia managed to reconstruct the inner workings of a Soviet cipher machine that they called Sauterne, which was used on Red Army radio networks in the Far East. On March 1, 1946, a veteran U.S. Army cryptanalyst at Arlington Hall named Robert Ferner managed to produce the first decrypted message from a Sauterne intercept. By the end of the month, U.S. Navy cryptanalysts had discovered a means of determining the daily rotor settings used to encipher all messages on the Sauterne cipher machine, with the result that on April 4, 1946, a regular supply of Sauterne decrypts began to be produced.29The translations of the Sauterne decrypts provided a window into what the Russian army was up to in the Far East.30