The Secret Sentry Read online

  The Secret Sentry

  THE SECRET SENTRY

  The Untold History of the

  National Security Agency

  MATTHEW M. AID

  Copyright © 2009 by Matthew M. Aid

  All rights reserved. No part of this book may be used or reproduced in any manner whatsoever without written permission from the publisher except in the case of brief quotations embodied in critical articles or reviews. For information address Bloomsbury Press, 175 Fifth Avenue, New York, NY 10010.

  Published by Bloomsbury Press, New York

  All papers used by Bloomsbury Press are natural, recyclable products made from wood grown in well-managed forests. The manufacturing processes conform to the environmental regulations of the country of origin.

  LIBRARY OF CONGRESS CATALOGING-IN-PUBLICATION DATA

  Aid, Matthew M., 1958–

  The secret sentry : the untold history of the

  National Security Agency / Matthew M. Aid.

  p. cm.

  Includes bibliographical references and index.

  eISBN: 978-1-60819-179-6

  1. United States. National Security Agency— History. 2. Intelligence service— United States. 3. Electronic surveillance—United States. 4. United States— History—1945– I. Title. II. Title: Secret sentry, the untold history of the NSA. III. Title: Untold history of the National Security Agency.

  UB256.U6A53 2009

  327. 1273—dc22

  2008037442

  First U.S. Edition 2009

  1 3 5 7 9 10 8 6 4 2

  Typeset by Westchester Book Group

  Printed in the United States of America by Quebecor World Fairfield

  To Harry, Rita, and Jonathan Aid

  My Family, My Best Friends, and My Staunchest Supporters

  Gratis eternum

  Know your enemy and know yourself, find naught in fear for 100 battles.

  Know yourself but not your enemy, find level of loss and victory.

  Know thy enemy but not yourself, wallow in defeat every time.

  —SUN TZU

  There are no secrets except the secrets that keep themselves.

  —GEORG BERNARD SHAW,

  BACK TO METHUSELAH

  Contents

  Prologue

  The Origins of the American Cryptologic

  Effort Against Russia

  1. Roller-Coaster Ride:

  The Travails of American Communications

  Intelligence: 1945–1950

  2. The Storm Breaks:

  SIGINT and the Korean War: 1950–1951

  3. Fight for Survival:

  The Creation of the National Security

  Agency

  4. The Inventory of Ignorance:

  SIGINT During the Eisenhower

  Administration: 1953–1961

  5. The Crisis Years:

  SIGINT and the Kennedy Administration:

  1961–1963

  6. Errors of Fact and Judgment:

  SIGINT and the Gulf of Tonkin Incidents

  7. The Wilderness of Pain:

  NSA and the Vietnam War: 1964–1969

  8. Riding the Whirlwind:

  NSA During the Johnson Administration:

  1963–1969

  9. Tragedy and Triumph:

  NSA During the Nixon, Ford, and Carter

  Administrations

  10. Dancing on the Edge of a Volcano:

  NSA During the Reagan and Bush

  Administrations

  11. Troubles in Paradise:

  From Desert Storm to the War on Terrorism

  12. Snatching Defeat from the Jaws of Victory:

  9/11 and the Invasion of Afghanistan

  13. A Mountain out of a Molehill:

  NSA and the Iraqi Weapons of Mass

  Destruction Scandal

  14. The Dark Victory:

  NSA and the Invasion of Iraq:

  March–April 2003

  15. The Good, the Bad, and the Ugly:

  SIGINT and Combating the Insurgencies

  in Iraq and Afghanistan

  16. Crisis in the Ranks:

  The Current Status of the National Security

  Agency

  Acknowledgments

  Notes Glossary

  Notes

  PROLOGUE

  The Origins of the American Cryptologic Effort Against Russia

  Another man’s soul is darkness. Does anybody ever really know anybody else?

  —RUSSIAN PROVERB

  The consensus of historians (and the overwhelming burden of evidence) dates the initial stages of the Cold War to well before the end of World War II. The United States would emerge from the war as a superpower with arguably the world’s strongest armed forces, sole possession of the atomic bomb, a vastly expanded industrial base, and an infrastructure untouched by the ravages of war. But on the negative side, the country had at best a rocky relationship with one of its war time allies, the Soviet Union. By the time Nazi Germany and Japan had surrendered, Russia was on a collision course with both the United States and Britain. It was not long before the Soviet Union was regarded as “the main enemy” by the Western nations. Since it remained a rigidly closed society under Joseph Stalin’s regime, the lack of transparency was a major factor driving the Cold War. Because the United States had only a very limited idea of what was going on in the Soviet Union, its satellite countries in Eastern Eu rope, and communist China, the emerging confrontation became all the more dangerous. But one of the most secret resources that had greatly contributed to the victory of the Allied Powers—the United States and Britain’s ability to intercept and read the communications of our former enemies Germany, Japan, and Italy, both in the clear and encoded—would be quickly redirected to the task of gathering communications intelligence about the new Sino-Soviet threat.

  It is difficult to imagine, many decades later, just how mortal that threat was perceived to be, particularly after the Soviet Union detonated its first atomic device in the summer of 1949. The prospect of a “nuclear Pearl Harbor” meant that the United States would rely heavily on an increasingly large and expensive communications intelligence effort.

  Carter Clarke Declares War on Russia

  In a certain sense, Brigadier General Carter Clarke was the founding father of the National Security Agency (NSA). A blunt, often profane, hard-drinking, and demanding individual, Clarke lacked the polish of his fellow officers who had gone to West Point. He began his career as an enlisted man and worked his way up through the ranks. Despite a lack of previous intelligence experience and a file drawer full of bad fitness reports (Clarke was a real maverick), he was the man the U.S. Army selected to run the analytic side of SIGINT Army G-2, the Special Branch. A college dropout (he joined the army and served under General John Pershing chasing Pancho Villa in Mexico), he was a highly intelligent man and an autodidact.

  Clarke was described by many who worked with him as being a tough, impatient, no-nonsense workaholic who abhorred conformity and was intolerant of bureaucracy. When things did not get done to his satisfaction, Clarke’s volatile temperament usually took over. Former colleagues recall that his temper tantrums were legendary. A former army officer said, “I knew that Clarke had an explosive temper. Although quite a decent person, he laced his language with frequent bursts of profanity.” His detractors, who were many, described him as loud, uncouth, brash, and argumentative, with a tendency toward overstatement when trying to make a point or win an argument. And yet, despite his brashness, gruff talk, and stern demeanor, Clarke earned the respect (and fear) of virtually all the U.S. Army intelligence officials he dealt with. A former senior NSA official, Frank B. Rowlett, described Clarke as “a very unconventional man and a man of considerable moral courage [who]
would spit in your face and laugh at you.”1

  Clarke’s Special Branch was a component of Army G-2 in the Pentagon created after Pearl Harbor, the unit to which all intercepts were sent for analysis and reporting to consumers. It only worked on SIGINT materials, while the rest of Army G-2 worked on more mundane materials, like military attaché reports. The army’s SIGINT organization, the Signal Security Agency (SSA), commanded by Brigadier General W. Preston Corderman, was a separate field agency that was (until 1944) part of the Army Signal Corps. As noted above, all its intercept material went to Clarke’s G-2 Special Branch.

  When Clarke took command of the Special Branch of Army G-2 (intelligence) in May 1942, the United States was able to read the top Japanese diplomatic and military encoded communications (which enabled U.S. forces to win the Battle of Midway in 1942, the turning point of the war in the Pacific) and the British were reading the German codes generated by the Enigma machine. Despite his rough edges, Clarke worked well with his British counterparts in the Bletchley Park code-breaking center. Deep down, however, he trusted no man and no nation. According to Rowlett, “Clarke was a good man to have in the intelligence business in our line of command [the communications intelligence, or COMINT, field] because he didn’t trust any nation. He just said, ‘They’re your friends today and they’re your enemies tomorrow, and when they’re on your side find out as much as you can about them because you can’t when they become your enemy.’ ”2

  The United States was not only reading the codes of the three Axis Powers; it was reading the encrypted diplomatic and military traffic of more than forty other countries—including our allies and neutral states. Well before the end of the war, Clarke, like many in the American military and government, decided that the Soviet Union would become our next “main enemy” after the war, and he issued an order in January 1943 to begin cracking Russian codes. So secret and delicate was this operation that very few people were allowed to even know it existed, and since virtually nothing was put in writing, the paper trail today is virtually non existent. The U.S. Navy had its own code-breaking operation headquartered in Washington. Though the two cryptanalytic organizations shared code-breaking responsibilities, cooperation was the exception rather than the rule.3

  The army code-breaking operation was headquartered in a former girls’ preparatory school named Arlington Hall, located in Arlington, Virginia. The main building on its large and beautifully landscaped campus housed the administrative offices. Tacked onto it, once the army took over and fenced it off from the world, were two wings that housed large open bays crammed with code breakers, linguists, and analysts, crowded together and forced to endure the scorching and humid Washington summers before the widespread use of air-conditioning. Hundreds of fans provided some relief— but unfortunately they blew working papers all over the place. The sole air-conditioning was reserved for the noisy and noxious IBM tabulating machines.4

  Clarke had some supervisory authority over Arlington Hall Station (its official designation), but he largely worked out of a high-security area in the Pentagon. The intercepts of enemy communications that were picked up by a far-flung network of listening posts, some of them in remote areas like Ethiopia and Alaska, went to Arlington Hall, where they were decrypted and translated. Then they were sent on to Clarke’s analytic organization. The intelligence product derived from intercepts was so sensitive that its distribution was extremely limited, reaching only a few hundred people with the highest security clearances. The paradox here is that in order to protect the sources and methods used to gather this invaluable signals intelligence (SIGINT) and not tip off the enemy that the United States was reading virtually all of its communications, the intelligence product often had to be “sanitized” (i.e., put in a form that would not disclose the source of the intelligence reporting) and sometimes did not reach those who needed it most. (Both Admiral Husband Kimmel and General Walter Short, who took the burden of blame for Pearl Harbor, were arguably deprived of information that could have made the events of December 7, 1941, a very different story.) Throughout the war, commanders in the field below a certain level of rank and responsibility were not furnished with this critical information, or got it in a very watered-down form, which tended to make the material not as useful as it should have been, particularly because these officers could not know just how definitive and reliable it was. The same complaints that were voiced back then are still heard today.

  Because the British had developed a formidable code-breaking operation that was in many ways superior to the Americans’, once the United States entered the war there was an almost complete sharing of information and coordination of efforts. But the British were not apprised of the U.S. attack on Russian codes. In any event, they were undertaking their own effort, which they also did not disclose to the United States.5

  Well before Germany, Japan, and Italy surrendered, the Cold War was under way, setting our quondam ally, the Soviet Union, on a collision course with the United States, Great Britain, and, in time, the other nations that would become the North Atlantic Treaty Organization (NATO). Accordingly, before Germany surrendered, the United States and the United Kingdom decided that everybody’s cards had to be put on the table. Prime Minister Winston Churchill and his commanders (particularly Brigadier General Sir Stewart Menzies, the head of the British spy agency MI-6) firmly believed that a concerted effort had to be made to penetrate what Churchill described as a “riddle wrapped up inside an enigma”—the essentially closed society of the Soviet Union. This belief was shared by General George Marshall, Admiral Ernest King, and just about everybody at senior levels of the U.S. government and military— with one exception, President Franklin Delano Roosevelt. FDR wistfully believed that the United States and Russia could “peacefully coexist” after the Allied victory. So it was decided that he not be informed that we were spying on our Russianally. The Russians, of course, were doing the same thing to the United States and Britain and, unfortunately, as we know now, doing a much better job. The full extent of Russian espionage was made clear when we began to read their enciphered messages. One key early break-through came in October 1943, when a thirty-seven-year-old lieutenant named Richard Hallock, who before the war had been an archaeologist at the University of Chicago, made the first break into the Russian ciphers. Incredibly, the Soviets had reused the pages of their one-time pad cipher keys on a number of occasions in different kinds of message traffic.6

  (A “one-time pad” used to encipher messages is a bound set of sheets, each one printed with randomly generated numbers—representing both words and numbers— organized as additive “keys” and a certain number of lines of numbers in separate “groups.” No one sheet in a pad and no pad or set of sheets duplicates any other, except for the matching pad’s sheets used for deciphering the encoded message. The sheets are to be used once only and then destroyed. If used properly, the pad provides a virtually unbreakable code.)

  The German invasion of Russia in June 1941 and the chaos that followed had created a severe shortage of cipher materials at Russian overseas diplomatic establishments, leading the NKVD’s* cryptographic department in Moscow, which produced all code and cipher materials, to take shortcuts to fill the increasing demand for cryptographic materials. As the German army drew ever closer to Moscow in the winter of 1941, the Russians apparently panicked, printing duplicates of twenty-five thousand pages of one-time pad keys during the first couple of months of 1942, then binding them into onetime pad books and sending them not only to their diplomatic and commercial establishments, but also to the various NKVD rezidenturas (or “stations”) around the world, thus unwittingly compromising the security of all messages encrypted with these duplicated pads. Then, to make matters worse, the Russians could not get new cipher materials to their diplomatic establishments in the United States and elsewhere because of German U-boat activity in the North Atlantic, which hampered Soviet merchant shipping traffic between Murmansk and the United States.7

  SIGIN
T Comes of Age

  Beginning in early 1943, the U.S. Army’s SIGINT collection effort slowly began to shift from Axis military communications targets to the pre–Pearl Harbor focus on foreign diplomatic communications traffic, largely because of dramatic changes taking place in the global geopolitical balance of power, with the United States rapidly emerging as the world’s top superpower. Senior U.S. government and military policy makers and intelligence officers alike fully understood that while military decrypts (Ultra) might be helping win World War II on the battlefield, diplomatic COMINT (Magic) would be essential to help the U.S. government “win the peace.” There was a determination within the U.S. government that this time around America would not be bullied or manipulated by its now less powerful European allies or the Russians at the peace talks that would inevitably follow the end of the war. It would soon become clear that Clarke’s suspicions about Soviet long-term intentions were not only widely shared by others in the military and the government—they would also become key factors in how the nations of the West would respond to and then counter Russia’s postwar strategy.8

  To achieve these goals, however, the United States had to become as self-sufficient as possible in the realm of SIGINT. This meant that it had to put some distance between itself and Great Britain and begin spying on those countries or organizations that might conceivably constitute a threat in the future. The secrecy of the Russian effort was particularly intense. When Corder-man inquired whether Russian traffic had been deliberately omitted from a target list just received by his agency, he was told that “[reference to] Russian traffic was intentionally omitted with Clarke’s approval.”9But the accumulating intercepts of Russian traffic from 1943 on would yield one of the greatest U.S. COMINT harvests ever—the program code-named Venona. Begun immediately after the end of World War II, the decoding and analysis would stretch over many, many years (until the program formally ended in 1980). Venona material gradually and retrospectively revealed the astounding extent of Soviet intelligence activity in America and Mexico. (Among other things, it made clear why Stalin was not surprised by Truman’s carefully vague reference to the atomic bomb at Potsdam.) As we will see, the ultimate irony was that Venona’s access was so valuable that it could not be compromised by using the material gathered as evidence (or even for counterintelligence measures) against those Soviet sources (and methods) revealed by decryption over many years.