History of the Internet: Part 20 – Global Surveillance: The Hayden Years

Welcome back to the history of the internet. In the last few episodes, we have been looking at a variety of issues regarding the freedoms we have in today’s online world. In “Part 17 – Surveillance, Cryptography, and Free Speech,” we learned that the internet was created at a time of great social and political change. 

The actions of whistleblower Daniel Ellsberg and its coverage by The New York Times and The Washington Post revealed systematic deceit from the Johnson Administration. Investigations into practices and scandals within the government intelligence agencies led to the Foreign Intelligence Surveillance Act (FISA) of 1978, which provided judicial and congressional oversight of covert surveillance activities.

In “Part 18 – Free as in Freedom,” we learned about the beginning of the Electronic Frontier Foundation (EFF) and the Free Software Movement. The EFF played a significant role in pressuring the government to soften its rules on the export of encryption, as we learned in “Part 19 – Bernstein Vs. United States.” 

In this episode, we cover the years 1998 through 2006, including the epic challenges faced by NSA Director Lt. Gen. Michael Hayden and the effects of national security concerns on people’s rights to online privacy.


In the fall of 1998, the civil liberties group Free Congress Foundation (now known as the conservative think tank American Opportunity) sent a report to Congress warning of the dangers of a secret mass-surveillance program called Echelon.

In May 1999, the U.S. House of Representatives debated a new Intelligence Authorization Act. Republican congressman Bob Barr warned that Echelon “engages in the interception of literally millions of communications involving United States citizens over satellite transmissions, involving e-mail transmissions, Internet access, as well as mobile phone communications and telephone communications.”

Barr proposed an amendment requiring the NSA to provide details of Echelon to the House Permanent Select Committee on Intelligence. Republican chairman Porter Goss agreed, stating “It is intolerable to think of the United States Government, of big brother, or anybody else invading the privacy of an American citizen without cause,” and Dan Burton of Indiana announced hearings into the secret program.

Although officials from Britain and the United States continued to insist that rumors were false, on Nov. 3, 1999 the BBC reported “confirmation from the Australian Government that such a network really does exist and politicians on both sides of the Atlantic are calling for an inquiry.”

Following a yearlong inquiry into allegations of a spy system being used to gather Europe’s sensitive industrial secrets, in May 2001, European MPs published a report into the network of spy stations called ECHELON, concluding that this worldwide spy network was operated by the intelligence agencies of the United States, United Kingdom, Canada, Australia, and New Zealand. European MPs said Britain’s participation in Echelon could be a breach of the European Convention on Human Rights.

The U.S. government responded by again asserting that Echelon did not exist. The U.K. government gave no substantive commentary on this issue. Despite the repeated denials, the report’s conclusions were correct, and this massive surveillance system continues to exist today.

In April 2001, the book Body of Secrets by James Bamford revealed new information about this secret project. The author posed a question to give readers pause for thought: “The real issue is whether Echelon is doing away with individual privacy, a basic human right.” This issue would become highly political in the months and years to come. 

Facing the Al-Qaeda Threat

By 1999, the U.S. Intelligence services fully recognized al-Qaeda as one of the most serious threats to U.S. national security, and Director of Central Intelligence George Tenet’s plan to counter this threat involved new leadership at the CIA’s Counterterrorism Center (CTC).

George John Tenet, DCI July 1997 to July 2004

The top job at the NSA was held by Lt. Gen. Michael Hayden, who was employed as Director in March 1999 after serving as the commander of the Air Intelligence Agency.

global surveillance
Lt. Gen. Michael Hayden

During this time, there were a number of squabbles between the NSA and the CTC over the role each group played in the analysis of intercepts. NSA analysts believed Tenet was always favoring his own appointees over them.

The most critical pieces of Intelligence that were intercepted by the NSA in this year were the phone calls of Osama Bin Laden. The NSA was able to gain the names of two suspected al-Qaeda terrorists in Yemen and passed on the first names of brothers “Khalid” and “Nawaf” to the FBI and the CTC. The NSA previously intercepted a message with the full name “Nawaf al-Hazmi,” but the task of identifying the individuals was left to the FBI and the CTC. 

Tragically, it was not recognized for some time that Nawaf al-Hazmi had been granted a visa to the United States. Poor relations between the CIA and the FBI obstructed their efforts to protect the United States. The CIA received a copy of Khalid’s passport from the United Arab Emirates in January 2000, learning that he was heading to the United States. However, they assumed the next attack would be outside of the United States and failed to alert the FBI or the State Department for inclusion on its terrorist watch list. 

The NSA continued to intercept communications from the terrorists while they lived in San Diego, without notifying the CIA, the FBI, or NSA Director Lt. Gen. Michael Hayden that the suspected terrorists were already living in the United States.

Another major crisis in January 2000 was that an outdated routing protocol caused a major system crash at Fort Meade. The NSA was forced to rely on the British Agency GCHQ to provide its customers with vital information while they got the system back up and running. A $1.5 million investment was required to get the network fully up to date.

Perhaps Hayden’s biggest challenge, however, was public relations. The Hollywood film Enemy of the State depicted the Agency as a lawless and even murderous group.

global surveillance
Theatrical poster for Enemy of the State,
Copyright © 1998 by Buena Vista Pictures. Fair use

Hayden testified to the House Intelligence Committee on April 12, 2000. Unaware that several al-Qaeda members were already in the USA, he presented a farcical picture of the NSA having their hands feebly tied by James Madison, telling the Committee: 

“If, as we are speaking here this afternoon, Osama bin Laden is walking across the Peace Bridge from Niagara Falls, Ontario, to Niagara Falls, New York, as he gets to the New York side, he is an American person. And my agency must respect his rights against unreasonable search and seizure, as provided by the Fourth Amendment to the Constitution.”

Two al-Qaeda members were Mohamed Atta and Ramzi Binalshibh, who communicated with each other via mobile phones and emails. Although the communications were intercepted, they seemed innocent enough, as they spoke as students who were interested in different fields of study including architecture, arts, law, and politics.

On Aug. 21, 2001, some good analysis work by FBI intelligence specialist Margarette Gillespie revealed that two suspect terrorists, Hamza and Mihdar, were loose in the USA. By Aug. 27, the FBI made an urgent information request to Hayden, who approved it immediately.

Unfortunately, these suspects were never added to the domestic terrorist list. On Sept. 11, 2001, 19 suicidal maniacs were allowed to board four Boeing aircraft carrying knives and bolt cutters, with each plane carrying hundreds of other passengers and thousands of gallons of high octane fuel. 

The Aftermath

During the month of September 2001, the world saw examples of the worst, and best, of humanity. 

Police and rescue workers from around the country took leaves of absence from their jobs and traveled to New York City to help out in any way they could. Citizens donated $1.4 billion for the victims. 

Nations around the world lined up in solidarity with the United States. In France, Le Monde’s front page headline was “NOUS SOMMES TOUS AMÉRICAINS.” In Italy’s Corriere della Sera, “The distance from the United States no longer exists because we, our values, are also in the crosshairs of evil minds.”

Muslim Americans condemned the attacks, and President Bush gave a speech at the Islamic Center of Washington on Sept. 17 acknowledging the “incredibly valuable contribution” that millions of American Muslims made to their country.

After the Taliban’s refusal to punish al-Qaeda for their crimes, Article 5 was invoked on Oct. 4, 2001, for the first time in NATO’s history, affirming “an armed attack against one or more of them in Europe or North America shall be considered an attack against them all.”

NSA staff reviewed the communications between Atta and Binalshibh and were now able to decipher their true horrific meaning: “architecture” meant the World Trade Center, “arts” was the Pentagon, “law” was the Capitol, and “politics” was the White House.

The Patriot Act

Earlier in 2001, a new Anti-Terrorism Act was debated but was not passed into law due to concerns that it might be too intrusive and restrict civil liberties.

In response to the Sept. 11 attacks and the anthrax attacks shortly thereafter, Congress quickly reevaluated. Attorney General John Ashcroft put Congress under immense pressure by warning that further terrorist acts were imminent and that Congress could be to blame for such attacks if it failed to pass the bill immediately. Both the House and the Senate then passed the Uniting and Strengthening America by Providing Appropriate Tools Required to Intercept and Obstruct Terrorism Act of 2001, best known by its acronym: the USA PATRIOT Act. This Act significantly expanded law enforcement’s authority to surveill and capture communications.

In John Podesta’s book, “USA Patriot Act – The Good, the Bad, and the Sunset,” he explains:

“The events of September 11 convinced … overwhelming majorities in Congress that law enforcement and national security officials need new legal tools to fight terrorism. But we should not forget what gave rise to the original opposition—many aspects of the bill increase the opportunity for law enforcement and the intelligence community to return to an era where they monitored and sometimes harassed individuals who were merely exercising their First Amendment rights.”

President Bush held a congratulatory White House press conference saying the Act was a balanced piece of legislation that protected the constitutional rights of all Americans.

The Inquiry

Lt. Gen. Michael Hayden gave a statement to the Select Senate Committee on Intelligence on Oct.17, 2002, where he said “Sadly, NSA had no SIGINT suggesting that al-Qa’ida was specifically targeting New York and Washington, D.C., or even that it was planning an attack on U.S. soil. Indeed, the NSA had no knowledge before September 11th that any of the attackers were in the United States.”

He made a number of arguments about the difficulty of the job that the NSA did, blaming 1990s budget cuts while the number of online communications was rocketing, and saying that it didn’t have enough analysts and linguists. 

Hayden asserted that the NSA’s post 9/11 performance had been “sustained excellence” but also that it had “a long way to go” in keeping up with new technology.

Hayden also referred back to the Church inquiry and how criticisms of ECHELON had made his job harder:

“This is one of the few times in the history of my Agency that the Director has testified in open session about operational matters. The first was in the mid-1970s when one of my predecessors sat here nearly mute while being grilled by members of Congress for intruding upon the privacy rights of the American people. Largely as a result of those hearings, NSA is governed today by various executive orders and laws and these legal restrictions are drilled into NSA employees and enforced through oversight by all three branches of government.

The second open session was a little over two years ago and I was the Director at that time. During that session the House intelligence committee asked me a series of questions with a single unifying theme: How could I assure them that I was safeguarding the privacy rights of those protected by the U.S. Constitution and U.S. law? 

During that session I even said—without exaggeration on my part or complaint on yours—that if Usama bin Laden crossed the bridge from Niagara Falls, Ontario to Niagara Falls, New York, U.S. law would give him certain protections that I would have to accommodate in the conduct of my mission. And now the third open session for the Director of NSA: I am here explaining what my Agency did or did not know with regard to 19 hijackers who were in this country legally.”

Hayden invited the Select Senate Committee to “find out where the American people want that line between security and liberty to be.” 

He finished by arguing, “If we fail in this effort by drawing the line in the wrong place, that is, overly favoring liberty or security, then the terrorists win and liberty loses in either case.”

Warrantless Surveillance

global surveillance

At the time that Hayden invited a discussion on the line between liberty and security, the public had no way of knowing that the issue had already been partly decided by the White House.

According to James Bamford, shortly after Bush was elected President, Hayden gave him a top secret transition book covering the challenges of and limitations on the NSA. Hayden saw an advantage in putting taps on the USA’s main fiber optic cables but was concerned about the legality of doing so. 

According to President Bush, after 9/11 he asked Hayden, “Mikey, is there anything more we could be doing, given the current laws?” Hayden described a plan to monitor calls from between the United States and another country, where at least one target was suspected to be affiliated with al-Qaeda. With regard to the law, Hayden argued that the FISA Act was outdated and needed modernising for the 21st century. The bureaucracy of applying for FISA warrants was wasting too much precious analyst time.

President Bush authorized the President’s Surveillance Program in 2001, which included the Terrorist Surveillance Program, a secret order for the NSA to conduct warrantless eavesdropping on American citizens.

When questioned about the controversy, Bush replied: “I fully understand people’s concerns about it, but ours is a town, by the way, in Washington, where when you don’t connect the dots, you’re held up to Congress, and when you do connect the dots, you’re held up to Congress. I believe what I’m doing is constitutional, and I know it’s necessary. And so we’re going to keep doing it.“

The presiding judge on FISA court, Royce Lamberth, later opined, “We have to understand you can fight the war and lose everything if you have no civil liberties left when you get through fighting the war.”

According to Hayden, in practice the warrantless surveillance system meant between 5 and 7 thousand overseas persons were targeted on any given day, with about 500 targets located in the U.S. at the start of the program. The number of U.S.- based targets fell to about 100 by 2007. Hayden said the system would have almost certainly identified the al-Qaeda terrorists based in San Diego if the program had been in effect before 9/11.

In addition to the specific targeting Hayden discussed, evidence of a wider program of data collection began to surface after AT&T technician Mark Kline retired and took a stack of AT&T internal papers with him.

AT&T had developed sophisticated splitter technology for tapping into fibre optic cables. 

In 2002, Kline received an email saying that somebody from the NSA was coming to visit their San Francisco office, but it gave no further details. Kline found out why the following year when he was transferred to work one floor above a secret NSA room containing various Narus computers, including a logic server and semantic traffic analyser. He said: 

“As soon as I saw the splitter, I knew this was completely unconstitutional and illegal because they were copying everything … It was everything that went across the Internet then, which was Web browsing and email and VoIP calls. … everything’s flowing across the Internet to this government-controlled room. The physical apparatus gives them everything. A lot of this was domestic.”

In 2006, the Electronic Frontier Foundation lodged a class-action lawsuit, Hepting v. AT&T, which alleged that AT&T had allowed the NSA to use their technology to monitor phone and internet communications of AT&T customers without warrants. 

Diagram of how alleged wiretapping worked. From EFF court filings.
By U.S. Government employees, Public Domain

The case was dismissed in 2008 after Bush signed the FISA Amendments Act into law, granting retroactive immunity to telecommunications companies for past violations.

Information Awareness Office

Other secret incubating programs came from the Defence Advanced Research Projects Agency (DARPA) in January 2002. It established a new department called the Information Awareness Office (IAO) to focus on new surveillance programs aimed at providing greater national security.

Seal of the Information Awareness Office. 

The motto “scientia est potentia” is latin for “knowledge is power.” 

Dr. John Poindexter gave a speech at the DARPATech 2002 conference outlining his team’s prototype system, which was given the Orwellian-like name, the Total Information Awareness (TIA) Program. 

Poindexter told the conference: “Total Information Awareness—a prototype system—is our answer. We must be able to detect, classify, identify, and track terrorists so that we may understand their plans and act to prevent them from being executed. To protect our rights, we must ensure that our systems track the terrorists, and those that mean us harm. 

IAO programs are focused on making Total Information Awareness—TIA—real. This is a high level, visionary, functional view of the world-wide system—somewhat over-simplified.

global surveillance
Work of a U.S. military or Department of Defense employee, made as part of official duties. As a work of the U.S. federal government, the image is in the public domain in the United States.

One of the significant new data sources that needs to be mined to discover and track terrorists is the transaction space. If terrorist organizations are going to plan and execute attacks against the United States, their people must engage in transactions and they will leave signatures in this information space. This is a list of transaction categories, and it is meant to be inclusive.”

The system was initially given $200 million in funding. Most of the research was performed by defence contractors Booz Allen Hamilton and Raytheon, with some done in universities such as Cornell and Columbia.

However, it was a speculative project because it was not known whether an individual’s online behavior gave much indication of whether (s)he had anything to do with terrorism, and there remained concern over how this system could be abused. One of the creepier ideas was to create hidden cameras to detect if citizens were acting suspiciously. 

On Nov. 14, 2002, The New York Times published William Safire’s scathing article “You Are a Suspect,” slamming Poindexter as “ring-knocking master of deceit” with a “plan even more scandalous than Iran-contra.”

In February 2003, Congress passed legislation suspending activities of the IAO pending a Congressional report of the office’s activities, and DARPA provided Congress with a report on the program, renamed to the more palatable sounding Terrorism Information Awareness Program. A footnote explains, “Previously known as Total Information Awareness, this name created in some minds the impression that TIA was a system to be used for developing dossiers on U.S. citizens. That is not DoD’s intent in pursuing this program.”

These assurances did not stop the criticism. Poindexter resigned on Aug. 29, 2003, and further funding for the IAO was prohibited as part of the Department of Defense Appropriations Act, 2004.

In 2005, Bamford wrote “NSA, the Agency That Could Be Big Brother,” cautioning that spying had gotten more intrusive since the 1970’s Church report was released.

“At the time, the agency had the ability to listen to only what people said over the telephone or wrote in an occasional telegram; they had no access to private letters. But today, with people expressing their innermost thoughts in e-mail messages, exposing their medical and financial records to the Internet, and chatting constantly on cellphones, the agency virtually has the ability to get inside a person’s mind.”

Political Arguments and Legislation

In the previous episode, we began with the controversy of the Pretty Good Privacy (PGP) encryption tool. The first version of PGP was released by Phil Zimmermann in 1991 and contained some significant security flaws, so much so that one of the NSA’s top mathematicians, Brian Snow, hinted to the author that they were not concerned by it (i.e., the NSA could easily decrypt any PGP 1.0 messages).

However, the software release led to a community of developers interested in dramatically improving the product’s security, and by 1993, Zimmermann became the target of a criminal investigation.

Part of the reason for PGP’s initial popularity was the Senate Bill titled the Comprehensive Counter-Terrorism Act of 1991. This was sponsored by Senator Joe Biden and included the clause:

Subtitle B: Electronic Communications – Expresses the sense of the Congress that providers of electronic communications services and manufacturers of electronic communications service equipment should ensure that communications systems permit the Government to obtain the plain text contents of voice, data, and other communications when appropriately authorized by law.”

The online community of cryptography advocates interpreted this as a requirement for all electronic traffic to essentially be insecure. A bulletin board post urged “I suggest you begin to stock up on crypto gear while you can still get it.” This warning prompted the PGP author to give away his software for free and the worldwide audience to spread the software rapidly upon release.

The Senate Bill received a backlash from civil liberties groups, and the legislation was never enacted. By 2006, Biden’s position on NSA surveillance appeared to change. He appeared on The Early Show and argued:

“I don’t have to listen to your phone calls to know what you’re doing. If I know every single phone call that you made, I am able to determine every single person you talked to. I can get a pattern about your life that is very, very intrusive.”

Joe Biden was a strong critic of NSA bulk metadata collection before he became vice president

In a future episode we will cover Biden’s handling of the Edward Snowden revelations. But next up, we will cover another highly controversial topic: Join us in Part 21 for the story of WikiLeaks.