By July 8, 2019

History of the Internet: Part 17 – Surveillance, Cryptography, and Free Speech

The beginnings of government regulation of the internet and the battle to prevent government overreach began before the World Wide Web. As explained in Part One, the internet began as ARPANET, a project created by the United States Department of Defense’s Advanced Research Projects Agency, which began in 1969.

Other important historical events at the time included the first man on the moon, the Vietnam War, and a growing counter-culture against this war and against the establishment in general.

Amongst the disaffected were hippies, second-wave feminists, and civil rights groups that were broadly referred to as the New Left. Not only were they against the Vietnam War, many were also against the Cold War with the Soviet Union and were particularly concerned that the nuclear arms race could lead to a nuclear holocaust.

It was in this context that new ideas, new forms of expression, and the concept of future technologies began to appear.

Google Before Google

Steve Jobs was one of these hippies, and in 2005 he described his youth as an amazing time:

“When I was young, there was an amazing publication called The Whole Earth Catalog, which was one of the bibles of my generation … It was sort of like Google in paperback form, 35 years before Google came along. It was idealistic and overflowing with neat tools and great notions.”

Whole Earth Catalog author Stewart Brand believed that the first NASA image of the whole planet as seen from space was a powerful symbol, evoking a sense of shared destiny for the human race. 

Unlike regular catalogs, it did not sell the products that it featured and was more of a specific lifestyle promotion than a product brochure. Part of that lifestyle included the adoption of new technologies, and it was one of the first publications to support computers, showing a photo of a Computer Club featuring “two Commodore calculators.”

history of surveillance and cryptography
Whole Earth Catalog 1st Edition. Photo courtesy of Stewart Brand

According to Stanford University professor Dr. Fred Turner, the Whole Earth Catalog “established a relationship between information technology, economic activity, and alternative forms of community that would outlast the counterculture itself and become a key feature of the digital world.”

The end of the decade was also important for another reason. It saw a landmark U.S. Supreme Court ruling that would influence free speech policy both online and offline for decades to come.

Free Speech and Trust in Government

In 1969, Ku Klux Klan leader Clarence Brandenburg was arrested and convicted of advocating violence following a white supremacist speech. 

In Brandenburg v. Ohio, the Court ruled that the government cannot punish inflammatory speech unless that speech is “directed to inciting or producing imminent lawless action and is likely to incite or produce such action.” Brandenburg’s criminal convictions were overturned on this basis.

According to the Pew Research Center, U.S. citizens’ trust in their government fell sharply throughout the 60s and 70s before beginning to rise again in the 80s. This is despite the Freedom of Information Act passing in 1966 and being strengthened as part of the Privacy Act of 1974

In a 1969 Time magazine article about creeping U.S. government surveillance, Nixon’s Attorney General John Mitchell attempted to assure readers by stating “any citizen of the United States who is not involved in some illegal activity has nothing to fear whatsoever.”

The truth was more complicated than that, as the next decade proved. Understanding the general climate and government activities in the 1970s is important in understanding subsequent actions, affecting free speech on the internet.

Conspiracies in the 1970s

The early ’70s saw several whistleblowers contacting the press with concerns about abuses of power within the intelligence services. Former intelligence officer Christopher H. Pyle reported in January 1970 that the U.S. Army was spying on civilians.

In 1971, anti-war activists seeking proof for their suspicion that they were under regular surveillance and infiltration stole classified information from an FBI branch office and discovered the COINTELPRO program. 

This program targeted various political groups, including the National Association for the Advancement of Colored People (NAACP), and attempted to induce members into committing criminal acts so that the FBI could arrest them. This led to a series of articles in The Washington Post.

In January 1971, the front cover of The New York Times revealed that the U.S. had secretly enlarged the scope of its actions in the Vietnam War with the bombings of nearby Cambodia and Laos. 

This information came from a then top-secret Pentagon study known as the Pentagon Papers and was disclosed to the media by Daniel Ellsberg. According to The New York Times, Ellsberg’s bravery revealed the Johnson Administration “systematically lied, not only to the public but also to Congress.”

The government sought a restraining order preventing the Times from publishing any further information, claiming it would cause “irreparable injury to the defense interests of the United States.”

The Pentagon Papers

Arguments on the right or lack of right to publish this information were heard in the Supreme Court case New York Times Co. v. United States, where it ruled 6-3 in favor of the press, ruling: 

 “paramount among the responsibilities of a free press is the duty to prevent any part of the government from deceiving the people and sending them off to distant lands to die of foreign fevers and foreign shot and shell.”

The Pentagon Papers were declassified and publicly released in 2011, and Steven Spielberg turned the story into a Hollywood film starring Tom Hanks and Meryl Streep in 2017. 

Such a major ruling in favor of freedom of information is something that could not happen in many countries around the world. The United States has a proud tradition in the right to free speech, as enshrined in the United States Bill of Rights

This was largely in response to the British government, which had a culture of secrecy and intrusion that included the covert opening of mail, and the routine searches of property in America in the days before it won its independence. 

Daniel Ellsberg. Photo By Kushal Das, CC BY-SA 4.0

However, on Jan. 3, 1973, Ellsberg was charged under the Espionage Act along with other charges of theft and conspiracy, carrying a total maximum sentence of 115 years. 

Around this time, the Watergate investigation revealed evidence of corruption existing at the highest levels of public office, including that the White House had ordered so-called White House Plumbers to engage in unlawful efforts to discredit Ellsberg, such as the bungled burglary of the office of Ellsberg’s psychiatrist. 

Due to governmental misconduct and illegal evidence-gathering, Judge William Matthew Byrne Jr. dismissed all charges against Ellsberg on May 11, 1973.

In August 1974, President Nixon resigned, citing a lack of political support for him to be able to continue in office. Four months later, The New York Times published an article by Seymour Hersh detailing CIA operations involving assassination attempts on foreign leaders and covert attempts to subvert foreign governments.


history of surveillance and cryptography
Senator Frank Church

These revelations led to two investigations. The Rockefeller Commission was created by President Ford in 1974 to look into allegations of CIA involvement in monitoring domestic political dissent.

The Church Committee investigation looked at the activities of the CIA, NSA, FBI, and the IRS. The Rockefeller Commission issued its report on June 6, 1975, five months after the Church Committee had been formed.

This investigation uncovered details on Project SHAMROCK, an NSA operation involving the accumulation of all telegraphic data entering into or exiting from the United States. 

Reading the Rockefeller report, Investigator L. Britt Snider found a couple of key references to the NSA. “The first was a reference to an office in New York that CIA had provided NSA for the purpose of copying telegrams. The other disclosed that CIA had asked NSA to monitor the communications of certain US citizens active in the anti-war movement,” he recounts on the CIA website

All of Snider’s efforts to get the NSA to talk about this failed until an August 1975 New York Times article alleging that NSA had eavesdropped on the international communications of U.S. citizens. With these allegations now in the open, the NSA wanted to explain its side of the story. 

After interviewing an unnamed NSA employee, Snider questioned former NSA deputy director Dr. Louis Tordella about SHAMROCK. Tordella replied by asking Snider what he already knew. 

When Tordella realized that Snider already knew quite a bit, he knew that he would need to cooperate: “He sighed a long sigh and then began a discourse on SHAMROCK that lasted into the early evening. The more he talked, the more he seemed to relax.” 

history of surveillance and cryptography
Louis W. Tordella, the longest serving deputy director of the NSA 

As part of SHAMROCK, a courier took reels of magnetic tape from New York telegraph companies to the NSA, where they searched for anything related to any intelligence target or anything that was encrypted. 

Tordella said that the material was mainly used for foreign intelligence purposes, but it later emerged that an NSA watchlist contained hundreds of domestic targets that SHAMROCK was matched against. The information was fed into a high-end IBM 7950 computer, better known by its nickname Harvest.

Harvest was a one-of-a-kind adjunct to the Stretch computer, which was installed at the NSA for the purposes of cryptanalysis.

Although primitive by today’s standards, at the time of purchase, the NSA found the Harvest to be between 50 and 200 times faster (depending on the task) than the best commercially available machine. It was able to search for keywords and scan 7 million messages for any of 7,000 keywords in under four hours. 

Snider asked Tordella whether it was legal to be reading the telegrams of American citizens. “You’ll have to ask the lawyers,” Tordella replied shrewdly, adding that it was all done for patriotic reasons.

history of surveillance and cryptography
A Harvest magnetic tape. Photo by an employee of the United States government, Public Domain

In August 1975, Senator Frank Church told NBC’s Meet the Press:

“If this government ever became a tyranny, if a dictator ever took charge in this country, the technological capacity that the intelligence community has given the government could enable it to impose total tyranny, and there would be no way to fight back, because the most careful effort to combine together in resistance to the government, no matter how privately it was done, is within the reach of the government to know. Such is the capability of this technology.”

The committee was also appalled by COINTELPRO, describing it as a “sophisticated vigilante operation aimed squarely at preventing the exercise of First Amendment rights of speech and association, on the theory that preventing the growth of dangerous groups and the propagation of dangerous ideas would protect the national security and deter violence.”

It also found that during Operation CHAOS (1967-1973), “some 300,000 individuals were indexed in a CIA computer system” and that “an estimated 100,000 Americans were the subjects of United States Army intelligence files created between the mid-1960’s and 1971.”

After hearing reports that the NSA was claiming their eavesdropping on American citizens had stopped a year-and-a-half previously, and knowing this to be untrue, James Bamford contacted the Church Committee, which was shocked to discover that the NSA had been lying to them.

This was also the start of a new investigative career for Bamford, who would become a thorn in the side of the NSA for decades to come with his numerous exposes on the otherwise ultra secretive organization known as “No Such Agency.” 

Data Encryption Standards Controversy

[Note: This section assumes the reader has some knowledge of information security concepts. Jason Boyer has written the articles “5 Security Concepts Every Developer Should Understand” and “Protect Your Passwords,” which I highly recommend.]

One of the exciting prospects in the 1970s was the news that computers were now able to communicate with each other. This gave rise to the dream of a new world where everyone would be able to communicate freely with one another without government interference. However, to ensure citizens privacy, communications need to be encrypted.

For professionals working in national security however, concerns have remained mostly the same throughout the history of the internet: the idea of the whole world being able to communicate with one another without the ability to intercept is something that poses grave dangers. Their job is to protect their respective countries, but if encryption is so strong that it is unbreakable, they are almost powerless to detect genuine threats.

The National Bureau of Standards (NBS, now known as the National Institute of Standards and Technology) was responsible for finding an encryption algorithm that would become a standard.

After consulting with the NSA, the NBS selected a modified version with a shorter 56-bit key size. The justification for a shorter key size was to keep costs down.

The proposal for the Data Encryption Standard (DES, commonly pronounced dez) requested public comments. Among the critics were doctoral students Whitfield Diffie and Martin Hellman.

Diffie and Hellman produced the 1976 paper New Directions in Cryptography. Beginning with the triumphant announcement “We stand today on the brink of a revolution in cryptography,” this paper explains approaches for private communications over a computer network and introduces a radical method of distributing cryptographic keys, now known as the Diffie-Hellman key exchange.

The basic concept of “non-secret encryption” (now called public key cryptography), had actually been invented six years earlier by British cryptographer James H. Ellis at the UK Government Communications Headquarters (GCHQ). The classified nature of his work meant Ellis received no public credit for his idea until after his death.

history of surveillance and cryptography
Whitfield Diffie at Computers, Freedom, and Privacy conference. Photo by Simon Law from Montréal, Canada

The following year Diffie and Hellman published a paper titled Exhaustive Cryptanalysis of the NBS Data Encryption Standard, which argued that the DES key size was too short, that it needed to be increased to protect against brute-force attacks (using a powerful computer to try every possible key), and that a larger key size would not have a significant bearing on costs.

Martin Hellman. By User .:Ajvol:. on en.wikipedia CC BY-SA 3.0 

The arguments in this paper led to the conspiracy theory (unproven to date) that the NSA had interfered with the design of the algorithm so that they, and only they, could read DES encrypted messages.

Cryptology and the NSA

Diffie and Hellman shared a strong anti-authoritarian outlook. They suspected that the NSA owned the world’s most sophisticated cryptanalysis machines and were skeptical about NSA intentions. Hellman later recalled, “My view at the time was I was Luke Skywalker in Star Wars and NSA was Darth Vader.”

Employees from the NBS and NSA, Arthur Levenson and Douglas Hogan, came to Stanford to discuss their objections and agreed to being tape recorded by Paul Baran (presumably to avoid reports of the conversation getting distorted).

Hellman admitted that he wasn’t a security expert at the level of those in the NSA, but opined, “we feel it’s vulnerable to attack by an agency such as NSA [sic] … And we feel that, looking 15 years down the line, and extrapolating the decreasing cost of computation, the standard would be insecure against attack by almost anyone.”

Hogan responded on behalf of the NSA, explaining there is no such thing as absolute future-proof security in the field of cryptology:

“… part of our job is to analyze threats and then say ‘this appears to be an appropriate level to counter against it.’ Because there is no such thing as 0/1 security, and in analyzing the threats, we had come up with a certain level that we thought things needed to be.”

Levenson said that there was a balance between being able to crack codes and securing information, adding, “We feel we’ve struck that balance.” Hellman replied, “I don’t think that the National Security Agency is the appropriate organ of the government to decide where that balance lies.”

Hellman’s attitudes evolved in the subsequent years. In a 2004 interview with Jeffrey Yost from the Charles Babbage Institute, he reflected: “My view back then was they were not interested in national security, they were interested in job security. Now I look at it very differently. I do think they were concerned. They had some legitimate concerns and I should have taken those into account.”

A 1978 review by the United States Senate Select Committee on Intelligence found the “NSA did not tamper with the design of the algorithm in any way.” 

The Director of the NSA at the time, Admiral Bobby Ray Inman, discussed this in an interview with Gordon Corera in 2014: “The NSA was actually engaging in a highly complex balancing act in working out how secure to make non-classified communications … You wanted it strong enough that it could not be broken easily … The great worry was that this effort would produce cryptographic systems that they couldn’t break and it wasn’t just worry about drug dealers and the rest of that. It was that they could be picked up by foreign countries.”

history of surveillance and cryptography
Admiral Bobby Ray Inman. Public Domain 

A series of books American Cryptology during the Cold War provides some further clues.

In Book III, author Thomas R. Johnson admitted the “NSA tried to convince IBM to reduce the length of the key from 64 to 48 bits. Ultimately they compromised on a 56-bit key.” And intriguingly, although declassified, key sections on DES remain redacted.

Also of historical interest is Book IV, written in 1999 and declassified from Top Secret, with some redactions, 20 years later. It covers the period of 1981 to 1989 and remains an important work in understanding both the Cold War and cryptographic advances. It includes chapters on the Reagan Revolution, the Second Cold War, and the Rise of Terrorism and Unconventional Targets in the 1980s.

history of surveillance and cryptography
Bruce Schneier. Photo by Rama, CC BY-SA 2.0 fr

Despite these fears, DES has protected the data of millions of computers for around 30 years. Security technologist Bruce Schneier disagrees with the conspiracy theory and has argued “It took the academic community two decades to figure out that the NSA ‘tweaks’ actually improved the security of DES. This means that back in the ’70s, the National Security Agency was two decades ahead of the state of the art.” 

These arguments concur with IBM engineer Walter Tuchman, who said he worked on DES with the NSA in good faith: “If they fooled me, I will go to my grave being fooled. I looked these guys eyeball to eyeball. I’m a bit of a film buff, and I’ve seen good acting and poor acting. And if the NSA people fooled me, they missed their profession. They should have gone to Hollywood and become actors.” No convincing evidence of a trapdoor has ever been found. However, Tuchman’s explanation that the key size was reduced to allow space for parity checks was neither affirmed by the NSA nor the Senate Intelligence Committee, whose unclassified report stated “The NSA convinced IBM that a reduced key size was sufficient.”

Trouble in Spook Valley

Relations between Hellman and the NSA took another turn in 1977 when Hellman and two of his graduate students attempted to present their work on Diffie–Hellman key exchange at The International Symposium on Information Theory.

A letter was received from someone in the NSA warning about his research and how it could fall foul of the International Traffic in Arms Regulations and be treated as exporting weapons to a hostile foreign power. Hellman explained to Corera that it was a stressful time: 

“Silicon Valley is sometimes called Spook Valley because not only is silicon made here but there is a huge presence here by the intelligence community. And some of my friends in Spook Valley told me I was stepping on some very dangerous toes. And by the way I wasn’t just stepping on NSA’s toes, I was stepping on GCHQ’s toes, I was stepping on the KGB, GRU’s toes. And so they warned me that not only was I potentially going to suffer legal action but my life might be in danger.”

I Am Your Father

Despite fear of prosecution, or worse, Hellman proceeded to give the talk. Rather than raise tensions with Hellman further, Admiral Inman decided to defuse them by talking with him directly. Hellman later explained:

“I get this call and it says Admiral Inman is coming to the Bay area and would like to meet with you if that’s possible. So here I’ve got Darth Vader in mind – at least in my mind at the time – wants to meet me. But of course I didn’t know – being Luke Skywalker – that he was in some sense my father. But I said I’d be happy to and we met in my office here at Stanford…

And I’ll never forget. His first words jokingly were, ‘It’s nice to see you don’t have horns.’ Because that’s what he was getting from everybody at NSA and from their perspective … And so I said, ‘Same here,’ because they’d seemed like the devil to me.”

In this meeting, Inman and Hellman took the first unlikely step of going from enemies to friends. They agreed to a voluntary system where students could submit planned papers to the NSA, which would in turn explain any sensitive issues, but not try to veto them. Inman’s thinking was “we made the presumption that, notwithstanding their political views might be different, they still were patriotic.”

Further information on Diffie and Hellman’s battles with the government can be found in the BBC article “Geeks v government” and in the Stanford magazine article “Keeping Secrets.” 

About the author

Kevin O'Shaughnessy

A senior Web developer based in the UK. Kevin runs a blog at and is also a regular guest blogger at Outlier Developer. He believes in continuous learning and improvement. Kevin is also active on the Fediverse at