Welcome to Part Seven of our journey through time, learning how the Internet has evolved and studying the effect it has had on our lives. During the last two posts, we looked at the bitter rivalry between Microsoft and Netscape for domination of the browser market. During the same period though, another battle was being fought in the search engine market.
Let’s begin with a look at some of the first search engines:
Archie and Veronica
The first Internet search engine was Archie, created in 1990 at McGill University in Montreal. The name Archie was a derivative of the word archive, as its job was to index File Transfer Protocol (FTP) based archives.
Two years later, a spiritual successor to Archie was created at the University of Nevada by Steven Foster and Fred Barrie. ‘She’ was named Veronica in a homage to both the original search engine and Archie Comics. Instead of using FTP, Veronica was based on the popular TCP/IP based protocol of its day, Gopher.
Gopher rivaled Tim Berners-Lee’s HTTP and the Wide Area Information Server (WAIS) protocol in the early 90s until HTTP emerged as the most popular choice.
The main advantage of Veronica over Archie was that it returned links to the actual documents queried. Archie merely returned the machine that the file belonged to.
WebCrawler
The earliest search engine (still in popular use today) is WebCrawler, which was created by Brian Pinkerton. Pinkerton was a molecular biotechnology and computer science researcher at the University of Washington, and also an employee of Steve Jobs’s company NeXT.
The main innovation of the early WebCrawler engine was the idea of using the number of hyperlinks to a website to determine the rank that site should show up as in the search results.
Pinkerton found the website that was most linked to in 1994 was the very first and original website of CERN. No doubt this fact was due to CERN’s highly successful World Wide Web project.
WebCrawler’s popularity was also due to its ease of use. With a simple HTML interface, it was the first engine that was well suited for the mass market.
WebCrawler went online in April, and seven months later somebody’s query for “Nuclear Weapons Design and Research” was the one millionth recorded request.
Pinkerton sold WebCrawler to AOL for $1 million in June 1995.
Lycos
In 1994, Dr. Michael Mauldin of Carnegie Mellon University was given a grant from the Defense Advanced Research Projects Agency (DARPA).
Fascinated with the metaphor of the web, he named his research project after the Latin name for the wolf spider family, lycosidae, because he intended to replicate their ability to seek out prey, as opposed to waiting for it to be caught on a web.
Mauldin’s inspiration grew from WebCrawler, but included groundbreaking algorithms such as analysing the hyperlink anchor text to gain a better understand the page’s meaning.
Lycos pioneered the idea of using hyperlinks as the basis for determining the host web page relevance to user’s search queries.
AltaVista
The AltaVista search engine was created at Digital Equipment Corp (DEC) by Paul Flaherty, Louis Monier, and Michael Burrows.
Some of the details remain as legend, the most famous of which is that the search engine was created to showcase the enormous power of DEC’s Alpha processor. Louis Monier has since disputed this story, calling it an “after-the-fact rationalization.”
DEC’s Network Systems Lab manager, Brian Reid, said the facts remained in dispute and cited backstabbing over the credit for the original idea.
One thing that is hard to dispute, however, is that AltaVista was a great technical achievement: whereas other early search engines were only able to identify a small proportion of up to date content on the web, AltaVista used a thousand web spiders working in parallel to return results that were very close to a complete index of the world wide web.
However, AltaVista failed to achieve its full business potential due to many flawed business decisions. We will explain a major missed partnership opportunity in Part 8.
Inktomi / HotBot
Eric Brewer and Paul Gauthier developed the Inktomi (pronounced “ink-to-me”) web search engine at the University of California, Berkeley in 1995.
The name Inktomi is derived from a mythological spider of the Plains Indians known for bringing culture to the people
Inktomi Corporation was founded and incorporated in February 1996. The company used parallel-processing technology to deliver high-performance searches.
Dr. Eric Brewer, assistant professor of computer science at the university, boasted: “Our engines are the only ones that can scale with the exponential growth of the Internet, so Inktomi will always be able to offer the whole Web — not just a subset of information we personally deem appropriate— to millions of users. By bringing supercomputing to the Internet, we're able to offer users the most relevant, timely and complete information, without sacrificing speed and without costly hardware.”
Inktomi partnered with HotWired Ventures LLC, and their database technology was used in the HotBot search engine. Wired magazine launched the Hotbot search engine by promoting it to their readers in May 1996. Lycos acquired HotBot as part of its acquisition of Wired in 1998.
Yahoo!, King of Search Portals
In the mid 90s, websites such as Yahoo!, AltaVista, Excite, and WebCrawler were typically used by consumers as the starting point for finding many useful and interesting websites. These sites were called Web portals, and the main focus of these technologies was making it possible to find anything easily.
Just like Google, Yahoo! was started at Stanford University. Yahoo! began as an idea, grew into a hobby, and then turned into a full-time passion. This is the story of David and Jerry.
Jerry Yang was named Yang Chih-Yuan at birth, and took the American name Jerry after moving from Taiwan to the U.S. in 1978. His mother had dreaded the thought of him being drafted by the Taiwanese army, and the move gave him many new opportunities. At Piedmont Hills High School he was the most talented student at math, he always won at math quizzes, and within three years he had gone from remedial English to advanced-placement English.
David Filo met Jerry Yang at Stanford in 1989, and they became good friends. They both shared interests in sports, math, and engineering. By late 1993, they began work on a website that could organize Internet sites into categories.
Working in a campus trailer, they created “Jerry and David's Guide to the World Wide Web” as a way to keep track of their personal interests on the Internet. It was one of the first Internet directory services to help surfers find their way around the Web.
A few months later they renamed it Yahoo!.
Filo and Yang have said Yahoo! stands for “Yet Another Hierarchical Officious Oracle,” but they selected the name because they loved the “rude, unsophisticated and uncouth” definition of a yahoo.
Their initial approach to guiding the website’s users to interesting web content was different: they employed a team of editors who selected Web sites for an alphabetized directory.
Yahoo! quickly became one of the most popular sites on the web, fostering a closely bound Internet community of thousands of users. Filo and Yang had no money to spend on advertising the website, but it did not matter. Word of mouth gave them free advertising, and the site celebrated its first million-hit day in the fall of 1994.
Stanford University officials politely requested that they to find a company that was willing and suited to hosting their computing-intensive service. Filo and Yang soon agreed to a smart deal with Marc Andreessen to host their software on the high performance computers housed at Netscape.
In March 1995, Filo and Yang incorporated the business and met with dozens of Silicon Valley venture capitalists. As Filo and Yang had not yet thought out a viable business model, most investors had no interest on taking on such a risk. However Michael Moritz of Sequoia Capital thought that the potential gains were good enough to take a gamble, and agreed to fund Yahoo! in April 1995 with an initial investment of nearly $2 million.
They were also joined by another bright Stanford student, Tim Brady, who helped them with their business plans, and they soon had banner ads funding their site.
Yahoo’s What’s Cool page had the ability to make, and possibly even break, Internet sites. On July 16, 1995, Amazon.com launched and the company quickly caught Yang’s eye, who promptly emailed Jeff Bezos saying they’d like to link to them.
Bezos thought “it might be like taking a sip from a fire hose.” Would his new site be able to handle the massive increase in traffic? Ultimately he decided it was an opportunity too good to turn down, and this was the right decision: within a few days orders shot up by more than 1,000 percent!
Koogle
Yet another Stanford graduate, Tim Koogle, already ran a successful company but was attracted to this new and exciting search engine venture. He became the first Chief Executive Officer of Yahoo! in July 1995.
One of his first decisions was to take the software that was being hosted by Netscape hardware and move it in house. The reason he did so was that, unlike Filo and Yang, he recognised Netscape as a potential competitor.
The alphabetized listings could never hope to be comprehensive, and were unable to keep up with the accelerating pace of change on the Web. Filo and Yang realized that they needed to add a search facility to the site, and asked Koogle for his opinion. Koogle felt that developing its own technology was too expensive and too much of a distraction from its other business objectives. So Yahoo! worked with partners to provide its search facility.
Initially the Yahoo! search was powered by OpenText technology, but the company soon switched to partner with AltaVista. As part of the agreement, AltaVista agreed to not to provide its technology to any other competitors in the search market.
In 1996 Yahoo! launched a highly-successful initial public offering (IPO) raising $35 million, and the share price tripled on the first day of trading. Market capitalization hit $1 billion.
Many new divisions of Yahoo! sprang up. Yahoo finance offered financial news, stock quotes, and press releases. Yahooligans launched in March 1996 to let children find safe and appropriate Internet content.
Although Yahoo! was clearly about much more than just search, by 1998 Yahoo! had been voted the best search engine for the third year running.
The Stanford Effect
Just like Yahoo!, two smart and passionate Stanford University students created Google.
And those are far from the only major Silicon Valley companies that were incubated at this university. A little known fact is Sun (Microsystems) stands for Stanford University Network. Hewlett Packard and Logitech are also great success stories with connections to Stanford.
Stanford president John Hennessy says, “We have an environment that promotes entrepreneurship and risk-taking research. You have this environment that get people thinking about ways to solve problems that are the cutting edge.” Another factor is the area is rife with venture capitalists and it has been easier for both Stanford students and professors to get funding than it is for many students elsewhere.
In January 1996, the computer science department moved into a new building named after William Gates. While Gates never went to Stanford himself, naming rights were agreed when he donated $6 million towards construction costs “to invest in the future of the industry.”
At the dedication ceremony, the dean of the engineering school made a bold prediction:
“Something will happen here, and there will be some place, some office, some corner, where people will point and say, ‘Yeah, that’s where they worked on the blank in 1996 and 1997. And you know, it was a big deal.’”
James Gibbons was absolutely correct. Something did happen, and it was and still is a big deal.
Join us in Part 8 for the story of how Google came to be.
