The brilliant idea of the World Wide Web was devised in the spring of 1989 at the head of Tim Berners-Lee, a physicist at CERN, but it didn’t gain any widespread popular use until the remarkable NCSA Mosaic web browser was introduced at the beginning of 1993.
In the spring of 1993, just months after the release of Mosaic, Matthew Gray, who studied physics at Massachusetts Institute of Technology (MIT) and was one of the three members of the Student Information Processing Board (SIPB) who set up the site www.mit.edu, decided to write a program, called World Wide Web Wanderer, to systematically traverse the Web and collect sites. Wanderer was first functional in the spring of 1993 and became the first automated Web agent (spider or web crawler). The Wanderer certainly did not reach every site on the Web, but it was run with a consistent methodology, hopefully yielding consistent data for the growth of the Web.
Matthew was initially motivated primarily to discover new sites, as the Web was still a relatively small place (in early 1993 the total number of websites all over the world was about 100, and in June of 1995, even with the phenomenal growth of the Internet, the number of Web servers increased to a point where one in every 270 machines on the Internet is a Web server). As the Web started to grow rapidly after 1993, the focus quickly changed to charting the growth of the Web. The first report, compiled using the data collected by Wanderer (see the table below) covers the period from June 1993 to June 1995.
| Results Summary | |||
|---|---|---|---|
| Month/Year | Nr. of Web sites | % of .com sites | Hosts per Web server |
| 06/93 | 130 | 1.5 | 13000 |
| 12/93 | 623 | 4.6 | 3475 |
| 06/94 | 2738 | 13.5 | 1095 |
| 12/94 | 10022 | 18.3 | 451 |
| 06/95 | 23500 | 31.3 | 270 |
| 01/96 | 100000 | 50.0 | 94 |
Wanderer was written using the Perl language and while crawling the Web, it generated an index called Wandex—the first web database. Initially, the Wanderer counted only Web servers, but shortly after its introduction, it started to capture URLs as it went along.
Matthew Gray’s Wanderer created quite a controversy at the time, partially because early versions of the program ran rampant through the Web and caused a noticeable network performance degradation. This degradation occurred because it would access the same page hundreds of times a day. The Wanderer soon amended its ways, but the controversy over whether spiders were good or bad for the Internet remained for some time.
Wanderer certainly was not the Internet’s first search engine, it was the Archie of Alan Emtage, but Wanderer was the first web robot, and, with its index Wandex, clearly had the potential to become the first general-purpose Web search engine, years before Yahoo and Google. Mathew Gray however does not make this claim and he always stated that this was not its purpose. Anyway, Wanderer inspired a number of programmers to follow up on the idea of web robots.
From 2001 to 2006, Matthew Gray was CTO of Newbury Networks, Inc., a provider of wireless location technology. As of 2022, he has spent more than 15 years as a software engineer and engineering director at Google.