A search engine is not a search engine is not a search engine. In other words, there are search engines and then there are directories. So as the World Wide Haystack grows from millions to billions of Web sites, different search portals find wildly different needles. I could split more metaphors, but to cut to the bottom line, there are search engines and then there's Google.
Search portals are Web sites with search engines on them. They are called portals because they try to be like entryways to the Web. In addition to search engines, most portals are clogged with advertisements, news headlines, stock quotes, shopping links, free e-mail and type in a ZIP code and get the local weather. But most people go to portals for their search engines -- type in a few key words and they are supposed to find the Web address of the site you seek. Sometimes they do and sometimes they don't. And the more the Web grows, the more they don't.
Part of the problem is that Web sites are being created faster than the search engines can catalog them, and they change addresses as often as scam artists, which in many instances is no coincidence. In a constant cat-and-mouse game to fool the search engines, webmasters employ various tricks so that when you type in your search words, their site pops up instead of the one you want. The search portals themselves are not always honest brokers, as they attempt to funnel traffic to advertisers or affiliated Web sites.
Scores of search engines exist to steer you through the morass, and each is slightly different. I'd explain how they work, only you wouldn't understand and neither do I. What I can tell you is that of all the search engines, Google is the best. I say this from personal experience, and it is also the favorite of other Web reviewers. But just to make sure, I put the most popular search portals through a rigorous test -- I tried to find my own obscure Web site, "Lost in Cyberspace" <hbkoplowitz.com/LIC>.
According to Top9.com, an index that rates Web sites by traffic, the most popular search portal is the venerable Yahoo! <www.yahoo.com>. Created in 1994 by Stanford University grad students David Filo and Jerry Yang, Yahoo became the first search portal and is still the most visited site on the Internet. Yahoo itself isn't a search engine but a directory compiled by human beings who sift through Web sites and categorize them by subject. Way back in the '90s, when Web sites could be measured in thousands, directories could be very helpful. But now that there are more than a billion Web pages, directories are often used by search portals to lure users to "featured" or "reviewed" Web sites that have paid for the placement.
Yahoo's directory didn't find my Web site. But Yahoo also has a search engine on its portal, which did find my site. Unlike directories, search engines use automated "spiders" instead of humans to index Web sites by electronically "crawling" the Web, following hyperlinks from page to page. And since June 1999, the search engine powering Yahoo's portal happens to be Google.
When the Google search engine on the Yahoo portal looked for the key words "Lost in Cyberspace," my Web site popped up a respectable fifth, behind an article of the same name in the journal "New Scientist"; a help page for the governmental Web site for Clark County, Nev.; an article called "Women: Lost in Cyberspace?" by Laurie Finke, professor of Women's and Gender Studies at Kenyon College in Gambier, Ohio; and a Web site of the same name that seemed to be about cancer, animal cruelty and missing children.
The second most popular search portal and third most visited Web site overall is "Microsoft Network" <www.msn.com>, which happens to be the default home page for Microsoft's Internet Explorer browser. Like Yahoo, the MSN portal has both a search engine and a directory, so subtly blended it's easy not to notice the difference. When I typed in "Lost in Cyberspace," the MSN directory came up with fan sites for the TV show and movie "Lost in Space," along with the "Cyberspace Association of U.S. Submariners" <www.subnet.com>, a foundation of ex-submariners that aids children in memory of lost shipmates. The MSN directory did not find my site, and neither did the MSN search engine.
MSN's search engine is powered by Inktomi <inktomi.com>, which used to power Yahoo's search engine until Yahoo switched to Google. Inktomi was created in 1996 by Berkeley computer science teacher Eric Brewer and grad student Paul Gauthier. Their first search engine was called HotBot <hotbot.com>, which gained popularity when it was licensed by "Wired" magazine's Web site, HotWired.com.
Inktomi also powers America Online's search portal AOLNet, which switched from the Excite search engine to Inktomi in August 1999. As mentioned, Inktomi is also the search engine for MSN, although Microsoft changed to the search engine AltaVista in September 1999, then back to Inktomi in December 1999, although if you use a Mac and hit the search button on Internet Explorer browser you could be using Excite.
Bottom line: None of the popular search portals except those using Google found my Web site. Number three Lycos.com, had Clark County government at the top; number four Excite.com had "Journey of Hearts: A Healing Place in CyberSpace" <www.journeyofhearts.org>, a grief counseling Web site first; and number five AltaVista.com also had Clark County government first.
The sixth most popular search portal is the default home page for Netscape Web browsers <home.netscape.com>, a customizable portal that is trying to compete with Yahoo by using 20,000 volunteer reviewers to create a directory. My site didn't make the reviewers' list, with top billing going to "Mupps" <www.mupps.com>, "cartoon adventures of a dyke lost in cyberspace." Other sites making the reviewed list included "Gethsemane Prayer Group" <www.prayergroup.net>, a "spiritual oasis in cyberspace," and a story about a firm called Certifiedemail.com that on Aug. 4, 1998, "launched a service aimed at keeping important email messages from getting lost in the cyberspace abyss." Apparently the company got lost in cyberspace, because Certifiedemail.com is a dead link.
Even though AOL now owns Netscape, the default search engine for the Netscape browser is Google, which again listed my site fifth. Created two years ago by another couple of Stanford grad students, Larry Page, 27, and Sergey Brin, 26, Google's ability to find "relevant" Web sites is downright uncanny. Supposedly Google uses some sort of patent-pending algoritm, but what really distinguishes it from other search engines is its simplicity.
Rather than buying some super-sized mainframe computer to power Google, they wired together hundreds of off-the-shelf PCs. And rather than coming up with some ingenuous way to evaluate the content of Web pages, Google uses the same hyperlinks all search engines follow to find Web pages, only it counts them as votes. The company compares that to democracy, as did an article in "Time" magazine <www.time.com>: "The most linked-to sites win the Google usefulness ballot and rise to the top of search results." Like democracy, Google may not be perfect, but it seems to work better than the alternatives.
Though rated by Top9 as only the 13th most popular search portal and 55th most-visited Web site overall, the Google Web site <google.com> is the best portal by being the anti-portal -- there's nothing there except the search engine. No news headlines, stock quotes, free e-mail accounts, not even advertising. Best of all, no directory. Well, recently Google did add a directory -- Netscape's Open Directory Project -- but unlike other portals, the search engine comes first and you are given the option to use the directory.
So cocky is Google of finding your site, it has
an "I'm Feeling Lucky" button that takes you directly to the Web site
its first search result. I was tempted to make Google my default home
but Google makes it easy to put its search engine on your own Web site.
Now I have the Google search engine on my "Lost in Cyberspace" Web
so I can have my cake and cut bait, too.