| Web Search Nightmare |
I had to say a few personal words on the entire SE (Search Engine) world, so to speak. Not so much as to give anyone the definative answer to any questions, but more to sort through the the 'strange goin-ons' in the search world. I'm also not going to discuss the mult-search tools like metasearch, and dogpile since they get their results directly from other sources. Due to the rapid changes in the SE world and searching the web, I've stared a "Search News" site that will cover the latest updates.
|
As a webmaster, it is very important to have a grasp of what the search engine world is, how it works, and where to place the importance of your site. For the most part, I'm not going to discuss the "webmaster" end of it in this article, but rather the search engine world in general. As a surfer, it is important to know what places to go to search for things, and who to go to in order to find the most relevant results for your search. If you find that many search engines provide similar results, that is because many of them partner with other sites, or use the information gathered by large search databases. For example, Direct Hit was very big in the late 90's, then all their data was purchased by Tehoma - then Ask Jeeves and Tehoma partnered so that ask.com will actually provide information that is in the Tehoma database. Even the biggest of search engines will make arrangements to use the information from a competitor as a back-up to their own data. That way, if you don't find the terms you are searching for directly within one site - you will see the results from another site. A prime example of this is Yahoo. They have now purchased the Inktomi company to provide results outside their own "directory" database - AND they continue to partner with Google. The result is: If the Yahoo database doesn't find relevant data or pages then they will look at the Inktomi database for websites. Next, they ask Google what they consider to be the top results for the terms you searched for. Many of these companies now work togeather in one form or another. Lycos and Hotbot are partners. Excite and iWon are partners. iWon uses Googles results. Many of them (such as Excite and Lycos) have expanded into more than just search engines. They may offer email, chat programs, portals, personalized home pages that provide your stock quotes, or favorite sports teams news. |
As of the middle of July, here is the way things are shaking out.
Google: Still number one
Yahoo: Now owns Overture, Altavista, Alltheweb (along with Fast a/k/a fastsearch.com), Lycos, Infospace, and Inktomi
MSN: Owns Looksmart
Ask: Owns Tehoma
Excite: Owns iWonIt looks like it will shake out into 3 major players: Google, Yahoo, and MSN. Ask just doesn't have the huge client base it needs to compete (although I like their services).
Search Engine Databases :
example: Google These are the work horses of the search engine world. Sites like Google compile large databases of the web sites on the Internet, catalog them, index them, and then rank them according to the words people search for. When a site is submitted to a database like this, the normal proceedure is for the SE to send out a 'bot' or 'spider' which are little computer programs that look at the site, read all the information on the pages, follow the links to other pages on the site, and review them as well - all the while reporting back to the database the content of the site. These are normally the easiest and quickest to get listed in because there is very little human intervention involved. Computer programs and alogrithyms are what indexes and ranks these page. If you submit to this type of search engine, you can often expect to see your site listed within 2 to 4 weeks.Search Directories:
example: Yahoo These sites are more the human element of the search engine world. They set up catagories or directories, (and within those directories, sub directories) and your site or pages are submitted to a particular 'group' or similar sites. These types of search tools are normally maintained and edited by people rather than computers. Since each site is reviewed by a person, it often takes a lot longer to get your site listed in this type of search service. There is also no promise that your site will be listed at all. This is a much more subjective resource than the computer database style of listing, but the upside is that if you do rank well here, then you have a much better chance of staying at the top of the list. The reason being that a computer program can't come along, figure out an alogrythim, and replace your ranking with a generated page that offers little content. The down side of this is that it is harder to get listed in these search tools, and if you are accepted, it can take as long as 3 or 4 months before your listing shows up at all. These sites are also refered to as portals in some circles.Multi-Search, or Meta Search:
example: Dogpile These are the "multi-tasking" search sites. They send a search request out to several different search sites (like Google, Yahoo, Looksmart, and Inktomi) all at the same time and pull the top 10 or 20 results from each individual site. Depending on the complexity of the site, it may remove duplicate findings, or it may group the results according to each engine. The meta search will then show you the list of the top sites from the various search tools. Normally to get 'listed' with these sites is not a direct project, rather you submit your site to the other search services that the meta search engine uses.
It is important to note that parts of this review are my own personal opinion. While I do state facts (as best as I know them) and I can say who is working with who - the opinion that one SE is good, and another is bad is just that MY OPINION.Google:
Far and away, Google is the top search engine today. It is the most popular among surfers, and for webmasters, it is the most important to be listed in. In my opinion, the reason for this is pure and simple logic. Quality. They offer the most relevant results when you do a search for something. Google has been the search leader now for a couple years, taking the "king of the mountain" spot from former search king Yahoo.Yahoo - Inktomi:
Yahoo is still at the moment, the second most used search tool. Yahoo is a bit different in that they offer "directories" or catorgies, that you can search through, all of which are reviewed and ranked by people rather than by computer programs. Yahoo has purchased Inktomi, which provided the data to MANY of the other search engines such as scrub the web, iWon, and NBCi. In early 2003, Yahoo also rolled out a newer, cleaner search interface to conside with the purchase. This accquisition will give Yahoo a HUGE boost in their data resources throughout the search world. As of June 2003, Inktomi announced a new search, index, and ranking solution. Time will tell how they use it.dmoz:
Another big player, although they are more of a "behind the scenes" player is dmoz. dmoz is a human edited directory listing which powers the big boys. The dmoz.com site may not be used as often as many others, but their information is. They supply information to Google, Netscape, AOL, and others. They don't rank the sites for these search tools, they only supply the raw data, and it's up to each to interpert the information as they see fit. And to be honest, AOL / Netscape do supply a VERY large number of results to folks, if for no other reason, than that when you start AOL, you are "searching" with their tools. This is very similar to the role that Inktomi had played in the past.Ask Jeeves - Tehoma:
Another very important search engine is ask.com. They started out as Ask Jeeves, and quickly became popular with the "Type in your question" style of search engine that made various sorts on this 'natural language' style of querry very popular. They purchased Tehoma a while back, and recently have developed a new look in efforts to keep up with the sweeping changes in the search industry. They still provide execellent results, although I don't like the way they want to keep their search results within their own frame. They do have a small link in the top right corner to remove frame however. Ask is very strong in finding things such as "when", "who invented", and "how does it work" types of search querrys. Tehoma is the engine that drives ask.com, and they provide Google results called "Sponsored Links" as well as their own results.All The Web - Fast:
alltheweb.com has become a very popular search engine, and they do return very good and relevant results. They probably enjoy more sucess in the European part of the world, but still get used greatly in the USA. They get most of their data from "fast" or fastsearch.com. Although fastsearch.com doesn't always return a good result for a search - alltheweb.com usually does because they sort and rank the pages better.Looksmart:
Looksmart at one time was making a good run at Google for the top spot in the SE world. When they couldn't quite make it, they partnered (sold out to) with Microsoft and MSN to be their primary provider. They also provide results to many other big SE pages like About.com, c|Net, Netscape, and Infospace. My personal view on the quality of their returns is that it has gone downhill over the last year or so.Netscape - AOL:
Netscape and AOL were at one time seperate companies. The two companies merged in the late 1990's due to the presence that Microsoft, MSN, and Internet Explorer were putting on the Internet world. Today (July 2003) Netscape also returns results from Ask.com, Looksmart, Overture, and Lycos. They also feature results from Google. AOL is part of Netscape, and provides similar results when you use their search engine, however they not only provide Google results, but they also boast the use of the dmoz Open Directory Project.Overture - Tehoma:
Overture is a huge database that drives a lot of the popular search engines today. AltaVista is one that was once king of the search engine world.MSN - Microsoft
MSN is the Microsoft search work horse, and they have set their gun-sights on Google. If past history has anything to bear, then this could be a major news story down the road. Normally this wouldn't bother me since I'm not really a Microsoft basher at heart. (although I do tend to root for the little guy). The one issue that does bother me about MSN is their results. Primarily they place the sites that "PAY" them money for a listing in their bCentral.com database, or the paid listings of looksmart, at the top. That means the results are not really relevant to the search you just did. You only get fed the results of those sites that paid the $$$, not ALL the sites on the web which cover the items you searched for. In fact, I've found the results of the MSN site to be very poor at many things. The one thing they are good at is a few alternates if you mis-spell a domain wrong in the search bar.After these big boys - it gets a little less clear. Excite, scrubtheweb.com, Lycos, Hotbot, iWon, and many others are popular. I know everyone has their favorite, and I'm not including, or omitting any because I have an ax to grind, or a favorite to promote. There have been some big changes recently, and there are some big changes coming.
Speciality Searchs:
Find Articles Good for researching articles and documentation. Dictionary Good for looking up words, meanings, and acroynyms Medical Web MD is the doctor / patient research center. White Pages Look up a persons phone number. Yellow Pages Look up a business number or address. Alternate Site 1-800 Find those toll-free 800 numbers Maps Good for finding maps, motels, reservations, etc.
Opinion: The thing that bothers me in this whole scheme of things is the role that Microsoft does play, and is playing. Not that MS forced Netscape out of business or anything, but they did land such a heavy hand that we saw Netscape and AOL merge just to stay alive. They place by default the whole MSN "search" engine as where you go when you click on "search" inside the Internet Explorer browser. Still not a problem, EXCEPT - you can't register your site with them unless you are willing to cough up some hard earned money. At the moment, they do post results from other databases that you can register for, but to be one of their "bCentral" database listings - you have to pay. The whole thing is, they never built a database to begin with, they purchased a very popular search engine a couple years ago (snap.com I think it was, but I could be wrong on that point), and went on from there. The advantage the public has with this "search threat" is that more and more people are aware of the Internet, how it works, and how to find the right search tools. Information is the best defense against computer monopolies.
What does this mean to you the surfer? It means that down the road, chances are that by default if you don't know the in's and out's of searching, and you just click on search ... the results you get won't be anywhere near as unbiased as it would be if you knew how search engines worked. You'll get fed the big money sights that pay Microsoft every month, and you'll never find your friends little site about his fishing tips.
Right now (mid 2003) Google is the big boy in the yard, and in my opinion they should be. Anyone can submit a site, and even say list my site in this catagory. Google does the search, looks at the page, and rates it as best it can. A site that offers hand painted eggs, but only has a page that says "coming soon, here's my phone number" will get listed much lower than the site that tells you what kind of paint to use, what brushes are best, and how to paint the egg.
Yahoo, the other big player, and with the purchase of Inktomi, may become just as big as Google. Remember too, not all that long ago, Yahoo was the number one engine, period. So between the two of them, it should keep things pretty honest.
Search Index | Help Index | Home | Sitemap
updated: 07.15.2003
Copyright© Charles H. Davis 1997 - 2003