Search Engine Basics

A search engine is one of the most important tool that helps you find information on the web. Search engines are giant sized automated cataloging & retrieval systems. They typically have large databases of web pages & other information found on the net. Upon specific query by the users these databases are scanned & the matching results displayed.

The utility of search engines in web search lies in the fact that they are the repositories of large amounts of information which can be searched very conveniently using certain keywords. This natural language searching & many advanced features now available with search engines makes the recipe tantalizingly attractive. Hence it is no wonder that 78% of the users start their search process for any information via a search engine.

Before we go any further lets clarify the difference between a search engine & a web directory. Search engines as we have learnt are automated software based systems which aid the web search. Directories are generally human based indexes in which the web sites are visited & cataloged by human beings. In the directories the web sites are included in the categories & sub categories . These categories or taxonomies are specific to specific directories.

Users can search for relevant sites in a directory by either mining on the subject they are looking for or searching through keywords. The most famous commercial directory is Yahoo. There are hundreds of directories available. We will be discussing them later. A point to note here is that most of the SE have now started to mix their web results with some kind of directory results.

Though there is a category of SE which are not crawler based. They accept money to list your web site. These paid listings search engines will be dealt in greater detail in subsequent section.
For the moment, lets look at search engine internals a little more closely:

What goes into making a search engine

To have a birds eye view of search engine functionality will help the searcher evaluate the relative search engine merits amongst themselves as also compared with other search resources.

Search engines as we know them, to put simply, are divided into three parts:

Crawler
Index
Search software
Lets take them one by one.

Search Engine Crawler: A traditional database is populated using conventional data entry methods. Not so on the web. Search engines send a software called spider or crawler to the wild web. This spider will visit a web site once it hears of it (through other links pointing to the web site, cataloguing in some directory or direct submission to the SE) & will start to crawl around. Based on links in the site it will digest as much information as it has been programmed to receive about the web site. Thus crawlers are search engines inventory(index) suppliers.

Search Engine Index: Index or the catalogue is that huge book which contains a copy of all the web pages that have been spidered. This in other words is the database of the search engine. Once the site has been crawled it is added to the index of the search engine. This database is refreshed & updated as per the settings done by specific search engines on a regular basis.

Search Engine Ranking Software: This is the sauce which is mostly instrumental in different crawler based search engines serving different platters on being queried about the same recipe. In other words, this is the area where most of the search engines differ in the way they handle queries & display results. This portion of the SE deals with scanning the database (index) according to specific user queries & then ranking the matches found.