Configuring your Robots.txt

Votes: 0

The ability of users successfully finding your website via search engines such as AltaVista, Excite, Google, HotBot, Infoseek, Lycos, and others can be greatly influenced by a few basic techniques.

The first step is to request that the various search engines visit and index your website. Although you may submit a simple request to a large number of search engines, experience shows that it is much more effective to optimize your website to attract only a few of the most popular engines. We recommend visiting the websites of popular search engines, such as Google http://www.google.com/, and following their suggested procedures.

The second step is to insure that once the search engines visit your site and begin their indexing procedure they do not create server errors attempting to index pages that are incompatible for search activities. Additionally, you may not wish to have sensitive pages such as order forms, customer information, etc. to be indexed. To help you manage how the search engines index your website, we have placed a default "robot.txt" document in your root directory. This document is recognized by the better search engines and some of the large engines will not index your website unless it is present. While we request that you do not remove this document, we encourage you to revise it to better suit your needs. The following information is provided only as a guide. You may wish to search the web for additional information.

The robot.txt file should be in a simple text format such as Notepad. The basic format for preventing all robots from indexing your website is:

User-agent: *
Disallow: /

The format for preventing all robots from visiting a specific directory is:

User-agent: *
Disallow: /DirectoryName

The format for preventing all robots from visiting a number of specific directories is:

User-agent: *
Disallow: /DirectoryName1
Disallow: /DirectoryName2
Disallow: /DirectoryName3

We encourage you to add those directories you feel may not be suitable for indexing. If your website has dynamically created webpages from a database, the search engines cannot index those pages. It is best to prevent the indexing of those pages. We strongly recommend that you have a number of static html pages that can be indexed by the search engines and are optimized to provide you with good positioning results.

You may also wish to use meta tags within your html documents. Here are some basic rules:

A ROBOTS meta tag with NOINDEX anywhere in its value will cause the page not to be indexed.

A ROBOTS meta tag with NOFOLLOW anywhere in its value will cause links on the page to not be followed. If these same links are encountered on any other page and that page does not also have the ROBOTS meta tag, they will be followed.

Here are some basic examples:

<META name="ROBOTS" content="NOINDEX">
(This means that the search engines can not index this page but will follow any links on it)

<META name="ROBOTS" content="NOFOLLOW">
(This means that the search engines can index this page but will not follow any links on it)

<META name="ROBOTS" content="NOINDEX,NOFOLLOW">
(This means that the search engines can not index this page and will not
follow any links on it)

The following information is provided only as a guide. You may wish to search the web for additional information.