November 19, 2003

What is recommended
Since most websites normally change often and new content is constantly created or updated, it is strongly recommended that the Robots.txt file in your website be re-evaluated at least once a month. If necessary, it only takes a minute or two to edit this small file in order to make the changes required. Never assume that ‘it must be OK, so I don’t need to bother with it’. Take a few minutes and look at the way it’s written. Ask yourself these questions:

1- Did I add some sensitive files recently?
2- Are there new sections I don’t want indexed?
3- Is there a section I want indexed but isn’t?

As a rule of thumb, even before adding a file or a group of files that contain sensitive information that you don’t want to be indexed by the search engines, you should edit your Robots.txt file before uploading those files to your server. Make sure you place them in a separate directory. You could name it: private_files or private_content and add each of those directories to your Robots exclusion file to prevent the spiders from indexing any of those private directories.

Also, if you find that you have files in a separate directory but you want them indexed, if those public files have been on your server for more than a month and are still not indexed, have a look at your Robots.txt file to make certain there are no errors in any of it’s commands.

<<< Previous page              Next: Re-evaluate your Robots file periodically >>>

Pages:     1     2     3     4     5     6     7

Article written by Serge Thibodeau,
President & CEO,
Rank for $ales
Copyright (c) Serge Thibodeau 2003

Unless otherwise specified, all content and material on this site is copyrighted by Serge Thibodeau of rankforsales.com and may not be reproduced by any means without express written permission. Using my content without permission is a theft of my work. Please contact sthibodeau@rankforsales.com to discuss certain reprint options that would be acceptable.

You can read some of Serge Thibodeau's exclusive comments that are not posted on this website. Visit his personal blog by clicking here. For hardware, software or IT-related technology questions, it is recommended you visit www.techblog.org

We strongly suggest you bookmark our web site by clicking here.

Get your business or company listed in the Global Business Listing directory and increase your business. It takes less then 24 hours to get a premium listing in the most powerful business search engine there is. Click here to find out all about it.

Rank for $ales strongly recommends the use of WordTracker to effectively identify all your right industry keywords. Accurate identification of the right keywords and key phrases used in your industry is the first basic step in any serious search engine optimization program. The keywords you think are the best may be totally different than the ones recommended by WordTracker. Click here to start your keyword and key phrase research.

You can link to the Rank for Sales web site as much as you like. Read our section on how your company can participate in our reciprocal link exchange program and increase your rankings in all the major search engines such as Google, AltaVista, Yahoo and all the others.