The Search Engine Professionals at Rank for $ --- In business since 1997.
Back to our Homepage SEO Tips that will make a big difference in your rankings and our most popular ** How To ** section The most common myths about SEO -- Read what the experts have to say about today's most common SEO myths and misconceptions Frequently Asked Questions to Search Engine Optimization and Positioning Search Engine Optimization Industry News -- Stay in tune with the most recent developments in search engine technology and the SEO industry Contact Rank for $ales today and get your site's rankings high in the engines-- Right where they should be!


Search this site

More on the Robots exclusion protocol

November 19, 2003

Back in the spring of 2003, I wrote an article on the Robots exclusion protocol that generated a lot of emails and many questions. I am still getting them, so it seems that a more extended article is warranted. Often referred to as the Robots.txt file, the Robots exclusion protocol can be a very important part of your search engine optimization program, but it needs to be carefully implemented to be really successful.

If used incorrectly, this small and “innocent looking” text file can cause a lot of problems. It can even cause your site from being excluded from the search engine’s databases, it it’s not written correctly! In this extended article, I will show you how to correctly write it and make the Robots exclusion protocol an important part of your SEO efforts in attaining good visibility in the major search engines.

How the robots exclusion protocol works
Some of you may ask what is it and why do we need it? In a nutshell, as it’s name implies, the Robots exclusion protocol is used by Webmasters and site owners to prevent search engine crawlers (or spiders) from indexing certain parts of their Web sites. It could be for a number of reasons, such as sensitive corporate information, semi-confidential data, information that needs to stay private, or to prevent certain programs or scripts from being indexed, etc.

Next: The robots.txt exclusion protocol was developed at the end of 1993 >>>

Pages:     1     2     3     4     5     6     7

Fill out your e-mail address
to receive our free newsletter!

Article written by Serge Thibodeau,
President & CEO,
Rank for $ales
Copyright (c) Serge Thibodeau 2003

Unless otherwise specified, all content and material on this site is copyrighted by Serge Thibodeau of and may not be reproduced by any means without express written permission. Using my content without permission is a theft of my work. Please contact to discuss certain reprint options that would be acceptable.

You can read some of Serge Thibodeau's exclusive comments that are not posted on this website. Visit his personal blog by clicking here. For hardware, software or IT-related technology questions, it is recommended you visit

We strongly suggest you bookmark our web site by clicking here.

Get your business or company listed in the Global Business Listing directory and increase your business. It takes less then 24 hours to get a premium listing in the most powerful business search engine there is. Click here to find out all about it.

Rank for $ales strongly recommends the use of WordTracker to effectively identify all your right industry keywords. Accurate identification of the right keywords and key phrases used in your industry is the first basic step in any serious search engine optimization program. The keywords you think are the best may be totally different than the ones recommended by WordTracker. Click here to start your keyword and key phrase research.

Pay Rank for $ales securely with your Visa, MasterCard, Discover, or American Express credit card through the secure PayPal network. (Note: PayPal is an eBay company, and maintains a net free capital of US $ 50 Million).
VisaMasterCardDiscoverAmerican Express

You can link to the Rank for Sales web site as much as you like. Read our section on how your company can participate in our reciprocal link exchange program and increase your rankings in all the major search engines such as Google, AltaVista, Yahoo and all the others.

Powered by Avantex         Sponsered by Blog         Traffic stats by Site Clicks™

Site design by Mtl. Web D.      Get listed in Global B. Listing          Sponsered by Press Broadcast

Call Rank for Sales toll free from anywhere in the US or Canada:   1-800-631-3221

Home | SEO Tips | SEO Myths | FAQ | SEO News | Articles | Sitemap | Contact

Copyright © Rank for Sales 1997-2005   Terms of use    Privacy agreement    Legal disclaimer

                  Ce site est disponible en Français     H. Canada