Search Engine Forums, SEO Forum

A robots.txt file will stop the Robot from going to some pages with sensitive material, web pages that you don't want to be found through Google search (for instance, the "Login page"), pages that are not important or can be negative for your rankings. And, you can direct the Robot to other, keyword-rich pages, instead.

So, if there's something to hide, a robots.txt file is a must for your website. It helps you keep the Robot away from anything that's not good for your Search Engine rankings. Just instruct him not go in there and he'll take your word.

You can make a robots.txt file yourself, though it's rather your webmaster's business. So ask your webmaster to write a robots.txt file for your site, or simply do it yourself.

Example: See our web design company robot.txt file by clicking the link and adding /robots.txt after the URL.

thanks for the info, in short robots.txt file tells search engine robots where to go and where not to when your site is crawled by robots.

Robots.txt file is a special text file that is always located in your Web server's root directory. Robots.txt file contains restrictions for Web Spiders, telling them where they have permission to search. A Robots.txt is like defining rules for search engine spiders (robots) what to follow and what not to. It should be noted that Web Robots are not required to respect Robots.txt files, but most well written Web Spiders follow the rules you define.

The format for the robots.txt file is special. It consists of records. Each record consists of two fields : a User-agent line and one or more Disallow: lines. The format is:
<Field> ":" <value>

The robots.txt file should be created in Unix line ender mode! Most good text editors will have a Unix mode or your FTP client *should* do the conversion for you. Do not attempt to use an HTML editor that does not specifically have a text mode to create a robots.txt file.

Thanks..

Mauritius honeymoon