what is robots.txt?

[seomongo]

It's the standard file for specifying exclusion of certain directories on a website such as the cgi-bin, tmp and whatever else you don't want search engines to crawl and index. It's easy to implement: http://en.wikipedia.org/wiki/Robots_exclusion_standard