Re: what is robots.txt?
Posted: Wed Oct 02, 2013 2:06 am
It's the standard file for specifying exclusion of certain directories on a website such as the cgi-bin, tmp and whatever else you don't want search engines to crawl and index. It's easy to implement: http://en.wikipedia.org/wiki/Robots_exclusion_standard