Search Engine Forums, SEO Forum

can any body sort me out form ROBOT.txt, and what are the uses of ROBOT.txt and how we use in our website.

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.

The location of robots.txt is very important. It must be in the main directory because otherwise user agents (search engines) will not be able to find it – they do not search the whole site for a file named robots.txt. Instead, they look first in the main directory (i.e. http mydomain com/robots.txt) and if they don't find it there, they simply assume that this site does not have a robots.txt file and therefore they index everything they find along the way. So, if you don't put robots.txt in the right place, do not be surprised that search engines index your whole site.

The concept and structure of robots.txt has been developed more than a decade ago and if you are interested to learn more about it, visit http www.robotstxt org/ or you can go straight to the Standard for Robot Exclusion because in this article we will deal only with the most important aspects of a robots.txt file. Next we will continue with the structure a robots.txt file.

Here's another good read: Freefind.com/library/howto/robots/

"Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all.

Thanks for sharing

Robots.txt is a text (not HTML) that is put in place to tell search robots which pages you want them not to visit. Robots.txt is not required for the search engines but generally search engines obey what they say no. It is important to note that robots.txt is not a way to prevent search engines index your site (ie, not a firewall, or some kind of password) and that put a robots.txt file is something like like putting a note: "Please do not enter" an open door - for example, you can prevent thieves from entering good, but the kids do not open the door and enter. That's why we say that if you really sen data as possible, which is too naive to expect that a robots.txt file to avoid being indexed and displayed in search results.

Thanks to all for sorting me out from ROBOT.txt

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit.

You don’t like search engine crawling for your website particular one page so you use robot.txt process

robot. txt is a text file to put your website for tell search engine robot which pages would you like visit or not. for example your website have many pages but some pages you do not want to crawl this pages. in this cases you put robot. txt as a disallow for not crawling pages.

It is nothing but one text file that command to spider.which URLs would allow to crawl.

The robots.txt file is a set of instructions for visiting robots (spiders) that index the content of your web site pages.

Robot.txt is a text file you put your website pages for which pages would you like visit or not.

The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots on how to crawl & index pages on their website.

The robots exclusion standard, also known more commonly as Robots.txt, is a text file present in the root directory of a website. The Robots.txt file is a convention created to direct the activity of search engine crawlers or web spiders.