Search Engine Forums, SEO Forum

What is robots.txt file, and how i add it in my website? If anyone have any idea please share with me.

Robots.txt is a file which sits in root directory of the site and tells search engines which files to crawl and which not.

You can check the following link for getting more insight on this:

robotstxt.org/robotstxt.html

Very good tutorial here for robots.txt: Tools.seobook.com/robots-txt/

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site.

Thanks and Regards

Smith Jones

A robots.txt file restricts which is directory of search engine robots that crawl the web. Robots.txt file usage is sometimes ignored. On the other hand, it is an important factor for the webpages being indexed properly and very easy to setup.

Basically Robots text file is the file where you can allow or disallow user agent means robots or spiders to crawl or not to crawl that web page which you restricted. It is not necessary but good to have.

When your website will be crawled by spiders of search engines, robot.txt helps to inform the spiders that which page of site should not be crawled. It is a text file that lies in root directory of your site.

Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. The robots.txt file is a simple text file that must be placed in your directory .

robot.txt is controller of your web site.. you can suggest spiders to crawl your site or not....etc

Hello,

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.

Thanks & Regards,
Cole

SEO Plans wrote:What is robots.txt file, and how i add it in my website? If anyone have any idea please share with me.

It is a file through which we disallow any age from being index in search engines.

Robots.txt is nothing but file,with help of it,you can allow or disallow any pages to crawl by search engine spiders.

Robots.txt is a text file you place on your site to inform search robots which pages you want them not to visit. Robots.txt is not compulsory for search engines but usually search engines follow what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sensitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.

Correct Format of Robots.txt
The structure of a robots.txt is pretty simple (and barely flexible) – it is an endless list of user agents and disallowed files and directories. Basically, the syntax is as follows:
User-agent:
Disallow:
“User-agent” are search engines' crawlers and disallow: lists the files and directories to be excluded from indexing. In addition to “user-agent:” and “disallow:” entries, you can include comment lines – just put the # sign at the beginning of the line:
# All user agents are disallowed to see the /temp directory.
User-agent: *
Disallow: /temp/

robots.txt is used in SEO purposes to hide the coded files from being crawled by search engines.
Make an account in Google webmaster tools, add and verify your website and make your robots.txt file there. Google Webmaster will guide you how to make it

Robot.txt is used when you have duplicate content that you don’t want to be indexed, or you want to create a no follow rule for specific areas of your Websites, you need one of these files