What is ROBOT.txt ?

For discussion of general Search Engine Optimization, not specific to any engine.
wlcindia
Posts: 64
Joined: Thu Sep 08, 2011 9:46 pm
Location: India
Contact:

What is ROBOT.txt ?

Postby wlcindia » Tue Oct 04, 2011 2:46 am

can any body sort me out form ROBOT.txt, and what are the uses of ROBOT.txt and how we use in our website.

yoangames
Posts: 2
Joined: Sun Oct 02, 2011 12:54 pm
Contact:

Re: What is ROBOT.txt ?

Postby yoangames » Tue Oct 04, 2011 8:16 am

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.

The location of robots.txt is very important. It must be in the main directory because otherwise user agents (search engines) will not be able to find it – they do not search the whole site for a file named robots.txt. Instead, they look first in the main directory (i.e. http mydomain com/robots.txt) and if they don't find it there, they simply assume that this site does not have a robots.txt file and therefore they index everything they find along the way. So, if you don't put robots.txt in the right place, do not be surprised that search engines index your whole site.

The concept and structure of robots.txt has been developed more than a decade ago and if you are interested to learn more about it, visit http www.robotstxt org/ or you can go straight to the Standard for Robot Exclusion because in this article we will deal only with the most important aspects of a robots.txt file. Next we will continue with the structure a robots.txt file.

VladTepes
Posts: 1463
Joined: Sat Dec 16, 2006 9:42 am
Contact:

Re: What is ROBOT.txt ?

Postby VladTepes » Thu Oct 06, 2011 1:08 am

Here's another good read: Freefind.com/library/howto/robots/
Pet Beds - Quality, style and comfort plus healing and soothing warm effects for cold winter season.

Ken Peterson
Posts: 184
Joined: Wed Aug 31, 2011 4:51 am
Location: Florida
Contact:

Re: What is ROBOT.txt ?

Postby Ken Peterson » Tue Oct 11, 2011 3:32 am

"Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all.

lindavanbuskirk2
Posts: 4
Joined: Tue Oct 11, 2011 3:44 am
Contact:

Re: What is ROBOT.txt ?

Postby lindavanbuskirk2 » Tue Oct 11, 2011 4:01 am

Thanks for sharing

Jennifer21
Posts: 57
Joined: Fri Jan 21, 2011 12:50 am
Contact:

Re: What is ROBOT.txt ?

Postby Jennifer21 » Thu Oct 13, 2011 1:48 am

Robots.txt is a text (not HTML) that is put in place to tell search robots which pages you want them not to visit. Robots.txt is not required for the search engines but generally search engines obey what they say no. It is important to note that robots.txt is not a way to prevent search engines index your site (ie, not a firewall, or some kind of password) and that put a robots.txt file is something like like putting a note: "Please do not enter" an open door - for example, you can prevent thieves from entering good, but the kids do not open the door and enter. That's why we say that if you really sen data as possible, which is too naive to expect that a robots.txt file to avoid being indexed and displayed in search results.

wlcindia
Posts: 64
Joined: Thu Sep 08, 2011 9:46 pm
Location: India
Contact:

Re: What is ROBOT.txt ?

Postby wlcindia » Sat Oct 15, 2011 2:43 am

Thanks to all for sorting me out from ROBOT.txt

sslcerts
Posts: 18
Joined: Tue Oct 11, 2011 11:39 pm
Contact:

Re: What is ROBOT.txt ?

Postby sslcerts » Sun Oct 16, 2011 9:10 pm

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit.

solidtrustpay
Posts: 23
Joined: Thu Oct 20, 2011 8:52 pm
Contact:

Re: What is ROBOT.txt ?

Postby solidtrustpay » Thu Nov 17, 2011 10:54 pm

You don’t like search engine crawling for your website particular one page so you use robot.txt process

seoservicesforeign
Posts: 6
Joined: Wed Nov 30, 2011 9:39 am
Contact:

Re: What is ROBOT.txt ?

Postby seoservicesforeign » Thu Dec 01, 2011 5:29 am

robot. txt is a text file to put your website for tell search engine robot which pages would you like visit or not. for example your website have many pages but some pages you do not want to crawl this pages. in this cases you put robot. txt as a disallow for not crawling pages.

nile244
Posts: 4
Joined: Sat Dec 03, 2011 1:54 am
Contact:

Re: What is ROBOT.txt ?

Postby nile244 » Sat Dec 03, 2011 1:58 am

It is nothing but one text file that command to spider.which URLs would allow to crawl.

Bodavhen
Posts: 24
Joined: Tue Dec 06, 2011 2:55 am
Contact:

Re: What is ROBOT.txt ?

Postby Bodavhen » Tue Dec 06, 2011 10:09 pm

The robots.txt file is a set of instructions for visiting robots (spiders) that index the content of your web site pages.

seoservicesforeign
Posts: 6
Joined: Wed Nov 30, 2011 9:39 am
Contact:

Re: What is ROBOT.txt ?

Postby seoservicesforeign » Tue Dec 13, 2011 5:39 am

Robot.txt is a text file you put your website pages for which pages would you like visit or not.

Jajwicoy
Posts: 27
Joined: Wed Dec 14, 2011 3:56 am
Contact:

Re: What is ROBOT.txt ?

Postby Jajwicoy » Wed Dec 14, 2011 4:27 am

The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots on how to crawl & index pages on their website.

Malrgaf
Posts: 34
Joined: Tue Dec 20, 2011 9:34 pm
Contact:

Re: What is ROBOT.txt ?

Postby Malrgaf » Tue Dec 20, 2011 10:35 pm

The robots exclusion standard, also known more commonly as Robots.txt, is a text file present in the root directory of a website. The Robots.txt file is a convention created to direct the activity of search engine crawlers or web spiders.


Return to “SEO Forum”

Who is online

Users browsing this forum: No registered users and 15 guests