Search Engine Forums, SEO Forum

What is robots.txt? Is it a part of on-page?

The robots.txt file is used to keep web crawlers, spiders and robots from indexing certain sections of a site. You can find guidelines on creating a robots.txt file at http://www.javascriptkit.com/howto/robots.shtml

Read this post for more information on Robot.txt.
http://www.ojr.org/ojr/stories/080509niles-robots/

The robots.txt used for search engine crawlers, spiders and robots from indexing certain sections of a site.

Robots.txt for all search engines.
User-agent: *
Allow: /

I've seen that many websites don't use robots.txt & sitemap yet these have really good traffic and PR and comes on top for various keywords...so my question is... how do these websites get on such a higher place????

The major search engines index your site now time for the wrong robots.txt file is deleted. I suggest you make an existing file instead of just deleting the previous one. You can use this syntax.

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit.
Let me tell you some thing by example

This code allows all robots to crawl all files
user-agent:*
Disallow:

and this allow all robots to crawl all files except this image file
user-agent:*
Disallow:/images/

* robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
* the /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use.

Robots.txt is a text file which is used by website owner to give instructions to the search engine spider.This file tells the search engine which pages should be indexed and which pages should be ignored.

@Aneeta, Thanks for sharing such important information!

Thanks for the great info nad links.

Thanks for Information!!

Robots.txt is a useful file that keeps search engines away from indexing pages you do not want to be crawled. yes it is on page activity.

Robots.txt is a text file you place on your site to tell search robots which pages you want not. Robots.txt is not mandatory for search engines, but generally search engines obey what was not said to do. It is important to clarify that robots.txt is not a way to prevent search engines to index your site and the fact that you put a robots.txt file.