Page 1 of 1
robot.txt
Posted: Thu Jul 27, 2006 10:54 am
by www.pricepoint.com
How effective is the robot.txt in all aspects of seo tactics and submission tools?
[URL removed by moderator]
Posted: Mon May 18, 2009 5:45 pm
by mrandrei
Yes, it can affect your ranking. Robots.txt is a file which commands the spider about what pages you want to get indexed.
hi
Posted: Fri Jun 19, 2009 12:03 am
by sampath
The Robots file you want only, if you want stop crawling any files or directory by google, otherwise there is not much effect for seo by Robots file.
Posted: Fri Jun 19, 2009 1:55 am
by Roger_Silvester
Robots.txt file contains restrictions for Web Spiders, telling them where they have permission to search. A Robots.txt is like defining rules for search engine spiders (robots) what to follow and what not to. It should be noted that Web Robots are not required to respect Robots.txt files, but most well written Web Spiders follow the rules you define.
Posted: Fri Jun 19, 2009 8:32 am
by foodsnark
Robot don't importance.
Posted: Sun Jun 21, 2009 6:25 pm
by rayter
^ Amen to that! You should add a robots.txt file to the root directory of all your sites to help control the indexing of your site by robots that ignore the <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"> command.
Posted: Sun Jun 21, 2009 6:48 pm
by mdamin76
Search engines would respect every "Allow" and "Disallow" attribute in your robot.txt file. Although it's not so important for some sites, it's a useful way to control which page within your site is allowed to be crawled by search engines spider/crawler.
Posted: Mon Jun 22, 2009 3:53 am
by signaltelevision
mdamin76 wrote:Search engines would respect every "Allow" and "Disallow" attribute in your robot.txt file. Although it's not so important for some sites, it's a useful way to control which page within your site is allowed to be crawled by search engines spider/crawler.
I agree with him.