robot.txt

For discussion of general Search Engine Optimization, not specific to any engine.
Post Reply
www.pricepoint.com
Posts: 6
Joined: Mon Jul 10, 2006 11:37 am
Location: Gardena, CA
Contact:
Contact www.pricepoint.com

robot.txt

Post by www.pricepoint.com »

How effective is the robot.txt in all aspects of seo tactics and submission tools?

[URL removed by moderator]
Matthew Miladinovich
www.pricepoint.com
Top
mrandrei
Posts: 371
Joined: Sun Mar 01, 2009 8:16 pm
Location: United States
Contact:
Contact mrandrei

Post by mrandrei »

Yes, it can affect your ranking. Robots.txt is a file which commands the spider about what pages you want to get indexed.
Top
sampath
Posts: 22
Joined: Tue Nov 18, 2008 7:20 am
Contact:
Contact sampath

hi

Post by sampath »

The Robots file you want only, if you want stop crawling any files or directory by google, otherwise there is not much effect for seo by Robots file.
[url=http://www.medchoiceny.com/]Medical supplies[/url]
Top
Roger_Silvester
Posts: 57
Joined: Fri Apr 17, 2009 11:17 pm
Contact:
Contact Roger_Silvester

Post by Roger_Silvester »

Robots.txt file contains restrictions for Web Spiders, telling them where they have permission to search. A Robots.txt is like defining rules for search engine spiders (robots) what to follow and what not to. It should be noted that Web Robots are not required to respect Robots.txt files, but most well written Web Spiders follow the rules you define.
[url=http://www.paydaybank.co.uk]Payday[/url] | [url=http://www.mypayday.co.uk]Payday Advance[/url]
Top
foodsnark
Posts: 108
Joined: Sat May 23, 2009 8:27 am
Contact:
Contact foodsnark

Post by foodsnark »

Robot don't importance.
Top
rayter
Posts: 923
Joined: Mon Mar 02, 2009 5:04 pm
Location: United States
Contact:
Contact rayter

Post by rayter »

^ Amen to that! You should add a robots.txt file to the root directory of all your sites to help control the indexing of your site by robots that ignore the <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"> command.
Top
mdamin76
Posts: 817
Joined: Wed Jan 14, 2009 6:58 pm
Location: On Stage
Contact:
Contact mdamin76

Post by mdamin76 »

Search engines would respect every "Allow" and "Disallow" attribute in your robot.txt file. Although it's not so important for some sites, it's a useful way to control which page within your site is allowed to be crawled by search engines spider/crawler.
Top
signaltelevision
Posts: 69
Joined: Sun Apr 12, 2009 4:00 pm
Contact:
Contact signaltelevision

Post by signaltelevision »

mdamin76 wrote:Search engines would respect every "Allow" and "Disallow" attribute in your robot.txt file. Although it's not so important for some sites, it's a useful way to control which page within your site is allowed to be crawled by search engines spider/crawler.
I agree with him.
[url=http://www.signaltelevision.com/]Signal Television[/url]
Top
Post Reply

Return to “SEO Forum”

Who is online

Users browsing this forum: Google [Bot] and 16 guests