Page 1 of 1
robot file
Posted: Tue Feb 20, 2007 6:39 pm
by bradystjames
i just started a new site. do i really need to put a robot.txt in it. or will it be ok with out one.
Robots.txt file
Posted: Wed Feb 21, 2007 5:19 pm
by peterko
Yes, that's right. You should use robots.txt file only if you want some of your webpages to get skipped from robot visiting and indexing.
But if you want all of your pages to be visible for the robots, you don't really need robots.txt file
Have a great day,
Peter
Posted: Wed Feb 21, 2007 6:01 pm
by harrysmith
Since your site is new, you should not play with robots.txt file as it will prevent spiders from crawling your site...which may affect your indexing process.
It seems that you are not much aware of robots.txt file, so before using it please go through a article at :
http://blog.webhosting.uk.com/2007/01/0 ... stxt-file/
Posted: Thu Feb 22, 2007 4:20 am
by AfricanMango1
Some search engines mess up with the robots.txt, I wouldn't recommend you to use it if you don't really need to exclude specific pages.
Posted: Thu Feb 22, 2007 4:32 pm
by bradystjames
It seems that you are not much aware of robots.txt file, so before using it
no, i know what it does, i was just asking what if i didn't put it in. google webmaster tools have it listed all over the place. i do use it for my main site.
Posted: Mon Sep 17, 2007 12:38 am
by weblaunchphxx
It depends on you wether you want to crawl a particular page by the spider or not.