robot file

[bradystjames]

i just started a new site. do i really need to put a robot.txt in it. or will it be ok with out one.

[peterko]

Yes, that's right. You should use robots.txt file only if you want some of your webpages to get skipped from robot visiting and indexing.
But if you want all of your pages to be visible for the robots, you don't really need robots.txt file

Have a great day,
Peter

[harrysmith]

Since your site is new, you should not play with robots.txt file as it will prevent spiders from crawling your site...which may affect your indexing process.
It seems that you are not much aware of robots.txt file, so before using it please go through a article at : http://blog.webhosting.uk.com/2007/01/0 ... stxt-file/

[AfricanMango1]

Some search engines mess up with the robots.txt, I wouldn't recommend you to use it if you don't really need to exclude specific pages.

[bradystjames]

It seems that you are not much aware of robots.txt file, so before using it

no, i know what it does, i was just asking what if i didn't put it in. google webmaster tools have it listed all over the place. i do use it for my main site.

[weblaunchphxx]

It depends on you wether you want to crawl a particular page by the spider or not.