Page 1 of 1
Posted: Wed Oct 11, 2006 2:24 pm
by kremit
No, certain search engines have certain amounts of time that they cache your robots.txt.
They will look at it again every time they feel like it. (for some every time they crawl your site)
Posted: Wed Oct 11, 2006 6:12 pm
by dudelookslikealady
To give you an idea, google will re-index every 30 days for sites with a PR less than 2. How do I know this I look at the google base stats for my sites. I use this fact when I but used domain names.
For a new site, your robots.txt file will do the trick, unless "your site has pages with backlinks from other indexed sites".
I know my SEO inside out and do not need to have my site listed in my footer.
What you need to know about google is it reads your content and makes sure it is quality and not articles or other material repeated elsewhere; unless you "google bomb". eg type "LIAR" for a search in google for example of google bombing.
You will be safe, but beware MSN works on page most updated (and unique content) to give ranking, I wish I could give examples but will not. Yahoo is different altogether and gives QUALITY search results, you will also find yahoo is the number one search engine, see ALEXA.COM for web rankings. Yahoo will index most of your site regardless of the robots.txt file if you have many external links to individual pages. Yahoo will index from these links before even reading your robots text file.
PS ignore any comments from moderators from india they are only trying to sell you their product, and know little about SEO or Web stats for that matter, especially if they sound like views from the creator........
PPS..... your site is so so so just a adsense type site it is very unlikely to be indexed in any depth, try to update the site with content people want, such as where is the cheapest petrol or a forum to debate. By adding your footer to this site your homepage will be indexed!!
_________________
www.yahoo.com
Posted: Thu Oct 12, 2006 10:34 am
by viradian
What is Google Base and how does it work?
Posted: Thu Oct 12, 2006 10:34 am
by viradian
What is Google Base and how does it work?
Re: Does robots file keep SE's away for good?
Posted: Thu Oct 12, 2006 9:49 pm
by link.ezer.com
gotmpg wrote:......... or does it say "don't look here for now".
Thanks in advance!
it is ok ..... or you can say:
don't look here
for this time
some basic
robots & disallow ussage memo:
http://3w.ezer.com/robots/robots.txt/disallow.asp
Posted: Tue Oct 24, 2006 4:15 am
by Adan
The SEs will try to spider your site everytime they see a BL from any site that they're spidering, so when you remove the robots.txt they will keep visiting your site everytime they follow a BL to your site.