Size Limit

[real_estate]

Hi,
Does google, MSN or Yahoo have a size limit ? Is their a size limit on robots.txt fiel ? After a number of URL's do the bots stop reading them ?

[kremit]

Theres a certain amount of pages Google will index on your site depending on:

1) the quality of that page (in Googles eyes)
2) The links to that page (influences what Google indexes)


The bottom line is, if you have a good site (and reputable), or got good SEO, you shouldnt have any problem getting a LOT of your pages listed in Google.


In short, Ive never heard of an index limit, and couldnt imagine it to be true.

[ChiefLee]

Your robots.txt file should be a VERY small text file. All you really want to do is tell the robots what sections of your site to ignore, and we list our sitemap.xml file in it too (not to ignore, but to allow).

Here's our robots.txt file:

User-agent: *
Disallow: /admin
Disallow: /proofs
Disallow: /order.php
Disallow: /search.php
Disallow: /account.php
Sitemap: /sitemap.xml

[shakir]

Hi,
Does google, MSN or Yahoo have a size limit ? Is their a size limit on robots.txt fiel ? After a number of URL's do the bots stop reading them ?

I thing no.... But for site map maximum 100 links are better