Page 1 of 1
Size Limit
Posted: Sun Oct 29, 2006 10:24 pm
by real_estate
Hi,
Does google, MSN or Yahoo have a size limit ? Is their a size limit on robots.txt fiel ? After a number of URL's do the bots stop reading them ?
Posted: Mon Oct 30, 2006 12:01 am
by kremit
Theres a certain amount of pages Google will index on your site depending on:
1) the quality of that page (in Googles eyes)
2) The links to that page (influences what Google indexes)
The bottom line is, if you have a good site (and reputable), or got good SEO, you shouldnt have any problem getting a LOT of your pages listed in Google.
In short, Ive never heard of an index limit, and couldnt imagine it to be true.
Posted: Sat Jul 04, 2009 6:33 am
by ChiefLee
Your robots.txt file should be a VERY small text file. All you really want to do is tell the robots what sections of your site to ignore, and we list our sitemap.xml file in it too (not to ignore, but to allow).
Here's our robots.txt file:
User-agent: *
Disallow: /admin
Disallow: /proofs
Disallow: /order.php
Disallow: /search.php
Disallow: /account.php
Sitemap: /sitemap.xml
Re: Size Limit
Posted: Sun Jul 05, 2009 6:57 am
by shakir
real_estate wrote:Hi,
Does google, MSN or Yahoo have a size limit ? Is their a size limit on robots.txt fiel ? After a number of URL's do the bots stop reading them ?
I thing no.... But for site map maximum 100 links are better