Search Engine Forums, SEO Forum

A "bot" is a piece of software from a search engine that is built to go through every page of your site, categorize it, and

place it into a database.

Google has three well known bots: The Adsense bot, the Freshbot and the DeepCrawl.

The Adsense bot is used for publishers who have Adsense on their sites. As soon as a new page is created, the JavaScript

within the Adsense code sends a message to the Adsense bot, and it will come within 15 minutes to index the page so that it

can serve up the most relevant ads.

The Freshbot crawls the most popular pages on our website. Sites like Amazon.com and CNN.com have pages that are crawled

every ten minutes, since Google has learned that those pages have that amount of frequent changes. A typical site should

expect to have a freshbot visit every 1 to 14 days, depending on how popular those pages are.

Freshbot visit is that it finds all of the deeper links in your site. It places those links into a database so that when the

DeepCrawl occurs, it has a reference.

Once a month, the DeepCrawl bot visits your site and goes over all the links found by the Freshbot. This is the reason why it

can take up to a month for your entire site to be indexed in Google - even with the addition of a Google Sitemap.

Thank for your informations.
My website get the Bot Bomb
Last month Yahoo Slurp visit my site about 100K+/mo.
But not for this month.

Thanks for sharing this information. xml sitemap helps in indexing a site.
http://www.xml-sitemaps.com/ I thinks this is good for generating a site map.

Thank you for the lesson!
It was great. Also, XML sitemaps are great!
Is adding an html sitemap a good idea? does it help?

thanks!

most of the site owner cares only for the google robots, and for your information there are a strategies to be able google robots indexed your site or even your fresh content in just a minutes.

ericajoieake wrote:most of the site owner cares only for the google robots, and for your information there are a strategies to be able google robots indexed your site or even your fresh content in just a minutes.

Do you know where I can read up on that? Google hasn't crawled my site yet. It was just launched 2 weeks ago. I submitte it on webmaster tools...

Thanks for the valuable information.

very interesting information on how spiders work, thanks!

if your site has dynamic pages then sitemaps can help a bit to index the pages, but there are other advantages over submitting sitemap to search engines, you can find broken links, redirection errors and other issues related to HTTP.

try to find quality link it will indexed your site fast.