How Google Indexes our site

priamvathaa · Post by **priamvathaa** » Tue Jun 03, 2008 4:45 am

A "bot" is a piece of software from a search engine that is built to go through every page of your site, categorize it, and

place it into a database.

Google has three well known bots: The Adsense bot, the Freshbot and the DeepCrawl.

The Adsense bot is used for publishers who have Adsense on their sites. As soon as a new page is created, the JavaScript

within the Adsense code sends a message to the Adsense bot, and it will come within 15 minutes to index the page so that it

can serve up the most relevant ads.

The Freshbot crawls the most popular pages on our website. Sites like Amazon.com and CNN.com have pages that are crawled

every ten minutes, since Google has learned that those pages have that amount of frequent changes. A typical site should

expect to have a freshbot visit every 1 to 14 days, depending on how popular those pages are.

Freshbot visit is that it finds all of the deeper links in your site. It places those links into a database so that when the

DeepCrawl occurs, it has a reference.

Once a month, the DeepCrawl bot visits your site and goes over all the links found by the Freshbot. This is the reason why it

can take up to a month for your entire site to be indexed in Google - even with the addition of a Google Sitemap.

Lokina2 · Post by **Lokina2** » Sun Jun 08, 2008 8:33 am

Thank for your informations.
My website get the Bot Bomb
Last month Yahoo Slurp visit my site about 100K+/mo.
But not for this month.

stevewaganer · Post by **stevewaganer** » Mon Jun 16, 2008 11:53 pm

Thanks for sharing this information. xml sitemap helps in indexing a site.
http://www.xml-sitemaps.com/ I thinks this is good for generating a site map.

Haz9119 · Post by **Haz9119** » Mon Jun 23, 2008 8:36 pm

Thank you for the lesson!
It was great. Also, XML sitemaps are great!
Is adding an html sitemap a good idea? does it help?

thanks!

ericajoieake · Post by **ericajoieake** » Wed Jun 25, 2008 4:08 pm

most of the site owner cares only for the google robots, and for your information there are a strategies to be able google robots indexed your site or even your fresh content in just a minutes.

Haz9119 · Post by **Haz9119** » Wed Jun 25, 2008 8:25 pm

ericajoieake wrote:most of the site owner cares only for the google robots, and for your information there are a strategies to be able google robots indexed your site or even your fresh content in just a minutes.

Do you know where I can read up on that? Google hasn't crawled my site yet. It was just launched 2 weeks ago. I submitte it on webmaster tools...

delhiseo · Post by **delhiseo** » Mon Jun 30, 2008 2:16 am

Thanks for the valuable information.

chizarium · Post by **chizarium** » Fri Jul 11, 2008 7:35 am

very interesting information on how spiders work, thanks!

seo-2009 · Post by **seo-2009** » Sat Jul 12, 2008 1:32 am

if your site has dynamic pages then sitemaps can help a bit to index the pages, but there are other advantages over submitting sitemap to search engines, you can find broken links, redirection errors and other issues related to HTTP.

Ka-canor · Post by **Ka-canor** » Tue Jul 22, 2008 9:27 pm

try to find quality link it will indexed your site fast.

Search Engine Forums, SEO Forum

How Google Indexes our site

How Google Indexes our site

Who is online