Spiders & Robots

*gayne* · 02-11-2009, 08:43 AM

I have a new web site that was indexed by Google 6 mths ago. Now I'm thinking of changing it substantially and wander how often the spiders and robots search for new pages.

Can I redirect from old URLs to new ones in the meantime?

Arthur · 02-11-2009, 08:49 AM

It differs from search engine to search engine, but for Google -

From:
How often does Google crawl the web?

Quote:

Google's spiders regularly crawl the web to rebuild our index. Crawls are based on many factors such as PageRank, links to a page, and crawling constraints such as the number of parameters in a URL. Any number of factors can affect the crawl frequency of individual sites.

Our crawl process is algorithmic; computer programs determine which sites to crawl, how often, and how many pages to fetch from each site. We don't accept payment to crawl a site more frequently. For tips on maintaining a crawler-friendly website, please visit our webmaster guidelines.

-Arthur

*gayne* · 02-11-2009, 08:56 AM

Thanks Arthur,
Can I tell by looking at my logs? If so, what do I look for?

Arthur · 02-11-2009, 09:12 AM

You can search for Googlebot (Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)) in the All Agents list, or you can get detailed stats from Google's Webmaster Tools (you need a Google account for that).

-Arthur

*gayne* · 02-11-2009, 09:30 AM

Great link, thanks a lot