3. The computer-implemented method of claim 1 , further comprising comparing a maximum crawl rate for the respective website over a defined period of time with the current crawl rate limit for crawling the respective website to determine if the current crawl rate limit is a limiting factor in crawling the respective website; and performing a website crawling management function in accordance with the determination of whether the current crawl rate limit is a limiting factor in crawling the respective website.
3. The computer-implemented method of claim 1 , further comprising comparing a maximum crawl rate for the respective website over a defined period of time with the current crawl rate limit for crawling the respective website to determine if the current crawl rate limit is a limiting factor in crawling the respective website; and performing a website crawling management function in accordance with the determination of whether the current crawl rate limit is a limiting factor in crawling the respective website. 5. The computer-implemented method of claim 3 , wherein the current crawl rate limit is a limiting factor only when a difference between the current crawl rate limit and the maximum crawl rate for the respective website over the defined period of time is less than a predefined quantity.