27. The computer program product of claim 25 , the web crawl control instructions further comprising, instructions for: comparing a maximum crawl rate for the respective website over a defined period of time with the current crawl rate limit for crawling the respective website to determine if the current crawl rate limit is a limiting factor in crawling the respective website; and performing a website crawling management function in accordance with the determination of whether the current crawl rate limit is a limiting factor in crawling the respective website.
27. The computer program product of claim 25 , the web crawl control instructions further comprising, instructions for: comparing a maximum crawl rate for the respective website over a defined period of time with the current crawl rate limit for crawling the respective website to determine if the current crawl rate limit is a limiting factor in crawling the respective website; and performing a website crawling management function in accordance with the determination of whether the current crawl rate limit is a limiting factor in crawling the respective website. 28. The computer program product of claim 27 , the web crawl control instructions further comprising: instructions for informing the respective owner that a request for a faster crawl rate may not change the current crawl rate for crawling the respective website when the current crawl rate limit is not a limiting factor in crawling the respective website.