10. The method of claim 1 , wherein providing the crawl rate mechanism includes providing, for concurrent display: the current crawl rate limit associated with the crawling of the respective website; crawl data, including statistical information associated with crawling the respective website; and an interface for enabling the respective owner to select the new crawl rate limit.
10. The method of claim 1 , wherein providing the crawl rate mechanism includes providing, for concurrent display: the current crawl rate limit associated with the crawling of the respective website; crawl data, including statistical information associated with crawling the respective website; and an interface for enabling the respective owner to select the new crawl rate limit. 11. The method of claim 10 , wherein the providing for concurrent display includes providing for display recommendations for selecting the new crawl rate limit, wherein the recommendations are based, at least in part, on whether the current crawl rate limit is the limiting factor in crawling the respective website.