23. A non-transitory computer readable storage medium storing at least one program for execution by a computer system, the at least one program comprising instructions for: storing, for each website of a multiplicity of websites, a corresponding current crawl rate limit; comparing a maximum crawl rate for the respective website over a defined period of time with the current crawl rate limit for crawling the respective website to determine if the current crawl rate limit is a limiting factor in crawling the respective website; and performing a website crawling management function in accordance with the determination of whether the current crawl rate limit is the limiting factor in crawling the respective website; and providing a crawl rate control mechanism to a respective owner of the respective website, wherein the crawl rate control mechanism enables selection of a new crawl rate limit corresponding to the respective website by the respective owner.
23. A non-transitory computer readable storage medium storing at least one program for execution by a computer system, the at least one program comprising instructions for: storing, for each website of a multiplicity of websites, a corresponding current crawl rate limit; comparing a maximum crawl rate for the respective website over a defined period of time with the current crawl rate limit for crawling the respective website to determine if the current crawl rate limit is a limiting factor in crawling the respective website; and performing a website crawling management function in accordance with the determination of whether the current crawl rate limit is the limiting factor in crawling the respective website; and providing a crawl rate control mechanism to a respective owner of the respective website, wherein the crawl rate control mechanism enables selection of a new crawl rate limit corresponding to the respective website by the respective owner. 32. The non-transitory computer program product of claim 23 , wherein the instructions for providing the crawl rate control mechanism further include instructions providing, for concurrent display: the current crawl rate associated with the crawling of the respective website; crawl data, including statistical information associated with crawling the respective website; and an interface for enabling the respective owner to select the new crawl rate limit.