12. A computer system comprising: memory; one or more processors; and at least one program stored in the memory and executed by the one or more processors, the at least one program including instructions for: storing, for each website of a multiplicity of websites, a corresponding current crawl rate limit; comparing a maximum crawl rate for a respective website over a defined period of time with the current crawl rate limit for crawling the respective website to determine if the current crawl rate limit is a limiting factor in crawling the respective website; performing a website crawling management function in accordance with the determination of whether the current crawl rate limit is the limiting factor in crawling the respective website; and providing a crawl rate control mechanism to a respective owner of the respective website, wherein the crawl rate control mechanism enables selection of a new crawl rate limit corresponding to the respective website by the respective owner.
12. A computer system comprising: memory; one or more processors; and at least one program stored in the memory and executed by the one or more processors, the at least one program including instructions for: storing, for each website of a multiplicity of websites, a corresponding current crawl rate limit; comparing a maximum crawl rate for a respective website over a defined period of time with the current crawl rate limit for crawling the respective website to determine if the current crawl rate limit is a limiting factor in crawling the respective website; performing a website crawling management function in accordance with the determination of whether the current crawl rate limit is the limiting factor in crawling the respective website; and providing a crawl rate control mechanism to a respective owner of the respective website, wherein the crawl rate control mechanism enables selection of a new crawl rate limit corresponding to the respective website by the respective owner. 19. The computer system of claim 12 , wherein the at least one program further comprises instructions for determining a number of bytes downloaded from the respective website during one or more crawl sessions and for storing corresponding crawl data.