1. A method of indexing documents in websites, the method comprising: on a server system having one or more processors and memory storing programs to be executed by the one or more processors: storing, for each website of a multiplicity of websites, a corresponding current crawl rate limit; comparing a maximum crawl rate for a respective website over a defined period of time with the current crawl rate limit for crawling the respective website to determine if the current crawl rate limit is a limiting factor in crawling the respective website; performing a website crawling management function in accordance with the determination of whether the current crawl rate limit is the limiting factor in crawling the respective website; and providing a crawl rate control mechanism to a respective owner of the respective website, wherein the crawl rate control mechanism enables selection of a new crawl rate limit corresponding to the respective website by the respective owner.
5. The method of claim 1 , further comprising: in response to a request to decrease the current crawl rate limit for crawling the respective website, decreasing the current crawl rate limit for crawling the respective website.