26. A a computer readable storage medium storing one or more programs for execution by one or more processors of a computer system, the one or more programs comprising: web crawl control instructions for controlling crawling of each website of a multiplicity of websites, each website having a corresponding current crawl rate limit, the web crawl control instructions including: instructions for crawling a respective website of the multiplicity of websites, in accordance with the current crawl rate limit corresponding to the respective website, to download documents from the respective website for inclusion in a database; instructions for storing crawl data associated with the crawling of the respective website; instructions for providing, for display, a crawl rate control mechanism to a respective owner of the respective website, including providing, for display to the respective owner, at least a portion of the crawl data, and enabling selection, by the respective owner, of a new crawl rate limit corresponding to the respective website; instructions for comparing a maximum crawl rate for the respective website over a defined period of time with the current crawl rate limit for crawling the respective website to determine if the current crawl rate limit is a limiting factor in crawling the respective website; and instructions for responding to a request to increase the current crawl rate for crawling the respective website by increasing the current crawl rate limit only when the current crawl rate limit is a limiting factor in crawling the respective website.
29. The computer readable storage medium of claim 26 , wherein the current crawl rate limit is a limiting factor only when a difference between the current crawl rate limit and the maximum crawl rate for the respective website over the defined period of time is less than a predefined quantity.