13. A computer system comprising: memory; one or more processors; and at least one program stored in the memory and executed by the one or more processors, the at least one program including: web crawl control instructions for controlling crawling of each website of a multiplicity of websites, each website having a corresponding current crawl rate limit, the web crawl control instructions including: instructions for crawling a respective website of the multiplicity of websites, in accordance with the current crawl rate limit corresponding to the respective website, to download documents from the respective website for inclusion in a database; instructions for storing crawl data associated with the crawling of the respective website; and instructions for providing, for display, a crawl rate control mechanism to a respective owner of the respective website, including providing, for display to the respective owner, at least a portion of the crawl data, and enabling selection, by the respective owner, of a new crawl rate limit corresponding to the respective website.
13. A computer system comprising: memory; one or more processors; and at least one program stored in the memory and executed by the one or more processors, the at least one program including: web crawl control instructions for controlling crawling of each website of a multiplicity of websites, each website having a corresponding current crawl rate limit, the web crawl control instructions including: instructions for crawling a respective website of the multiplicity of websites, in accordance with the current crawl rate limit corresponding to the respective website, to download documents from the respective website for inclusion in a database; instructions for storing crawl data associated with the crawling of the respective website; and instructions for providing, for display, a crawl rate control mechanism to a respective owner of the respective website, including providing, for display to the respective owner, at least a portion of the crawl data, and enabling selection, by the respective owner, of a new crawl rate limit corresponding to the respective website. 22. The computer system of claim 13 , including instructions for providing, for display, resource usage statistics corresponding to resources of the respective website used during a plurality of prior crawl visits of the respective website.