23. A non-transitory computer readable storage medium, for scheduling documents to be crawled by a search engine in an appropriate order to reduce visibility of stale content in web searching, storing one or more programs to be executed by a computer system, wherein the computer system includes a search engine program, the one or more programs comprising: instructions for associating with each of a plurality of documents a respective initial web crawl interval, wherein the initial web crawl interval is determined based at least in part on a document's importance metric; instructions for retrieving each of the plurality of documents from a respective host at one or more times associated with each document's respective initial web crawl interval; instructions for associating a revised web crawl interval with a respective document of the plurality of documents based on the document's respective initial web crawl interval and any changes to content of the document; instructions for partitioning the plurality of documents into multiple tiers according to their respective web crawl intervals, each tier having a distinct associated range of web crawl intervals, including instructions for storing data for each tier identifying documents in the plurality of documents assigned to that tier in accordance with their respective web crawl intervals; instructions for determining, for each tier of the multiple tiers, a web crawl order for the documents assigned to the tier in accordance with their respective revised web crawl intervals; and instructions for moving a respective document between tiers of the multiple tiers when the respective revised web crawl interval of the respective document is associated with a different tier of the multiple tiers than a previous web crawl interval of the respective document.
23. A non-transitory computer readable storage medium, for scheduling documents to be crawled by a search engine in an appropriate order to reduce visibility of stale content in web searching, storing one or more programs to be executed by a computer system, wherein the computer system includes a search engine program, the one or more programs comprising: instructions for associating with each of a plurality of documents a respective initial web crawl interval, wherein the initial web crawl interval is determined based at least in part on a document's importance metric; instructions for retrieving each of the plurality of documents from a respective host at one or more times associated with each document's respective initial web crawl interval; instructions for associating a revised web crawl interval with a respective document of the plurality of documents based on the document's respective initial web crawl interval and any changes to content of the document; instructions for partitioning the plurality of documents into multiple tiers according to their respective web crawl intervals, each tier having a distinct associated range of web crawl intervals, including instructions for storing data for each tier identifying documents in the plurality of documents assigned to that tier in accordance with their respective web crawl intervals; instructions for determining, for each tier of the multiple tiers, a web crawl order for the documents assigned to the tier in accordance with their respective revised web crawl intervals; and instructions for moving a respective document between tiers of the multiple tiers when the respective revised web crawl interval of the respective document is associated with a different tier of the multiple tiers than a previous web crawl interval of the respective document. 24. The computer readable storage medium of claim 23 , wherein the one or more programs further comprise: instructions for determining for respective documents of the plurality of documents, content update rates of the respective documents, user click rates of the respective documents, and at least one document importance metric of the respective documents; instructions for associating the revised web crawl interval with each document of the plurality of documents based on the document's respective initial web crawl interval, any changes to content of the document, the user click rate of the document, and the at least one document importance metric of the document; and instructions for downloading and recording new copies of at least a subset of the documents in accordance with the determined web crawl order.