1. A method, comprising: at a computing system having one or more processors and memory storing one or more programs executed by the one or more processors: setting a respective reuse flag for a corresponding document in a plurality of documents, wherein the respective reuse flag is based on a document importance score associated with the corresponding document, wherein the document importance score associated with a document is based, at least in part, on a query-independent metric of an importance of the document for a search engine; and performing a document crawling operation on the plurality of documents in accordance with the reuse flag for respective documents in the plurality of documents, including reusing a previously downloaded version of a respective document in the plurality of documents instead of downloading a current version of the respective document from a host computer in accordance with a determination that the reuse flag associated with the respective document meets a predefined criterion.
6. The method of claim 1 , further comprising: retrieving, from a first database, at least a subset of contents of a first document in the plurality of documents, the first document corresponding to a document identifier in a first scheduler record, the first scheduler record including a reuse flag for the first document set to a first state; and retrieving, from a second database, at least a subset of contents of a second document in the plurality of documents, the second document corresponding to a document identifier in a second scheduler record, the second scheduler record including a reuse flag for the second document set to a second state; wherein the first and second databases are not the same and wherein the second database stores content from previously crawled documents, including the second document.