1. A method, comprising: at a computing system having one or more processors and memory storing one or more programs executed by the one or more processors: setting a respective reuse flag for a corresponding document in a plurality of documents, wherein the respective reuse flag is based on a document importance score associated with the corresponding document, wherein the document importance score associated with a document is based, at least in part, on a query-independent metric of an importance of the document for a search engine; and performing a document crawling operation on the plurality of documents in accordance with the reuse flag for respective documents in the plurality of documents, including reusing a previously downloaded version of a respective document in the plurality of documents instead of downloading a current version of the respective document from a host computer in accordance with a determination that the reuse flag associated with the respective document meets a predefined criterion.
1. A method, comprising: at a computing system having one or more processors and memory storing one or more programs executed by the one or more processors: setting a respective reuse flag for a corresponding document in a plurality of documents, wherein the respective reuse flag is based on a document importance score associated with the corresponding document, wherein the document importance score associated with a document is based, at least in part, on a query-independent metric of an importance of the document for a search engine; and performing a document crawling operation on the plurality of documents in accordance with the reuse flag for respective documents in the plurality of documents, including reusing a previously downloaded version of a respective document in the plurality of documents instead of downloading a current version of the respective document from a host computer in accordance with a determination that the reuse flag associated with the respective document meets a predefined criterion. 2. The method of claim 1 , the method further comprising: retrieving a plurality of records corresponding to prior scheduled crawls of a respective document in the plurality of documents, each record comprising a set of entries, the set of entries comprising a document identifier, a crawl time, a crawl type, and a content summary, the plurality of records including: a first record, the first record comprising a first document identifier, a first document importance score, a first crawl time, a first crawl type, and a first content summary, and a second record, the second record comprising a second document identifier, a second document importance score, a second crawl time, a second crawl type, and a second content summary; wherein the first document identifier and the second document identifier correspond to the respective document; and wherein setting the reuse flag is further based on a comparison of the first content summary with the second content summary.