5. The method of claim 1 , wherein scheduling documents for downloading comprises generating a list of document identifiers that identify the scheduled documents.
1. A method of scheduling documents for crawling, performed on a computer system having one or more processors and memory storing one or more programs for execution by the one or more processors, the method comprising: receiving from a website a notification that includes a sitemap URL corresponding to a sitemap for the website; in response to the notification: accessing the sitemap at the sitemap URL; and retrieving from the sitemap document location information and metadata for a plurality of documents associated with the website; scheduling for downloading documents, from among the plurality of documents, based at least in part on the metadata retrieved from the sitemap; and downloading at least a subset of the documents scheduled for downloading.