9. The method of claim 1 , wherein generating the sitemap comprises generating a list of documents modified after a particular time.
1. A method performed by a website server system having one or more processors and memory storing one or more programs for execution by the one or more processors, comprising: accessing one or more sources of document information, wherein the one or more sources of document information are associated with a website server; extracting the document information including metadata from the sources; generating a sitemap of a website at the website server, the sitemap including a list of documents and corresponding metadata for each of a plurality of documents in the list of documents based on the document information; storing the sitemap at a location; and transmitting a notification from the website server to a remote computer associated with a web crawler system, the notification including information that identifies the location of the sitemap, the notification functioning as an indication that the sitemap is available for access.