1. A method performed by a website server system having one or more processors and memory storing one or more programs for execution by the one or more processors, comprising: accessing one or more sources of document information, wherein the one or more sources of document information are associated with a website server; extracting the document information including metadata from the sources; generating a sitemap of a website at the website server, the sitemap including a list of documents and corresponding metadata for each of a plurality of documents in the list of documents based on the document information; storing the sitemap at a location; and transmitting a notification from the website server to a remote computer associated with a web crawler system, the notification including information that identifies the location of the sitemap, the notification functioning as an indication that the sitemap is available for access.
1. A method performed by a website server system having one or more processors and memory storing one or more programs for execution by the one or more processors, comprising: accessing one or more sources of document information, wherein the one or more sources of document information are associated with a website server; extracting the document information including metadata from the sources; generating a sitemap of a website at the website server, the sitemap including a list of documents and corresponding metadata for each of a plurality of documents in the list of documents based on the document information; storing the sitemap at a location; and transmitting a notification from the website server to a remote computer associated with a web crawler system, the notification including information that identifies the location of the sitemap, the notification functioning as an indication that the sitemap is available for access. 2. The method of claim 1 , wherein the document metadata information comprises document modification date information associated with the plurality of documents.