15. The system of claim 12 , wherein the document metadata information comprises document priority information associated with the plurality of documents, wherein the document priority information indicates a crawling priority.
12. A system, comprising: one or more processors and memory, the memory comprising one or more sources of document information; and one or more modules including instructions to: access the sources of document information, wherein the sources are associated with a website server; extract the document information including metadata from the sources; generate a sitemap of a website at the website server, the sitemap including a list of documents and corresponding metadata for each of a plurality of documents in the list of documents based on the document information; store the sitemap at a location; and transmit a notification from the website server to a remote computer associated with a web crawler system, the notification including information that identifies the location of the sitemap, the notification functioning as an indication that the sitemap is available for access.