21. A non-transitory computer readable storage medium and one or more computer programs embedded therein, the computer programs comprising instructions, which when executed by a computer system, cause the computer system to: access one or more sources of document information, wherein the sources are associated with a website server; extract the document information including metadata from the sources; generate a sitemap of a website at the website server, the sitemap including a list of documents and corresponding metadata for each of a plurality of documents in the list of documents based on the document information; store the sitemap at a location; and transmit a notification from the website server to a remote computer associated with a web crawler system, the notification including information that identifies the location of the sitemap, the notification functioning as an indication that the sitemap is available for access.
21. A non-transitory computer readable storage medium and one or more computer programs embedded therein, the computer programs comprising instructions, which when executed by a computer system, cause the computer system to: access one or more sources of document information, wherein the sources are associated with a website server; extract the document information including metadata from the sources; generate a sitemap of a website at the website server, the sitemap including a list of documents and corresponding metadata for each of a plurality of documents in the list of documents based on the document information; store the sitemap at a location; and transmit a notification from the website server to a remote computer associated with a web crawler system, the notification including information that identifies the location of the sitemap, the notification functioning as an indication that the sitemap is available for access. 25. The computer readable storage medium of claim 21 , wherein the document metadata information comprises document update rate information associated with the plurality of documents.