9. The configurable web crawler of claim 1 , wherein the default crawl behavior further comprises determining whether the retrieved page includes any images, when an image identification is detected, retrieving and storing the image associated with the image identification to a resource repository for further analysis, determining whether the image identification has associated therewith dimensional information for the image, and if the fetched page does not include dimensional information for the image, determining the dimensional image from the image itself, and adding the dimensional information and other image attributes to an image metadata database associated with the page resource in the resource repository.
9. The configurable web crawler of claim 1 , wherein the default crawl behavior further comprises determining whether the retrieved page includes any images, when an image identification is detected, retrieving and storing the image associated with the image identification to a resource repository for further analysis, determining whether the image identification has associated therewith dimensional information for the image, and if the fetched page does not include dimensional information for the image, determining the dimensional image from the image itself, and adding the dimensional information and other image attributes to an image metadata database associated with the page resource in the resource repository. 10. The configurable web crawler of claim 9 , wherein the user-specified web crawl configuration further comprises user-specified throttling rules, the throttling rules comprising at least a plurality of a user-specified page pause time, a user-specified number of page crawler threads, a user- specified number of image crawler threads, and a user-specified maximum number of threads allocated per crawled domain; and wherein the default crawl behavior further comprises pausing at least the user-specified page pause time before fetching the next page resource, spawning up to the user-specified number of page crawler threads to perform the crawl, spawning up to the user-specified number of image crawler threads to determine image dimensional information for page images, and for any given crawled domain, spawning up to the user-specified maximum number of threads allocated per crawled domain.