19. The method of claim 12 , wherein the crawling rules further specify crawling behavior for one or more domain-specific crawl behavior to be applied during a crawl to one or more identified domains; fetching the page resource addressed by the retrieved URL and assigning a resource identifier for the fetched page resource only if allowed by the specified general crawl behavior, the domain-specific crawl behavior, and page-specific crawl behavior of the retrieved crawling rules; processing the fetched page resource on an element-by-element basis, by parsing a next page element from the fetched page resource, and if an element-specific behavior is specified in the retrieved crawling rules for the parsed next page element then operating with regard to the element according to the specified element-specific behavior, and otherwise if a page-specific behavior is specified for the fetched page resource then operating with regard to the element according to the specified page-specific behavior, and otherwise if a domain-specific behavior is specified for the fetched page resource then operating with regard to the element according to the specified domain-specific behavior, and otherwise if a global crawl behavior is specified then operating with regard to the element according to the specified global crawl behavior, and otherwise operating with regard to the element according to a default crawl behavior.
20. The method of claim 19 , wherein the user-specified web crawl configuration further comprises user-specified throttling rules, the throttling rules comprising at least a plurality of a user-specified page pause, time, a user-specified number of page crawler threads, and a user-specified maximum number of threads allocated per crawled domain; and wherein the default crawl behavior further comprises pausing at least the user-specified page pause time before fetching the next page resource, spawning up to the user-specified number of page crawler threads to perform the crawl, and for any given crawled domain, spawning up to the user-specified maximum number of threads allocated per crawled domain.