18. The method of claim 12 , wherein the user-specified web crawl configuration further comprises user-specified throttling rules, the throttling rules comprising at least a plurality of a user-specified page pause time, a user-specified number of page crawler threads, and a user-specified maximum number of threads allocated per crawled domain; and wherein the default crawl behavior further comprises pausing at least the user-specified page pause time before fetching the next page resource, spawning up to the user-specified number of page crawler threads to perform the crawl, and for any given crawled domain, spawning up to the user-specified maximum number of threads allocated per crawled domain.
12. A method for performing a user-configured web crawl, comprising: receiving a user-specified web crawl configuration for crawling a web to inspect text of resources accessible on the web, each resource identified by a unique uniform resource locator (URL), the user-specified web crawl configuration comprising user-specified crawling rules specifying crawling behavior, maintaining a queue of unprocessed URLs of resources; retrieving from the queue a next unprocessed URL; retrieving the crawling rules for the retrieved URL, the crawling rules specifying crawling behavior for at least one each of a general crawl behavior, one or more page-specific crawl behavior to be applied during a crawl to one or more identified pages, and one or more element-specific crawl behavior to be applied during a crawl to one or more identified page elements; if allowed by the specified general crawl behavior and page-specific crawl behavior of the retrieved crawling rules, fetching the page resource addressed by the retrieved URL; processing the fetched page resource on an element-by-element basis, by parsing a next page element from the fetched page resource, and if an element-specific behavior is specified in the retrieved crawling rules for the parsed next page element then operating with regard to the element according to the specified element-specific behavior, and otherwise if a page-specific behavior is specified for the fetched page resource then operating with regard to the element according to the specified page-specific behavior, and otherwise if a global crawl behavior is specified then operating with regard to the element according to the specified global crawl behavior, and otherwise operating with regard to the element according to a default crawl behavior; and repeating the processing until either all elements have been parsed and processed in accordance with the retrieved crawling rules, and wherein the default crawl behavior comprises extracting URLs of outbound links and adding the extracted URLs to the queue, storing text to a resource repository for further analysis; and repeating the steps of retrieving the next unprocessed URL through this repeating step until the queue is empty of unprocessed URLs; and wherein the retrieved crawling rules define the crawling behavior in terms of whether to record the resource text, whether to follow hyperlinks of the resource, whether to record text description in metadata associated with the page, whether to record keywords identified in the metadata associated with the page, whether to override a page title element, whether to exclude one or more identified resource pages, whether to exclude one or more identified domains.