1. A configurable web crawler system, comprising: computer-readable media storing a queue of unprocessed URLs of resources; one or more processing units configured to receive a user-specified web crawl configuration for crawling a web to inspect text of resources accessible on the web, each resource identified by a unique uniform resource locator (URL), the user-specified web crawl configuration comprising user-specified crawling rules, the crawling rules specifying crawling behavior for at least one each of a general crawl behavior, one or more page-specific crawl behavior to be applied during a crawl to one or more identified pages, and one or more element-specific crawl behavior to be applied during a crawl to one or more identified page elements, wherein the received crawling rules define the crawling behavior in terms of whether to record the resource text, whether to follow hyperlinks of the resource, whether to record text description in metadata associated with the page, whether to record keywords identified in the metadata associated with the page, whether to override a page title element, whether to exclude one or more identified resource pages, whether to exclude one or more identified domains; the one or more processing units configured to retrieve from the queue a next unprocessed URL and, if allowed by the specified general crawl behavior and page-specific crawl behavior of the retrieved crawling rules, to fetch the page resource addressed by the retrieved URL and to process the fetched page resource on an element-by-element basis by parsing a next page element from the fetched page resource, and if an element-specific behavior is specified in the retrieved crawling rules for the parsed next page element then operating with regard to the element according to the specified element-specific behavior, and otherwise if a page-specific behavior is specified for the fetched page resource then operating with regard to the element according to the specified page-specific behavior, and otherwise if a global crawl behavior is specified then operating with regard to the element according to the specified global crawl behavior, and otherwise operating with regard to the element according to a default crawl behavior; and repeating the processing until either all elements have been parsed and processed in accordance with the retrieved crawling rules, and wherein the default crawl behavior comprises extracting URLs of outbound links and adding the extracted URLs to the queue, storing text to a resource repository for further analysis, the one or more processing units repeating the processing on available unprocessed URLs in the queue.
2. The configurable web crawler of claim 1 , wherein the one or more processing units are further configured to receive a specified crawl depth representing the maximum number of URL links deep from the starter seed URL to crawl, and wherein the default crawl behavior comprises extracting URLs of outbound links and adding the extracted URLs to the queue so long as the specified crawl depth has not been reached.