1. A configurable web crawler system, comprising: one or more processors configured to allow a user to configure a web crawl according to one or more of thread throttling rules, domain restriction rules, and crawling rules; a crawling function which receives a starter seed uniform resource locator, and crawls a web, beginning at the starter seed uniform locator, according to the configured web crawl.