1. A method for defining a web crawl space to be crawled, comprising: at a computer including a processor: receiving a seed list including one or more seed names from a user, wherein each seed name represents a website with a two part name or a three part name, wherein the last part of the seed name is a top level domain name; and in response to receiving the seed list, generating a web crawl space for the received seed list by generating one or more new allow rules that determine what part of a webspace is to be crawled by: for each of the one or more seed names, generating a new allow rule by appending a slash and a wildcard character at an end of the seed name to allow crawling of content of the website; determining whether the seed name has less than three parts or at least three parts; in response to determining that the seed name has less than three parts, updating the allow rule for the seed name by appending a wildcard character and a dot before a beginning of the seed name to allow crawling of content of one or more subdomains; and in response to determining that the seed name has at least three parts, updating the allow rule for the seed name by replacing a first portion of the seed name before a dot with a wildcard character.
1. A method for defining a web crawl space to be crawled, comprising: at a computer including a processor: receiving a seed list including one or more seed names from a user, wherein each seed name represents a website with a two part name or a three part name, wherein the last part of the seed name is a top level domain name; and in response to receiving the seed list, generating a web crawl space for the received seed list by generating one or more new allow rules that determine what part of a webspace is to be crawled by: for each of the one or more seed names, generating a new allow rule by appending a slash and a wildcard character at an end of the seed name to allow crawling of content of the website; determining whether the seed name has less than three parts or at least three parts; in response to determining that the seed name has less than three parts, updating the allow rule for the seed name by appending a wildcard character and a dot before a beginning of the seed name to allow crawling of content of one or more subdomains; and in response to determining that the seed name has at least three parts, updating the allow rule for the seed name by replacing a first portion of the seed name before a dot with a wildcard character. 2. The method of claim 1 , further comprising: attempting to fetch a document by crawling the web crawl space; determining that the document has been moved; and performing redirect processing by: identifying a new seed name that represents a new location of the document; and generating one or more additional allow rules on the new seed name to enlarge the web crawl space.