5. The computer program product of claim 4 , wherein the computer readable program when executed on a computer causes the computer to: attempt to fetch a document by crawling the web crawl space; determine that the document has been moved; and perform redirect processing by: identifying a new seed name that represents a new location of the document; and generating one or more additional allow rules based on the new seed name to enlarge the web crawl space.
5. The computer program product of claim 4 , wherein the computer readable program when executed on a computer causes the computer to: attempt to fetch a document by crawling the web crawl space; determine that the document has been moved; and perform redirect processing by: identifying a new seed name that represents a new location of the document; and generating one or more additional allow rules based on the new seed name to enlarge the web crawl space. 6. The computer program product of claim 5 , wherein the computer readable program when executed on a computer causes the computer to: add the one or more additional allow rules to the set of allow rules; and crawl the web crawl space defined by the set of allow rules.