26. The method of claim 21 , further comprising using the rewritten plurality of tokens as inputs in determining whether or not to retrieve one or more of the pages referenced by the plurality of related URLs.
26. The method of claim 21 , further comprising using the rewritten plurality of tokens as inputs in determining whether or not to retrieve one or more of the pages referenced by the plurality of related URLs. 28. The method of claim 26 , wherein the determining operates to avoid retrieval of web pages having duplicative content referenced by multiple different URLs.