26. The method of claim 21 , further comprising using the rewritten plurality of tokens as inputs in determining whether or not to retrieve one or more of the pages referenced by the plurality of related URLs.
26. The method of claim 21 , further comprising using the rewritten plurality of tokens as inputs in determining whether or not to retrieve one or more of the pages referenced by the plurality of related URLs. 27. The method of claim 26 , wherein the retrieval occurs for indexing content of a retrieved page in a crawling process.