1. A method of processing a candidate URL, comprising: storing information for a plurality of equivalence classes, a respective equivalence class of the plurality of equivalence classes including a plurality of URLs, wherein all the URLs in the respective equivalence class have the same hostname; identifying the candidate URL, wherein the candidate URL is a text string that identifies exactly one document, and wherein the candidate URL contains a set of URL parameters, each URL parameter comprising a text string; identifying an equivalence rule, if any, for the candidate URL, and applying the identified equivalence rule to identify, from among the set of URL parameters, any URL parameters of the candidate URL that are content-irrelevant parameters; updating the candidate URL by removing from it each identified content-irrelevant parameter; checking if there is an equivalence class, of the plurality of equivalence classes, associated with the updated candidate URL, the equivalence class including a single representative URL, wherein all the URLs in a respective equivalence class of the plurality of equivalence classes have the same hostname and reference documents having substantially the same content; if an equivalence class associated with the updated candidate URL is found, applying a first set of predefined operations to only the representative URL of the equivalence class; and applying a second set of predefined operations to the candidate URL if no equivalence class is found; wherein the first set of predefined operations is distinct from the second set of predefined operations.