16. A method for determining a rule applicable to uniform resource locators (URLs), comprising: grouping web resources on a web server according to substantially identical content, wherein each group of substantially identical web resources is referred to as an equivalence class; analyzing URLs addressing all substantially identical web resources in an equivalence class; constructing a per equivalence class URL normalization rule applicable to the URLs corresponding to all substantially identical web resources in the equivalence class; analyzing the per equivalence class URL normalization rule compared to at least one other per equivalence class URL normalization rule for at least one different equivalence class to determine a trans-equivalence class URL normalization rule; and applying the trans-equivalence class URL normalization rule to additional web resources to predict that different URLs reference substantially identical web resources, thereby avoiding a plurality of references to or downloads of substantially identical web resources.
18. The method of claim 16 , further comprising validating the per equivalence class URL normalization rule.