1. A method comprising: obtaining a set of Uniform Resource Locators (URLs) and corresponding content from a targeted website; decomposing each URL into a group of key-value pairs; constructing, by a processor, a tree having a plurality of nodes, each node of the tree representing a group of URLs having a common pattern; identifying one or more pairs of nodes corresponding to duplicate content in which a first node in a pair of nodes corresponds to first content that substantively matches second content corresponding to a second node in the pair of nodes; generating a candidate rule for each of the one or more pairs of nodes, the candidate rule relating a URL of the first node to a URL of the second node; and selecting one or more of the candidate rules as one or more deployable rules.