1. A computer-implemented method for removing duplicate content items in a result set, the method comprising: identifying a subset of content items of a search index based at least upon content items returned in a result set generated by a search engine for a particular query that was previously submitted to the search engine; for each content item in the subset, determining a similarity value with respect to all other content items in the subset; upon determining that a similarity value of a first content item and a second content item of the subset of content items satisfy a particular threshold value, determining which of the first content item and second content item are less relevant to the particular query; and marking, in the search index, the less relevant to the particular query of the first content item and second content item as a duplicate content item for the particular query; in response to determining that the more relevant of the first content item and the second content item to the particular query is no longer available: determining which of the content items are marked as a duplicate and satisfy the particular threshold value is the most relevant content item to the particular query; and updating the search index by removing the mark as duplicate for the most relevant content item to the particular query; wherein the method is performed by one or more computing devices.
1. A computer-implemented method for removing duplicate content items in a result set, the method comprising: identifying a subset of content items of a search index based at least upon content items returned in a result set generated by a search engine for a particular query that was previously submitted to the search engine; for each content item in the subset, determining a similarity value with respect to all other content items in the subset; upon determining that a similarity value of a first content item and a second content item of the subset of content items satisfy a particular threshold value, determining which of the first content item and second content item are less relevant to the particular query; and marking, in the search index, the less relevant to the particular query of the first content item and second content item as a duplicate content item for the particular query; in response to determining that the more relevant of the first content item and the second content item to the particular query is no longer available: determining which of the content items are marked as a duplicate and satisfy the particular threshold value is the most relevant content item to the particular query; and updating the search index by removing the mark as duplicate for the most relevant content item to the particular query; wherein the method is performed by one or more computing devices. 5. The method of claim 1 , wherein determining a similarity value further comprises performing: generating a digital signature for each content item; and calculating the difference in bits between the digital signature for one content item to the digital signature of another content item.