1. A computer-implemented method for removing duplicate content items in a result set, the method comprising: generating a subset of content items of a search index based at least upon content items returned in a result set from a particular query; for each content item in the subset, determining a similarity value with respect to all other content items in the subset; upon determining that a similarity value of a first content item and a second content item of the subset of content items is below a particular threshold value, determining which of the first content item and second content item are less relevant to the particular query; marking the less relevant of the first content item and second content item as a duplicate content item for the particular query; and updating the search index to reflect the marked duplicate content item for the particular query, wherein the method is performed by one or more special-purpose computing devices.