12. A system comprising: a data processing apparatus; and computer storage medium encoded instructions that when executed by a data processing apparatus cause the data processing apparatus to perform operations comprising: receiving data identifying images determined to be responsive to a query, each image having an associated search score that is a measure of responsiveness to the query, and the images ranked according to a first order based on the search scores, and in response to receiving the data identifying the images: for each image in a first set of the images, detecting a face depicted in the image and generating a face template from the depicted face by processing the image using a facial template generation process that identifies machine-identifiable properties of structures of a face, and where each face template is a temporary face template that is not persisted to memory; clustering the images in the first set of images into a plurality of clusters based on similarity of the facial templates to each other, each cluster including at least one of the images, and each image belonging to only one cluster; determining a quantity of images belonging to each cluster; selecting the cluster with the highest quantity relative to the quantity of other clusters as an inlier cluster; for each image in the first set of images, determining an inlier score for the image that is a measure of similarity of the facial template of the image to the facial templates of the images that belong to the inlier cluster; and re-ranking the images determined to be responsive to a query based on the inlier scores of the set of images; wherein determining the inlier score comprises, for images that belong to the inlier cluster that are determined to be duplicate images of each other, using only one of the images from the images determined to be duplicate images of each other to determine the inlier score.
12. A system comprising: a data processing apparatus; and computer storage medium encoded instructions that when executed by a data processing apparatus cause the data processing apparatus to perform operations comprising: receiving data identifying images determined to be responsive to a query, each image having an associated search score that is a measure of responsiveness to the query, and the images ranked according to a first order based on the search scores, and in response to receiving the data identifying the images: for each image in a first set of the images, detecting a face depicted in the image and generating a face template from the depicted face by processing the image using a facial template generation process that identifies machine-identifiable properties of structures of a face, and where each face template is a temporary face template that is not persisted to memory; clustering the images in the first set of images into a plurality of clusters based on similarity of the facial templates to each other, each cluster including at least one of the images, and each image belonging to only one cluster; determining a quantity of images belonging to each cluster; selecting the cluster with the highest quantity relative to the quantity of other clusters as an inlier cluster; for each image in the first set of images, determining an inlier score for the image that is a measure of similarity of the facial template of the image to the facial templates of the images that belong to the inlier cluster; and re-ranking the images determined to be responsive to a query based on the inlier scores of the set of images; wherein determining the inlier score comprises, for images that belong to the inlier cluster that are determined to be duplicate images of each other, using only one of the images from the images determined to be duplicate images of each other to determine the inlier score. 18. The system of claim 12 , wherein determining the inlier score comprises determining, for each image that belongs to a cluster with a quantity less than or equal to a minimum quantity threshold, a minimum inlier score for the image.