1. A method for performing a visual search with a visual search device, the method comprising: computing, with the visual search device, a distance between a query feature descriptor provided by way of a visual search query and each of a plurality of reference feature descriptors, wherein the visual search query initiates the visual search; determining, with the visual search device, a first group of one or more of the computed distances and a second group of the computed distances in accordance with a clustering algorithm, wherein the first group of the computed distances includes those of the computed distances that indicate the associated one of the plurality of reference feature descriptors are near to the query feature descriptor relative to the those of the computed distances determined to be in the second group of computed distances, and wherein the second group of the computed distances includes those of the computed distances that indicate the associated one of the plurality of reference feature descriptors are far from the query feature descriptor relative to the those of the computed distances determined to be in the first group of the one or more of the computed distances; determining, with the visual search device, whether the query feature descriptor matches one of the plurality of reference feature descriptors associated with a smallest one of the computed distances based on the determined first group of the computed distances and the second group of the computed distances; determining a unique group of reference images to include reference images associated with those of the plurality of reference feature descriptors from which the computed distances determined to be in the first group were computed to a corresponding group of reference feature descriptors such that the set of reference images does not include duplicate reference images; assigning a vote to each of the reference images determined to be in the unique group of reference images; ordering the reference images based on the assigned votes; and returning the ordered reference images in response to the visual search query.