1. A computer-implemented method comprising: identifying images associated with a user, wherein the images are identified as at least one of captured by a user device associated with the user, stored on the user device associated with the user, and stored in cloud storage associated with the user; for each of the images: identifying one or more objects in the image; and for each of the one or more objects, determining labels, wherein the labels are based on at least one of metadata and a primary annotation and at least one of the images includes a first person; generating a mapping of each of the labels to a confidence score such that each object in one image is associated with multiple labels and a corresponding confidence score for each of the multiple labels, wherein the confidence score indicates an extent to which a label applies to a corresponding image; determining that the first person is identified as being present in at least a threshold number of the images associated with the user; and responsive to determining that the first person is identified as being present in at least the threshold number of the images associated with the user, interacting with the user to obtain identifying information or a confirmation of an identity of the first person that is used to categorize the first person by providing graphical data that displays the first person and requesting the user to identify the first person.