1. A method for determining similarity of objects depicted in images when the images pertain to different modalities, the method comprising: obtaining images that depict the objects and that pertain to the different modalities; applying an embedding function to each of the images, the embedding function selected from a set of two or more embedding functions, each of the embedding functions corresponding to a modality of the different modalities, the selected embedding function corresponding to the modality of the image to which it is applied, so as to map that image to a representation in a representation space such that when the images are mapped to the representation space, a distance between the representations of the images is indicative of a similarity of their depicted objects; and determining the similarity of the depicted objects based on the location of the corresponding representations in the representation space.