1. A method for determining similarity of objects depicted in images when the images pertain to different modalities, the method comprising: obtaining images that depict the objects and that pertain to the different modalities; applying an embedding function to each of the images, the embedding function selected from a set of two or more embedding functions, each of the embedding functions corresponding to a modality of the different modalities, the selected embedding function corresponding to the modality of the image to which it is applied, so as to map that image to a representation in a representation space such that when the images are mapped to the representation space, a distance between the representations of the images is indicative of a similarity of their depicted objects, the set of embedding functions being by: obtaining a training set of images pertaining to the different modalities, the images depicting objects, and measures dissimilarity between the objects, calculating a value of a discrepancy criterion for each candidate set of embedding functions from a set of candidate sets of embedding functions, so as to measure the discrepancy between the measures of dissimilarity between the objects in the training set and the distance between representations of the images corresponding to these objects in a representation space created by applying that candidate set of embedding functions to the images of the training set, and selecting as the set of embedding functions the candidate set of embedding functions corresponding to an optimal value of the discrepancy criterion; and determining the similarity of the depicted objects based on the location of the corresponding representations in the representation space.