Siamese vs matching network for correct image category matching
A standard approach for this kind of problem is to take the representations of the images computed by some pre-trained neural network (e.g. imagenet). As &quot;representation&quot; of the image we usually take the vector after flattening the output of the last convolutional layer block.
If your images are not of general domain, but are of a very specific domain, and you have enough data, you may want to train an autoencoder on them instead of using a pre-trained model.
You now compute and store the representations of all the images you want to compare against. Once you have the representations, for each image you want to match, you simply compute the closes image in the representation space. There are libraries like Facebook's faiss that are very fast computing binary similarities over a large number of vectors.
This kind of approach is cheap in terms of compute, it is scalable and it is widely used.