Given that two images are similar, how do I find and classify the difference between them? We have used a derivative of the P-hash algorithm to determine similar images when the user selects an image.

This allows the customer to check out very similar looking photos together. It has also helped us cluster pictures.

When the user selects an image, we would like to show him/her how each of the similar images is different from the selected image, by highlighting the difference, and by classifying the difference.

(eg: this is the hard cover version of the book; The font is different on this cover; original dress in black: same dress in blue, green and yellow ) 

I'm unsure what kind of annotated training data would be useful for Tensorflow. Could someone please point us in the right direction?

My problem is that we have successfully been able to classify single pictures and even do a bit of object detection by drawing bounded boxes, but we are unable to do pairwise difference identification.
In order for you to do pairwise difference, you need to do the following:


Generate Ground truth data by drawing bounding boxes on elements you need comparison for and label them i.e if you want to compare text on the hard cover, you need to mark the text region and add a label to it like font type.
Once you have ground truth data, you can then use object detection networks like different variants of Yolo to identify multi class objects in each document.


This should provide you a starting point to solving the problem at hand.