Building ML to finding the closest matching image based on users drawing I am trying to build a ML Agent to find the closest matching image from a given set. The user will draw something and the agent should list the closest matching images. 

Very similar to these examples


https://sketchx.eecs.qmul.ac.uk/
Emoji search in Android keyboard


One unique problem I've is, each image will represent a category. Imagine we have product images and user will draw something and we have to find the products close to the drawing. So category in my case will be product Id. 



I would like to evaluate the approach before trying out. 

There are lot of examples to classify images, however if I use the item identifier instead of category it should work. But am trying to find the best approach for this problem.
There is a good project you can research to find information on this specific task, it is Google quickdraw. It has 50 million drawings across 345 categories and it was used in Google autoDraw. AutoDraw guesses what you’re trying to draw.

You can use quickdraw dataset to train a neural network (Sequence-to-Sequence Variational Autoencoder) in order to get draws from your Sketches. for more info on this read this papper

Classified Sketches into categories is a difficult task because Sketches are a high-level representation that does not always convey enough information to distinguish between different categories.

The the best approach will depend on your specific needs