Opinion about this aproach on Machine Learning
This is a fine idea and pretty standard. However, your problem will be the augmentation of the data. If you render the image from many viewpoints/rotations etc. and just use a plain background, the network would include that information into the training stage e.g. Like a shape cue. So, you have to use different backgrounds. This is of course scene specific. However, you might be able to narrow down your scenes so that you could randomly generate/perturb different background structures. 

Also note that, you might be able to scale up to thousands of classes, but getting to millions could become a headache, since training this network would be very tough.

You might as well go with standard indexing / hashing approaches, which already perform pretty well.