Measuring similarity between two images
I don't quite know what you mean by different viewpoints, however, there are a lot of different similarity metrics you could use to compare images. You could try:


extracting features and comparing the features Like Scale-invariant feature transform (SIFT) features
you could try dimensionality reduction and comparing that representation, like taking the discrete cosine transform (DCT). 
I know you said you tried mean squared error but you could also try comparing the structural similarity (SSIM).


Edit:

Now that I better understand your question, I believe I found a resource that could be of assistance. Here is a very aptly named paper that is titled "A Novel Algorithm for View and Illumination
Invariant Image Matching".