Measure similarity between quantized grayscale images I want to measure the similarity between quantized grayscale images (8 levels) that mostly depict organic patterns found in nature, say pigmentation patterns in animals (zebra pattern, leopard spots,..). I cannot use some kind of trained model, and I need a continuous similarity value. Feature scale tolerance is not important, but differences in rotation and density of the features could be large. Ideally two different flat textures of zebra patterns should have a similarity of 1.0 and say zebra-tiger a similarity of 0.7. I understand how impossible that sounds, but I hope someone could point me to relevant current research or methods somewhat suitable to what I'm looking for.

EDIT: More info about the desired usage.

Examples of images undergone grayscale conversion and quantization:
(512x512 pixels, 8 gray levels). I do not own these images.




These are then used as templates for a procedural pattern generation technique I am developing. Images are randomly generated, generation settings that produce high similarity are kept to be further tweaked. The procedurally generated images could be anything from random pixels to intricate shapes, slowly converging at the image of highest possible similarity.
This question is very broad, yet interesting. Your way to go are co-occurence matrices.

Co-occurence matrices can be calculated for each image and provide further indicators which specifically allow identification of different textures. 

Note that there does not exist a single co-occurence matrix for an image, but several ones, since the co-occurence matrix is defined for specific directions of occurence. Further, you can vary the granularity of the co-occurence matrix. For each co-occurence matrix you can derive several features which enable you to distinguish the textures. 

This example is very simple, yet should enable you to understand: 

Image you have black-white picture of only a single line of pixels such as:

[0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0]


Now there are there are four possible co-occurences of pixel neighbours: 

1) [0,0]
2) [0,1]
3) [1,0]
4) [1,1]


For each these we will now count the times that you can them within the strings: 

1) [0,0] ---&gt; 4
2) [0,1] ---&gt; 3
3) [1,0] ---&gt; 3
4) [1,1] ---&gt; 0 


This will give us our co-occurence matrix:

[[4,3],[3,0]] which is specific for each pattern. 


Based on this co-occurence matrix, you are able to calculate features, such as Contrast, Correlation, Energy, Homogenity and many more.

Matlab provides you with tools to get the co-occurence matrix. Further, wikipedia provides an introduction.  

Once you implemented this and encountered your first problems, I am looking forward to your next question. Good luck.