Measure similarity between quantized grayscale images
This question is very broad, yet interesting. Your way to go are co-occurence matrices.

Co-occurence matrices can be calculated for each image and provide further indicators which specifically allow identification of different textures. 

Note that there does not exist a single co-occurence matrix for an image, but several ones, since the co-occurence matrix is defined for specific directions of occurence. Further, you can vary the granularity of the co-occurence matrix. For each co-occurence matrix you can derive several features which enable you to distinguish the textures. 

This example is very simple, yet should enable you to understand: 

Image you have black-white picture of only a single line of pixels such as:

[0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0]


Now there are there are four possible co-occurences of pixel neighbours: 

1) [0,0]
2) [0,1]
3) [1,0]
4) [1,1]


For each these we will now count the times that you can them within the strings: 

1) [0,0] ---&gt; 4
2) [0,1] ---&gt; 3
3) [1,0] ---&gt; 3
4) [1,1] ---&gt; 0 


This will give us our co-occurence matrix:

[[4,3],[3,0]] which is specific for each pattern. 


Based on this co-occurence matrix, you are able to calculate features, such as Contrast, Correlation, Energy, Homogenity and many more.

Matlab provides you with tools to get the co-occurence matrix. Further, wikipedia provides an introduction.  

Once you implemented this and encountered your first problems, I am looking forward to your next question. Good luck.