Confusion about shannon's entropy of the grayscale image after lossy compression I have a .pgm format 480 * 640 gray scale image named 'columns.pgm'. Using PCA (principal component analysis), by preserving 40 principal components I compressed original image with 1218 KB to 301 KB using a Matlab function which I wrote. Now about the entropy. The entropy of the original file was 6.93 and of the compressed file was 7.17. Actually the entropy of the compressed version must be lesser, right? I am confused. Can someone give me a intuition based explanation. And are the cases similar for lossy and lossless compression?
Shannon's entropy only makes sense when you define it over a probability distribution. You're using a probability distribution where each pixel is sampled independently from the histogram of pixel intensities. Clearly, real pictures don't come from such a distribution. If they did, they would all look like white noise.

Also note that individual elements don't have entropy. Only distributions have entropy. So if you have two images it doesn't make sense to compare their entropy. It only makes sense to compare the entropy of the random distributions that you sampled the images from.