Difficulties in understanding mutual information concept
What does mutual information (MI) convey?


It indicates that there is a relationship between the two signals- i.e. that they are not independent.  It could be that they are correlated, but the relationship does not have to be that simple.  The higher the mutual information, the stronger the relationship (i.e. the more one of the signals can "tell" you about the other).


  What does this imply and mean in the case of mutual information of error, and in general definition of MI and entropy.


If the error has mutual information with the observation this implies that the receiver did not use all of the information in the observation- i.e. the receiver is not ideal.  If the receiver was ideal it would have used all of the information in the observation, so what was left in the error signal would be independent of the observation.


  What is the interpretation if MI of error is decreasing or getting minimized?


Decreasing MI of error implies that the receiver is doing a better job.  Once the MI is zero it is impossible to do any better.