Issues related to entropy
I will try to tackle questions 1 and 4.


  1) Entropy- It is desired that the entropy of the system be maximized. Maximizing entropy means no symbol is better than the others or we do not know what the next symbol / outcome would be. However, the formula states a negative sign before the summation of the probability logarithms. Thus, it means we are maximizing a negative value!!


No, the values of the logarithms themselves are negative, so the negative sign makes them positive.  All probabilities are a real number from 0 to 1, inclusive.  The log of 1 is zero, and the log of anything less than 1 is negative.  This may seem problematic since the log of 0 is $-\infty$, but we are really trying to maximize the expected value of these logs, so when we multiply by the probability itself the entire value approaches 0, not $\infty$.  Entropy peaks when the probability is $1/2$.


  So,why do we want to maximise entropy since it would mean that we are maximizing the uncertainty of the next symbol whereas we want to be certain about waht the next occurence of the symbol would be.


No, in communicating information we absolutely do NOT want to be certain about what the next symbol will be.  If we are certain, what information is gained by receiving it?  None.  It is only through uncertainty about what the transmitter will send that we can receive any information.


  4) What information does mutual information between two vectors convey?


When there is mutual information between two vectors knowing something about one tells you something about the other.  Mathematically this equates to the following- knowledge of one vector affects the probabilities of the other vector.  If they were independent, this would not be the case.

An example of mutual information is digital walkie-talkies.  One vector is the bit stream that the first walkie-talkie sends.  The second vector is the signal that the second walkie-talkie receives.  The two are obviously related, but due to noise and unknown channel conditions the second walkie-talkie cannot know for certain what the first one sent.  It can make some really good guesses based on the signal, but it cannot know for sure.