 irreversibility. It was R.V. Hartley (1928), of the Bell Telephone laboratories who It should be noted that an event that occurs with high probability gave the logarithmic definition of information from a communica- conveys less information than a low probability event. tions viewpoint, and it was C. Shannon who further refined the ideas Consider the expression given above for the mutual information, in 1949. and since we can write Equation 4.22 then the mutual information 4/6 Information theory obeys the symmetry relation given in Equation 2.23 by virtue of Bayes' theorem. (4.31) (4.22) (4.32) = m(x) dx (4.23) Discrete memoryless channels The average value of the mutual information defined above is The discrete memoryless channel serves as a statistical model with given by Equation 4.24 an input X and an output Y which is a noisy version of X, and both X and Y are random variables. The channel is said to be 'discrete' when both the alphabets from which X and Y are selected have finite size, not necessarily the same size, and the channel is said to be 'memoryless' when the current i=1 j=1 P(x,yj) (4.24) output symbol depends only on the current input symbol and not on any previous ones (c.f. Markov models and ARMA processes etc.). In terms of the elements of the input and output alphabets, one can define a set of transition probabilities for all j and k, as in Equation Likewise, the average value of the self information, H(X) is 4.33 and this is just the conditional probability that the channel defined by Equation 4.25. output is Y y y = given that the channel input is X (4.33) Due to transmission errors, when k =j, the transition probability (4.25) represents the conditional probability of correct reception, and when k j j the conditional probability of error. Just as for Markov processes, one can define a transition prob- For any given communication source transmitting an 'alphabet' of ability matrix as follows. symbols, the entropy of the source is a maximum when the output symbols are equally probable, as is