 clear since if the symbols are p(y,1x) p(y1/xx) p(yk-1/xo) equally probable, Equations 4.26 and 4.27 and in general Equation p(y)/x1) p(y,x1)p(ykx1) 4.28 are obtained. (4.26) (4.27) This is a J by K matrix called the channel matrix. The sum of elements along any row is given by Equation 4.34 for all j. H(X) s l log n (4.28) (4.34) It is very straightforward to show that Equation 4.29 follows with equality if and only if X and Y are independent, and that Equation 4.30 is true where H (X|Y) is a conditional uncertainty, etc. If the inputs to a discrete memoryless channel are selected accord- ing to the probability distribution p(x), j=0,1 J-1, then the event H(X,Y) V H(X)+H(Y) (4.29) X = x, occurs with probability given by Equation 4.35. X,Y)H(X)+H(YX) (4.35) =H(Y)+H(XY) (4.30) The above definitions of mutual information, self information and The joint probability distribution of the random variables X and Y entropy for discrete random variables may be easily extended to the is given by Equation 4.36 and the marginal distribution of the output case of continuous random variables, and if the joint probability variable Y is obtained by integrating out, or averaging out, the density functions and the marginal probability density function are dependence on X, for (j=0,1,..., J-1), as in Equation 4.37 for k = p(x,y), p(x) and p(y) respectively, then the average mutual informa- 0,1,...K-1. tion between X and Y (or the cross entropy) is given by Equation 4.31 and the entropy of X in the continuous case by Equation 4.32 where m(x) is an appropriate measure function (Jaynes, 1989). Discrete memoryless channels 4/7 P(Y=y,|X+x))(X=x) This may be compared to the method employed for the maximum entropy analysis of images and inverse problems (Jaynes, 1989; (4.36) Skilling, 1988). The channel capacity is an extremely important quantity, since it is possible to transmit information through a channel at any rate less (4.37) than the channel capacity with an arbitrary small probability of error; completely reliable transmission is