 viewpoint came about due to a paradox conceived of by Maxwell concerning two chambers separated by a common parti- H(XY)=H(X,Y)-H(Y) (4.17) tion which could be removed to allow objects (gas molecules) to move freely between the two chambers. Normal experience tells us It should also be mentioned that the statistician R.A. Fisher in that if one of the chambers initially has gas present and the partition 1922 defined the information content of a statistical distribution, is withdrawn, then gas will rapidly diffuse to fill both chambers. The and this Fisher information still plays a fundamental role in statisti- reverse never occurs. cal estimation and parameter estimation in general, and allows one Maxwell proposed his famous 'demon' whose purpose was to to put bounds on the accuracy with which one can estimate par- open a small frictionless door present in the partition, and when a ameters. particle, due to thermal motion heads towards the door, the demon opens the door to let the particle through the partition, in one direction only, and as time passes, one chamber gets filled up with 4.4 Mutual and self information particles at the expense of the other, thus seeming to violate the second law of thermodynamics. This paradox was explained by Let X and Y be two discrete variables with possible outcomes Xi, i Szilard, who pointed out that in order for the demon to perform, it = 1,2,...,n, and yi, i = 1,2, m, respectively. If an outcome = yj is had to be very well informed about the position and velocity of the observed, and we wish to quantitatively determine the amount of particle as it approached the door. Szilard's argument is very well information that this observation provides about the event X = Xi, i given in the book by Leff and Rex, (1990), and led to the realisation = 1,2,.. n, then the appropriate measure of information has to be of the connection between entropy and information. selected. If X and Y are statistically independent, then the occur- rence of Y = yj provides no information about the occurren