ce of X 4.3.1 Entropy of finite schemes = Xi. If, however, the random variables are fully dependent on one another, then the outcome of one determines the outcome of the Consider a complete system of events, A1, A2, An such that one and other. An obvious measure that satisfies these 'boundary conditions' only one event may happen at a given trial, e.g. tossing a coin, for is that of the logarithm of the ratio of the conditional probability which n = 2, and the events are mutually exclusive. P(xilyi) divided by P(xi), and this defines the mutual information Every finite scheme describes a state of uncertainty, and we only I(xi, between Xi and yj as in Equation 4.18. know the probability of possible outcomes, i.e. a one in six chance of a particular face of a dice being uppermost (for a fair dice). (4.18) If for one scheme the particular probabilities are different, then there will be different degrees of uncertainty associated with states having different probabilities. It is therefore essential to derive a measure of uncertainty of a particular scheme, and the quantity When the random variables are statistically independent, Equa- given in Equation 4.15 is a very suitable measure. tion 4.19 and therefore Equation 4.20 are obtained. (4.15) (4.19) (4.20) The logarithm is taken to be an arbitrary base, and The quantity H is called the entropy of the However, when the occurrence of event Y = y, uniquely deter- scheme. mines the occurrence of event X = x., the conditional probability Around the time when Szilard was considering Maxwell's demon above is unity, and therefore Equation 4.21 is obtained which is and the connection between entropy and information, the whole of called the self information of the event X = physics was undergoing vast changes due to the development of quantum theory. Of particular interest is the work of J. Von Neu- (4.21) mann, who in 1932, showed that in the process of making measure- ments, the observer must be taken into account, and this gathered together ideas of entropy, information and