escribes X. According to late the average gain of information associated with such events. As Kolmogorov, Information theory must precede probability theory, an example, consider coin tosses, such that the outcome, heads or and should not be based on it, and his contribution to information tails, occurs with equal probability of 1/2. theory followed by a direct development of his ideas in algorithmic The information gain acquired by learning that the outcome of an complexity. In particular, he was interested in finding any determin- experiment is heads, say, is given by Equation 4.13. ism in random events, and his work on turbulence must be seen in this light as an attempt to find deterministic order in chaotic pro- (4.13) cesses, and indeed turbulence was one of the key phenomena that motivated the resurgence of interest in non-linear dynamical sys- items, and the range of chaotic phenomena and strange attractors. In general, therefore, the information gain associated with many It is interesting that many of the ideas borrowed from information states having probabilities associated with each state, is given by theory are used to classify many of the chaotic signals, such as heart Equation 4.14. Information and thermodynamics 4/5 (4.14) Suppose that X is a random variable or random vector that can be described by its probability density function p (x), where x takes values in a certain alphabet, A. Shannon (1949), defined the en- tropy of X by Equation 4.16. 4.3 Information and thermodynamics (4.16) Boltzmann realised that entropy was a measure of disorder of the configuration of the system being considered. This viewpoint was extended by Szilard in 1929, and he identified entropy with infor- Similarly, given two discrete random variables X and Y, one can mation, and the measure of information with the negative of the define various entropies H(X,Y), H(X), H(Y) as well as the condi- measure of entropy. This laid the foundations of information theory. tional entropy as in Equation 4.17 (see the next section). This