1978), the entropy of a thermodynamic The rules for conducting scientific inference follow from the system is a measure of the degree of ignorance of a person whose simple laws of probability as in Equations 4.4 and 4.5. sole knowledge about its microstate consists of the values of the macroscopic quantities, e.g. pressure and temperature, which define P(ABI)=p(AI)p(B|AI) its thermodynamic state, and it is a completely objective quantity in =p(B|I)p(A|BI) (4.4) the sense that it can be measured in the laboratory. A very good source of reference concerning both the thermody- p(A|B)+p(A)B) 1 (4.5) namic and the information processing approach to entropy and information can be found in the recent book edited by Leff and Rex 0 then Bayes' theorem follows as in Equation 4.6, (1990). where I stands for any prior information. Before we concentrate on information theory proper, it will be worth laying a few foundations concerning the role of probability (4.6) theory in information, and in trying to use probability theory in information theory, one immediately sees that there exist many different views as to what probability theory is, a sample of the Using Bayes' theorem, it is possible to incorporate chains of various views being: evidence into one's reasoning, the posterior probability becoming the prior for the next iteration as new data become available. This 1. Kolmogorov; The theory of additive measures. method of inference goes back to Laplace in the 18th century, but 2. Jeffreys; The theory of rational belief. the mathematics necessary to show that Bayesian inference is the 3. Fisher; The theory of frequencies in random experiments. only logical way to proceed was given in 1946 by R.T. Cox. 4. de Morgan; The calculus of inductive reasoning. However, the formalism does not show us how to assign the initial 5. Laplace; Common sense reduced to calculation. probabilities. There are several methods that may be used to assist 6. Bernoulli; The art of conjecture. in the initial assignment of probabilities, the first one