ave cemented Shannon's place as the father of information theory. Information Theory Information theory can be defined as a field of science and technology which quantifies the amount of information included in a message. The following example may serve to better explain the theory. The following two statements convey information on winter weather in two cities where Carnegie Mellon University has a campus. Which of the two statements has more information? Last winter it snowed in Adelaide, Australia. Last winter it snowed in Pittsburgh, USA. Pittsburgh Adelaide Information Theory It is difficult to know which of the two statements conveys more information without knowing about the general probability of snowing in each city. Therefore, let us assume that over the past 100 winters there has been only one snowfall in Adelaide, whereas there has been only one winter when it has not snowed in Pittsburgh. In other words, the chance (probability) of snowfall during an Adelaide winter is 1% whereas it is 99% for Pittsburgh. Given this information which statement gives a recipient of the message more information? Which one is more newsworthy? It is reasonable to say that the first statement is more likely to make it to the news as it occurs less frequently (which in fact does - a mere snow flake or two on the hills near Adelaide is sure to make the headline news on all TV stations!) Given the close relation between information content of a message and the probability of its occurrence, information theory quantifies the information content of a message, I(m), by: m)=-log2p(m) (4.1) where p(m) is the probability that message m is generated. The unit for information content is a 'bit'. For example, the information content of the 'Last winter it snowed in Adelaide, Australia' message is -log, (0.01)=6.64 bits, whereas the information content of the 'Last winter it snowed in Pittsburgh, USA' message is -log, (0.99)=0.01 bits [note, log,(x) 22 This quantifies how informative these two statements are, and makes a comparison po