bits that must be transmitted to identify if only the probability distribution is available to the receiver, not the fact that . Mutual information The mutual information, is the relative entropy of the product of the two marginal probability distributions from the joint probability distribution — i.e. the expected number of extra bits that must be transmitted to identify and if they are coded using only their marginal distributions instead of the joint distribution. Equivalently, if the joint probability is known, it is the expected number of extra bits that must on average be sent to identify if the value of is not already known to the receiver. Shannon entropy The Shannon entropy, is the number of bits which would have to be transmitted to identify from equally likely possibilities, less the relative entropy of the uniform distribution on the random variates of , , from the true distribution — i.e. less the expected number of bits saved, which would have had to be sent if the value of were coded according to the uniform distribution rather than the true distribution . This definition of Shannon entropy forms the basis of E.T. Jaynes's alternative generalization to continuous distributions, the limiting density of discrete points (as opposed to the usual differential entropy), which defines the continuous entropy as which is equivalent to: Conditional entropy The conditional entropy, is the number of bits which would have to be transmitted to identify from equally likely possibilities, less the relative entropy of the product distribution from the true joint distribution — i.e. less the expected number of bits saved which would have had to be sent if the value of were coded according to the uniform distribution rather than the conditional distribution of given . Cross entropy When we have a set of possible events, coming from the distribution , we can encode them (with a lossless data compression) using entropy encoding. This compresses the data by replacing each fixed-length input symbol with a corresponding unique, variable-length, prefix-free code (e.g.: the events (A, B, C) with probabilities p = (1/2, 1/4, 1/4) can be encoded as the bits (0, 10, 11)). If we know the distribution in advance, we can devise an encoding that would be optimal (e.g.: using Huffman coding). Meaning the messages we encode will have the shortest length on average (assuming the encoded events are sampled from ), which will be