Many classification algorithms, both in machine and in deep learning, adopt the cross-entropy as cost function. This is a brief explanation why minimizing the cross-entropy allows to increase the mutual information between training and learned distributions.

If we call p the training set probability distribution and q, the corresponding learned one, the cross-entropy is: By manipulating this expression, we get: Therefore, the cross-entropy is equal to the sum of H(p), which is the entropy of the training distribution (that we can’t control) and the Kullback-Leibler divergence of the learned distribution from the training one. As the first term is a constant, minimizing the cross-entropy is equivalent to minimizing the Kullback-Leibler divergence.

We know that: Therefore, the training process will “remodel” q(x) in order to minimize its divervenge from p(x). In the following figure, there’s a schematic representation of this process before the initial iteration, at iteraion n and at the end of the training process: If we consider, the mutual information between p(x) and q(x), we obtain: The mutual information is the amount of information shared by both distributions and it is expressed as the entropy of the training distribution minus the “unresolved” uncertainty that q provides when chosen instead of p. In other words, if we have modeled q perfectly, it means that q = p, therefore H(q|q) = 0 and I(p;q) = H(p) (that represent the maximum amount of information that we can learn). Therefore, when we minimize the cross-entropy, we implicitly minimize the conditional entropy H(p|q), obtaining a maximization of the mutual information.

A good introductory book on Information Theory:
{"ft_lang_label":"__label__en","ft_lang_prob":0.9072631,"math_prob":0.96687526,"size":2289,"snap":"2019-51-2020-05","text_gpt3_token_len":472,"char_repetition_ratio":0.14529541,"word_repetition_ratio":0.011560693,"special_character_ratio":0.19397117,"punctuation_ratio":0.104116224,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.9989767,"pos_list":[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14],"im_url_duplicate_count":[null,4,null,2,null,2,null,2,null,2,null,null,null,null,null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2019-12-08T13:38:01Z\",\"WARC-Record-ID\":\"<urn:uuid:b2d7d2e3-c77c-4652-b1fd-bee2c5885919>\",\"Content-Length\":\"76323\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:ceddaa29-1e19-4a6f-b3fe-94d48e91a47b>\",\"WARC-Concurrent-To\":\"<urn:uuid:2fef986e-ffdc-47d9-b5d4-cc0dc1409a9c>\",\"WARC-IP-Address\":\"104.25.94.117\",\"WARC-Target-URI\":\"https://www.bonaccorso.eu/2017/08/18/ml-algorithms-addendum-mutual-information-in-classification-tasks/\",\"WARC-Payload-Digest\":\"sha1:6DW4S4OFP32T4YMP6Q3AGQQ2IXOOPMAL\",\"WARC-Block-Digest\":\"sha1:CPADNIMLXLBXT2AWUGVCO3RVSFN7PYG2\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2019/CC-MAIN-2019-51/CC-MAIN-2019-51_segments_1575540510336.29_warc_CC-MAIN-20191208122818-20191208150818-00399.warc.gz\"}"}