to understand thatlet us pretend we want to use features from the feature set house_sizenumber_of_levelsand avg_rent_price to train classifier that outputs whether the house has an elevator or not in this examplewe intuitively see that knowing house_size means we don' need number_of_levels anymoresince it somehow contains redundant information with avg_rent_priceit is different as we cannot infer the value of rental space simply from the size of the house or the number of levels it has thus we would be wise to keep only one of them in addition to the average price of rental space mutual information formalizes the previous reasoning by calculating how much information two features have in common but unlike correlationit does not rely on sequence of databut on the distribution to understand how it workswe have to dive bit into information entropy let' assume we have fair coin before we flip itwe will have maximum uncertainty as to whether it will show heads or tailsas both have an equal probability of percent this uncertainty can be measured by means of claude shannon' information entropyin our fair coin scenariowe have two caseslet case of tailswith be the case of heads and the thuswe get the followingfor conveniencewe can also use scipy stats entropy([ ]base= we set the base parameter to to get the same result as the previous one otherwisethe function will use the natural logarithm via np log(in generalthe base does not matter as long as you use it consistently now imagine we knew upfront that the coin is actually not that fairwith the heads side having percent chance of showing up after flipping