Concept of Cross-Entropy
Entropy can be calculated using formula:-

$Entropy = -p\log_2 p − q\log_2 q$

Here $p$ and $q$ are probability of success and failure respectively in that node.
Entropy is also used with categorical target variable. It chooses the split which has
lowest entropy compared to parent node and other splits. The lesser the entropy,
the better it is.

Steps to calculate entropy for a split:


Calculate entropy of parent node
Calculate entropy of each individual node of split and calculate weighted
average of all sub-nodes available in split.