It’s just a metric.
It’s not important to know how it came to be.
It’s important to know how to read it and what it tells us, which we just did above.
Entropy is a measure of disorder or uncertainty and the goal of machine learning models and Data Scientists in general is to reduce uncertainty.
Now we know how to measure disorder.
Next we need a metric to measure the reduction of this disorder in our target variable/class given additional information( features/independent variables) about it.
This is where Information Gain comes in.
Mathematically it can be written as:Information Gain from X on YWe simply subtract the entropy of Y given X from the entropy of just Y to calculate the reduction of uncertainty about Y given an additional piece of information X about Y.
This is called Information Gain.
The greater the reduction in this uncertainty, the more information is gained about Y from X.
Let me illustrate with an example using a simple contingency table and then I’ll bring all of it together with how decisions trees use entropy and information gain to decide what feature to split their nodes on as they are being trained on a data set.
Example: Contingency TableContingency TableHere our target variable is Liability which can take on two values “Normal” and “High” and we only have one feature called Credit Rating which can take on values “Excellent”, “Good” and “Poor”.
There are a total of 14 observations.
7 of them belong to the Normal Liability class and 7 belong to High Liability Class.
So it’s an even split by itself.
Summing across the top row we can see there are 4 observations that have value Excellent for the feature credit rating.
Furthermore , I can also see how my target variable is split for “Excellent” Credit Rating.
For observations that take have value “Excellent” for their credit rating there are 3 that belong to the Normal Liability class and only 1 that belongs to the High Liability class.
I can similarly figure out such values for other values of Credit Rating from the contingency table.
For this illustration , I will use this contingency table to calculate the entropy of our target variable by itself and then calculate the entropy of our target variable given additional information about the feature, credit rating.
This will allow me to calculate how much additional information does “Credit Rating” provide for my target variable “Liability”.
Let’s get to it then.
The entropy of our target variable is 1, at maximum disorder due to the even split between class label “Normal” and “High”.
Our next step is to calculate the entropy of our target variable Liability given additional information about credit score.
For this we will calculate the entropy for Liability for each value of Credit Score and add them using a weighted average of the proportion of observations that end up in each value.
Why we use a weighted average will become clearer when we discuss this in the context of decision trees.
We got the entropy for our target variable given the feature Credit Rating.
Now we can compute the Information Gain on Liability from Credit Rating to see how informative this feature is.
Knowing the Credit Rating helped us reduce the uncertainty around our target variable, Liability!.Isn’t that what a good feature is supposed to do?.Provide us information about our target variable?.Well that’s exactly how and why decision trees use entropy and information gain to determine which feature to split their nodes on to get closer to predicting the target variable with each split and also to determine when to stop splitting the tree!.( in addition to hyper-parameters like max depth of course).
Let’s see this in action with another example using decision trees.
Example: Decision TreeConsider an example where we are building a decision tree to predict wether a loan given to a person would result in a write-off or not.
Our entire population consists of 30 instances.
16 belong to the write-off class and the other 14 belong to the non-write-off class.
We have two features, namely “Balance” that can take on two values -> “< 50K” or “>50K” and “Residence” that can take on three values -> “OWN”, “RENT” or “OTHER”.
I’m going to show you how a decision tree algorithm would decide what attribute to split on first and what feature provides more information, or reduces more uncertainty about our target variable out of the two using the concepts of Entropy and Information Gain.
Feature 1: BalanceProvost, Foster; Fawcett, Tom.
Data Science for Business: What You Need to Know about Data Mining and Data-Analytic ThinkingThe dots are the data points with class right-off and the stars are the non-write-offs.
Splitting the parent node on attribute balance gives us 2 child nodes.
The left node gets 13 of the total observations with 12/13 ( 0.
92 probability) observations from the write-off class and only 1/13( 0.
08 probability) observations from the non-write of class.
The right node gets 17 of the total observation with 13/17( 0.
76 probability) observations from the non-write-off class and 4/17 ( 0.
24 probability) from the write-off class.
Let’s calculate the entropy for the parent node and see how much uncertainty the tree can reduce by splitting on Balance.
Splitting on feature ,“Balance” leads to an information gain of 0.
37 on our target variable.
Let’s do the same thing for feature, “Residence” to see how it compares.
Feature 2: ResidenceProvost, Foster; Fawcett, Tom.
Data Science for Business: What You Need to Know about Data Mining and Data-Analytic ThinkingSplitting the tree on Residence gives us 3 child nodes.
The left child node gets 8 of the total observations with 7/8 (0.
88 probability) observations from the write-off class and only 1/8 (0.
12 probability) observations from the non-write-off class.
The middle child nodes gets 10 of the total observations with 4/10 (0.
4 probability) observations of the write-off class and 6/10( 0.
6 probability) observations from the non-write-off class.
The right child node gets 12 of the total observations with 5/12 ( 0.
42 probability) observations from the write-off class and 7/12 ( 0.
58 ) observations from the non-write-off class.
We already know the entropy for the parent node.
We simply need to calculate the entropy after the split to compute the information gain from “Residence”The information gain from feature, Balance is almost 3 times more than the information gain from Residence!.If you go back and take a look at the graphs you can see that the child nodes from splitting on Balance do seem purer than those of Residence.
However the left most node for residence is also very pure but this is where the weighted averages come in play.
Even though that node is very pure, it has the least amount of the total observations and a result contributes a small portion of it’s purity when we calculate the total entropy from splitting on Residence.
This is important because we’re looking for overall informative power of a feature and we don’t want our results to be skewed by a rare value in a feature.
By itself the feature, Balance provides more information about our target variable than Residence.
It reduces more disorder in our target variable.
A decision tree algorithm would use this result to make the first split on our data using Balance.
From here on, the decision tree algorithm would use this process at every split to decide what feature it is going to split on next.
In a real world scenario , with more than two features the first split is made on the most informative feature and then at every split the information gain for each additional feature needs to be recomputed because it would not be the same as the information gain from each feature by itself.
The entropy and information gain would have to be calculated after one or more splits have already been made which would change the results.
A decision tree would repeat this process as it grows deeper and deeper till either it reaches a pre-defined depth or no additional split can result in a higher information gain beyond a certain threshold which can also usually be specified as a hyper-parameter!There you have it!.You now know what entropy and information gain are and how they are computed.
You understand how a decision tree either by itself or in a tree based ensemble decides on the best order of features to split on and decides when to stop when it trains itself on given data.
If you every have to explain the intricacies of how decision trees work to someone, hopefully you won’t do too bad.
I hope you were able to get some value from this post.
If there is anything that I missed or something was inaccurate or if you have absolutely any feedback , please let me know in the comments.
I would greatly appreciate it.
Ni kipimo tu. Si muhimu kujua ilitokeaje. Ni muhimu kujua jinsi ya kusoma na nini inatuambia, ambayo sisi tu alifanya juu. Entropy ni kipimo cha machafuko au kutokuwa na uhakika na lengo la mifano ya kujifunza mashine na Wanasayansi wa Takwimu kwa ujumla ni kupunguza kutokuwa na uhakika. Sasa tunajua jinsi ya kupima machafuko. Kisha tunahitaji kipimo kupima kupunguzwa kwa ugonjwa huu katika variable yetu lengo / darasa kutokana na habari ya ziada (viungo / vigezo huru) kuhusu hilo. Hapa ndipo Information Gain inapoingia. "Kutokana na hii, tunaweza kuandika ""Information Gain from X on Y"" na ""Kupunguza entropy ya Y kutokana na X"" na ""Kupunguza entropy ya Y kutokana na Y"" na ""Kupunguza entropy ya Y kutokana na Y.""" Hii inaitwa Kupata Habari. Kadiri ya kupunguzwa katika kutokuwa na uhakika huu, habari zaidi ni kupatikana kuhusu Y kutoka X. Hebu kuonyesha na mfano kwa kutumia rahisi contingency meza na kisha mimi itabidi kuleta yote pamoja na jinsi maamuzi miti kutumia entropy na kupata habari kuamua nini kipengele kugawanya nodes yao juu ya kama wao ni mafunzo juu ya seti ya data. Mfano: Contingency TableContingency TableHere variable yetu lengo ni dhima ambayo inaweza kuchukua juu ya maadili mawili <unk> Normal<unk> na <unk> High<unk> na tuna tu kipengele kimoja aitwaye Credit Rating ambayo inaweza kuchukua juu ya maadili <unk> Excellent<unk>, <unk>Good<unk> na <unk>Poor<unk>. Kwa jumla, kuna uchunguzi 14 uliofanywa. 7 kati ya hizo ni za darasa la Normal Liability na 7 ni za darasa la High Liability. Hivyo ni hata kugawanywa na yenyewe. Kwa kuzingatia idadi ya alama zilizopo, tunaweza kuona kwamba kuna alama nne ambazo zina thamani ya juu kwa kiwango cha sifa ya sifa. Zaidi ya hayo, naweza pia kuona jinsi variable yangu lengo ni kugawanywa kwa <unk>Excellent<unk> Credit Rating. Kwa maoni ambayo kuchukua kuwa na thamani <unk>Excellent<unk> kwa ajili ya rating yao ya mikopo kuna 3 kwamba ni mali ya darasa la kawaida wajibu na 1 tu kwamba ni mali ya darasa la juu wajibu. Mimi inaweza vivyo hivyo kufikiri maadili kama kwa maadili mengine ya Credit Rating kutoka meza contingency. Kwa mfano huu, tutatumia meza hii ya hali ya dharura kuhesabu entropy ya variable yetu ya lengo yenyewe na kisha kuhesabu entropy ya variable yetu ya lengo kutokana na habari ya ziada kuhusu kipengele, sifa ya mikopo. Hii itaruhusu mimi kuhesabu kiasi gani cha habari ya ziada haina <unk>Credit Rating<unk> kutoa kwa ajili ya variable yangu lengo <unk>Liability<unk>. Hebu tufikie hilo basi. Entropy ya variable yetu lengo ni 1, katika machafuko ya juu kutokana na hata kugawanywa kati ya darasa lebo <unk> Normal <unk> na <unk> High <unk>. Hatua yetu ya pili ni kuhesabu entropy ya lengo letu variable dhima kutokana na habari ya ziada kuhusu alama ya mikopo. Kwa ajili ya hii, tutaweza kuhesabu entropy kwa wajibu kwa kila thamani ya alama ya mikopo na kuongeza yao kwa kutumia wastani wa uzito wa uwiano wa uchunguzi kwamba kuishia katika kila thamani. Kwa nini sisi kutumia wastani uzito itakuwa wazi zaidi wakati sisi kujadili hii katika muktadha wa miti ya uamuzi. Tulipata entropy kwa ajili ya variable yetu lengo kutokana na kipengele Credit Rating. Sasa tunaweza kuhesabu Taarifa Gain juu ya dhima kutoka Rating Credit kuona jinsi taarifa kipengele hiki ni. Kujua Rating ya Mikopo ilituwezesha kupunguza kutokuwa na uhakika karibu na variable yetu ya lengo, Liability!.Isn't hiyo nini kipengele kizuri inapaswa kufanya?.Tupe habari kuhusu variable yetu ya lengo?.Naam hiyo ni jinsi gani na kwa nini miti ya uamuzi hutumia entropy na kupata habari kuamua ni kipengele gani cha kugawanya nodes zao juu ya kupata karibu na kutabiri variable ya lengo na kila kugawanyika na pia kuamua wakati wa kuacha kugawanyika mti!. (zaidi ya hyper-parameters kama kina cha juu bila shaka). Hebu kuona hii katika hatua na mfano mwingine kutumia miti ya uamuzi. Mfano: Mti wa Uamuzi Fikiria mfano ambapo sisi ni kujenga mti wa uamuzi kutabiri kama mkopo kupewa mtu ingekuwa kusababisha kuandika-off au la. Idadi ya watu wote katika jamii hii ni 30 na ni pamoja na: 16 ni za aina ya kuondolewa na 14 ni za aina isiyo ya kuondolewa. "Hii ni pamoja na ""malipo"" ambayo inaweza kuchukua juu ya thamani mbili - 50k au 50k, na ""makazi"" ambayo inaweza kuchukua juu ya thamani tatu - mmiliki,"" kukodisha"" au ""mzazi.""" Mimi nina kwenda kuonyesha jinsi uamuzi mti algorithm ingekuwa kuamua nini sifa ya kugawanywa juu ya kwanza na nini kipengele hutoa habari zaidi, au hupunguza kutokuwa na uhakika zaidi kuhusu variable yetu lengo nje ya mbili kwa kutumia dhana ya Entropy na Habari Gain. Kipengele cha 1: BalanceProvost, Foster; Fawcett, Tom. Data Sayansi kwa Biashara: Nini Unahitaji Kujua Kuhusu Data Mining na Data-Analytic Kufikiria: Dots ni data pointi na darasa haki-off na nyota ni non-kuandika-offs. Kugawanya mzazi node juu ya sifa usawa inatupa 2 mtoto nodes. Node ya kushoto inapata 13 ya jumla ya uchunguzi na 12 13 (0. "Kama matokeo ya uchunguzi wa ""Uwezekano wa kuanguka"" wa aina ya ""Uwezekano wa kuanguka"" wa aina ya ""Uwezekano wa kuanguka"" wa aina ya ""Uwezekano wa kuanguka"" wa aina ya ""Uwezekano wa kuanguka"" wa aina ya ""Uwezekano wa kuanguka"" wa aina ya ""Uwezekano wa kuanguka"" wa aina ya ""Uwezekano wa kuanguka.""" (Mfano: Uwezekano wa kutoweza kuandika kwa sababu ya kutoweza kuandika kwa sababu ya kutoweza kuandika kwa sababu ya kutoweza kuandika) Node ya kulia inapata 17 ya jumla ya uchunguzi na 13 17 0. "Kama matokeo ya uchunguzi wa ""Uwezekano wa kuanguka"" na ""Uwezekano wa kuanguka"" na ""Uwezekano wa kuanguka"" na ""Uwezekano wa kuanguka"" na ""Uwezekano wa kuanguka"" na ""Uwezekano wa kuanguka"" na ""Uwezekano wa kuanguka"" na ""Uwezekano wa kuanguka"" na ""Uwezekano wa kuanguka"" na ""Uwezekano wa kuanguka.""" 24) kutoka darasa la kufuta. Hebu mahesabu entropy kwa ajili ya mzazi node na kuona jinsi mengi ya kutokuwa na uhakika mti unaweza kupunguza kwa kugawanyika juu ya usawa. Kugawanywa juu ya kipengele, <unk> Balance <unk> inaongoza kwa faida ya habari ya 0. 37 kwa ajili ya variable yetu lengo. Hebu kufanya kitu kimoja kwa ajili ya kipengele, <unk>Residence<unk> kuona jinsi inalinganisha. Sehemu ya 2: ResidenceProvost, Foster; Fawcett, Tom. Data Sayansi kwa Biashara: Nini Unahitaji Kujua Kuhusu Data Mining na Data-Analytic Thinking: Kugawanya Mti juu ya Makazi Hutoa 3 Watoto nodes Kwa upande wa kushoto, node ya mtoto inapata 8 ya jumla ya uchunguzi na 7 8 (mfano). 88% ya matokeo ya utafiti wa utafiti wa majaribio ya majaribio ya majaribio ya majaribio ya majaribio ya majaribio ya majaribio ya majaribio ya majaribio ya majaribio ya majaribio ya majaribio ya majaribio ya majaribio ya majaribio ya majaribio ya majaribio ya majaribio ya majaribio. 12 (k) Uwezekano wa kugundua kwamba kuna uwezekano wa kugundua kwamba kuna uwezekano wa kugundua kwamba kuna uwezekano wa kugundua kwamba kuna uwezekano wa kugundua kwamba kuna uwezekano wa kugundua kwamba kuna uwezekano wa kugundua kwamba kuna uwezekano wa kugundua kwamba kuna uwezekano wa kugundua. Mtoto node inapatikana katika jumla ya 10 ya uchunguzi na 4 katika 10. 4 (Uwezekano) wa kuamua kwa ajili ya kuondoa au kuondoa 6 probability) uchunguzi kutoka darasa lisilo la kuandika. Kwa upande wa kulia, node ya mtoto inapata 12 ya jumla ya uchunguzi na 5 12 (0. 42) na 7 (mfano wa) 58) Maelezo kutoka kwa darasa lisiloandikwa. Tayari tunajua entropy kwa ajili ya mzazi node. "Kama tunavyojua, entropy ya ""kugawanyika"" ni sawa na entropy ya ""kugawanyika"" na ""kugawanyika"" na ""kugawanyika"" na ""kugawanyika"" na ""kugawanyika"" na ""kugawanyika"" na ""kugawanyika"" na ""kugawanyika"" na ""kugawanyika"" na ""kugawanyika"" na ""kugawanyika"" na ""kugawanyika.""" Hata hivyo kushoto zaidi node kwa ajili ya makazi pia ni safi sana lakini hii ni ambapo wastani uzito kuja katika kucheza. Hata ingawa node hiyo ni safi sana, ina kiasi kidogo cha jumla ya uchunguzi na matokeo inachangia sehemu ndogo ya usafi wake wakati tunahesabu jumla ya entropy kutoka kugawanyika juu ya Makazi. Hii ni muhimu kwa sababu sisi ni kuangalia kwa nguvu ya jumla ya habari ya kipengele na hatutaki matokeo yetu kuwa skewed na thamani nadra katika kipengele. Kwa kipengele yenyewe, Balance hutoa habari zaidi kuhusu variable yetu lengo kuliko Makazi. Inapunguza machafuko zaidi katika variable yetu lengo. Uamuzi mti algorithm ingekuwa kutumia matokeo haya kufanya mgawanyiko wa kwanza juu ya data yetu kwa kutumia Balance. Kutoka hapa na kuendelea, uamuzi mti algorithm ingekuwa kutumia mchakato huu katika kila mgawanyiko kuamua nini kipengele ni kwenda kugawanyika juu ya ijayo. Katika hali ya ulimwengu halisi, na zaidi ya vipengele viwili, mgawanyiko wa kwanza unafanywa kwenye kipengele cha habari zaidi, na kisha katika kila mgawanyiko, faida ya habari kwa kila kipengele cha ziada inahitaji kuhesabiwa tena kwa sababu haingekuwa sawa na faida ya habari kutoka kwa kila kipengele peke yake. Entropy na taarifa faida ingekuwa na kuhesabiwa baada ya moja au zaidi kugawanyika tayari wamekuwa kufanywa ambayo ingekuwa kubadilisha matokeo. Mti wa uamuzi utafanya mchakato huu mara kwa mara, kwani unakua kwa kina na kina hadi kufikia kina cha awali au hakuna mgawanyiko wa ziada unaweza kusababisha faida ya habari ya juu zaidi ya kizingiti fulani ambacho pia kinaweza kuelezwa kama hyperparameter. Mti wa uamuzi, au kwa wenyewe au katika ensemble ya mti, huamua juu ya utaratibu bora wa vipengele vya kugawanyika na kuamua wakati wa kuacha wakati unajifunza juu ya data iliyotolewa. Kama kila mmoja ana kueleza intricacies ya jinsi miti ya uamuzi kazi kwa mtu, hopefully wewe si kufanya mbaya sana. Natumaini uliweza kupata thamani kutoka kwa chapisho hili. Kama kuna kitu ambacho mimi amekosa au kitu ilikuwa sahihi au kama una kabisa maoni yoyote, tafadhali nijulishe katika maoni. Ningethamini sana jambo hilo.