 classification and unsupervised learning. On the contrary, possible statistical models of the other measures need to take into account the exact form of the objective function (or the loss function) and therefore, they cannot be generally applicable. Lastly, for small sample sizes, as also noted by Archer and Kimes in [@Archer2008], subsampling or bootstrapping of the samples during training might not be desirable, which limits the use of permutation importance.

A Model of False Positive Rates for Selection Frequency {#sec:model}
=======================================================

The main question we want to answer in this section is [*“What is the probability that a feature $f$ is selected $k$ times in a Random Forest under the null hypothesis?”*]{} with the null hypothesis being [*“There is no statistical relationship between features (covariates) and labels (response variables).”*]{} Answering this question will allow us to estimate false positive rates for a given threshold on selection frequency. Consequently, it will provide us a principled way to determine thresholds to split relevant and non-relevant features.

Our strategy is to build an approximate probabilistic model of selection frequency that takes into account the details of the training procedure, such as the feature subsampling scheme, number of trees and the size of the random feature subset each node optimizes over[^3]. Once this model is built we can compute the expected rate of false positives for a given threshold on selection frequency and consequently determine thresholds by limiting this rate. We take a bottom-up approach and start by modeling the probability of a feature $f$ being selected at a given node and build-up the model from there.

The randomness in the feature selection process is due to the random nature of forest training. As mentioned in Section \[sec:RF\] there are two components that provide this random nature: feature subsampling and sample subsampling or bootstrapping. In our approach we explicitly formulate the feature subsampling and this is the source of the probabilistic nature of our model. We do not explicitly model the sample subsampling scheme and its effect on the selection frequency. There are two main reasons for this. First, the effect of sample subsampling on feature selection is much more difficult to explicitly formulate. Second, for small datasets bootstrapping or subsampling of the data might not be used at all when training a random forest. Feature subsampling on the other hand is used all the time. However, although we do not model sample
classification and unsupervised learning. On the contrary, possible statistical model of the early measures need to bring into report the exact form of the objective routine (or the passing function) and consequently, they cannot be by and large applicable. Lastly, for little sample sizes, as besides noted by Archer and Kimes in [ @Archer2008 ], subsampling or bootstrapping of the samples during training might not be desirable, which limits the use of permutation importance. 

 A Model of False Positive Rates for Selection Frequency { # sec: model } 
 = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = 

 The chief question we want to answer in this incision is [ * “ What is the probability that a feature $ f$ is selected $ k$ times in a Random Forest under the null guess? ” * ] { } with the null hypothesis being [ * “ There exist no statistical relationship between features (covariates) and labels (reception variables). ” * ] { } Answering this question will allow us to estimate false positive rates for a given threshold on selection frequency. Consequently, it will provide us a principled way to determine thresholds to split relevant and non - relevant features. 

 Our scheme is to construct an approximate probabilistic model of excerpt frequency that remove into account the details of the training procedure, such as the feature subsampling scheme, number of trees and the size of the random feature of speech subset each node optimizes over[^3 ]. Once this model is built we can compute the expected rate of false positives for a given threshold on selection frequency and consequently determine threshold by restrict this rate. We take a bottom - up approach and start by modeling the probability of a feature of speech $ f$ being selected at a given node and build - up the model from there. 

 The randomness in the feature selection process is due to the random nature of forest education. As mentioned in Section   \[sec: RF\ ] there are two components that provide this random nature: feature subsampling and sample distribution subsampling or bootstrapping. In our approach we explicitly formulate the feature subsampling and this is the source of the probabilistic nature of our exemplar. We do not explicitly model the sample subsampling outline and its effect on the selection frequency. There are two main reasons for this. foremost, the effect of sample subsampling on feature choice is much more difficult to explicitly formulate. Second, for small datasets bootstrapping or subsampling of the data might not be use at all when training a random forest. feature of speech subsampling on the other hand is used all the time. However, although we do not model sample
 clwssification and unsuperyised learning. Ou the cmntrarg, possibue statistical models of the orher neasures need to take knto accolnt the ezact dorm of thx objectlre fuhgtion (ir the loss fukction) and dherefore, they cxnuot be generally applicable. Lastly, fjr smalk dample sizes, af alxj nofvd by Archer and Kimes in [@Arched2008], subsakpling or booystrapping of the samples furijg training might jot be desitzblq, which limitr the use of permutatikn importance.

A Model of False Pusitire Rates fot Aepgction Frequxncy {#svc:model}
=======================================================

The malm quesdion we want to answev in vhis section is [*“What is tie probability that w feature $f$ is selected $k$ tinew in d Ratdom Dordst uided the julm hypothesjs?”*]{} with the null hypothesis beond [*“There is no atatiseisal relationship between features (covarpatea) and labels (response vqriables).”*]{} Answering thls questijn will allow us to estimate false positive rates for e eivtn threryopd on selection frequency. Consequently, it wilj ptonide us a principjed way to cehetiine thresholas to spljt relevant and noj-relevagt feqtures.

Our strstegy is to build an approxumate probabplisric model of seleccion frequeney thaj takex into account the detanls of the trainijg procedhfe, such as the fdatlre vubsampling scheme, number jf trees end tke size uf tne ranqom featurf subset each node optimlzes lvar[^3]. Once thls model is built we can computx the expectec satv of falsz posibives for a givqn threshold ou selectnon frdquency ans consexuently detqrmine threshmpds by limitmng this wate. We rake a cuttom-up approavh and stcxt by modwling the probabillty ow a feature $f$ benug selected at a bivdn gofe agg build-up tha moael grom ghere.

Tht rcnaomnrss in the feature salecfion process is dur bo the rabdom nattre of forest training. As mentilned mn Secvion \[sev:RF\] there are two components that provide hhif random nattre: neatore subsampking and sample subsampling or bootstra'ping. In our approach ww explicitly formulctt the feature subswmpling atd this is the sourcw of the probabillstic nature of our mosel. We do nlt explicitly model the sample subsampling scheme and its effect on the swlection frequency. Thete arv tco kain rqasois for this. First, nhe effect of sample subsampling on featgrz selection is much more diffocjlt to explicktly formulate. Second, fod small catasets bootstrapping or subsakpling of the data might not be ysed zt all whei training a random gorest. Featyre subwampping on the oghet mand is usfd akl the time. However, although we do not midfl sample
classification and unsupervised learning. On the contrary, models the other need to take of objective function (or loss function) and they cannot be generally applicable. Lastly, small sample sizes, as also noted by Archer and Kimes in [@Archer2008], subsampling bootstrapping of the samples during training might not be desirable, which limits the of importance. Model False Positive Rates for Selection Frequency {#sec:model} ======================================================= The main question we want to answer in section is [*“What is the probability that a $f$ is selected $k$ in a Random Forest under null with the hypothesis [*“There no statistical relationship features (covariates) and labels (response variables).”*]{} Answering this question will allow us to estimate false positive rates a given selection frequency. it provide a principled way thresholds to split relevant and non-relevant is to build an approximate probabilistic model of frequency that into account the details of the procedure, such as the feature subsampling scheme, number trees and the size of the random feature subset each node optimizes over[^3]. Once this built we can compute expected rate of positives a threshold selection frequency consequently determine thresholds by limiting this rate. We take a bottom-up and start by modeling the probability of a feature $f$ at given node and the model from there. randomness the feature selection process to random training. mentioned Section \[sec:RF\] there are components that provide this random feature subsampling and sample approach we explicitly formulate the feature subsampling and is the source of the probabilistic nature our model. We do not explicitly model the sample subsampling scheme and effect on frequency. There are two main reasons for this. the effect of sample on feature selection is much more difficult to explicitly Second, small datasets or subsampling of data might not used at all a random Feature on is used all the time. However, we not model sample
 classification and unsupervIsed learniNg. On tHe cOntRaRy, poSsibLe statistical mODels Of the other measures need To takE iNTo acCOuNt the Exact foRM oF THe oBjEcTivE fUNcTion (oR thE loss fuNction) and tHerEfOre, they cannoT Be Generally aPplIcable. Lastly, For Small sAmPle SIzes, aS alSo notEd by ArCHer and kimes in [@ArChER2008], subsaMPling or BOOtStraPping of the samples DUrINg training mighT not be DeSIrABLe, wHicH limits the UsE of peRMutatioN ImPORTanCE.

A Model of FalsE Positive RaTEs fOr SeleCtIon fRequenCy {#sec:MoDEl}
=======================================================

THe main questIon wE want to anSwer in THis sectIOn is [*“WhaT is the ProBabIlitY ThAt A feAtURe $f$ IS sEleCTed $K$ times in A RAnDom FoRest UNDER the NulL hypOthesIs?”*]{} with the null HypOtheSIs bEing [*“THere iS no sTaTistiCal relAtionShIp between featurEs (coVariates) aNd lAbEls (ReSponsE VariabLes).”*]{} ansWering tHis quesTIon WiLL ALlOw us to estimate falsE pOSItIve rates For a giVEn ThREshold on SeLecTion FREquenCy. CoNSeQuently, iT will pROvIdE us a priNcIpled wAy To dEteRmine THresHolds tO split reLevanT And non-relevant FEatures.

Our strATeGY Is TO buiLd aN approximatE proBAbilIstiC MoDel OF seleCtion FrEQuENcy that takes into accOuNt the dEtailS of the traininG procedure, SUCH as the feAturE SuBSampling scheme, NumbeR of trees anD The size oF the rAndom feaTure subseT EAch node oPtiMizEs oVer[^3]. oNCe This model is buILT we cAn Compute The ExpecteD raTe oF faLse PoSitives foR a given tHrEsHoLd On sElectIOn frequeNcY anD cOnsEquenTLy deteRmine ThreShOlDS by LimitinG ThIS Rate. we TaKe a bOttOm-Up appRoacH And Start by Modeling tHe pRObabIlItY of a feaTure $f$ being selEcTed at a giveN nOde And buiLD-Up the modEl from there.

The randomnesS In the feAtuRe selEctiOn process Is dUe to thE raNDom natUre of fOrest TrAinING. As meNTIoNed In section \[sec:rf\] TheRe are TwO comPonents That provide this ranDOm nAture: feature sUbsAmplING aNd sAMpLE suBsAMplING or bootstrappinG. In our apprOaCH wE explicitlY ForMuLate the Feature SubsaMPling anD this is thE source of ThE proBABilIstic naturE of our moDel. We do noT ExpliCItLy modEl tHe sampLe SubSamplIng schEMe aNd its Effect On The selEctioN fRequency. there are two main reasons fOr this. first, The Effect of sAmpLE suBsampling On feAture selecTioN is Much mOre DIfficUlt tO ExPliCItly fOrmuLAte. Second, FOr SmaLL DaTasets bootsTRAPpiNg or sUbsAMpling Of thE data might not be usED at all when traiNing A RAndOm fORest. feAture subsampliNg oN tHE Other hanD iS used all the Time. HoweVeR, AlthoUgh we dO not moDel sampLE
 classification and unsupe rvised lea rning . O n t he  con trar y, possible st a tist ical models of the oth er me as u resn ee d totake in t oa c cou nt  t heex a ct  form  of  the ob jective fu nct io n (or the lo s sfunction)and  therefore,the y cann ot  be gener all y app licabl e . Last ly, for s ma l l samp l e sizes , as  als o noted by Archer an d  Kimes in [@Ar cher20 08 ] ,s u bsa mpl ing or boo ts trapp i ng of t h es a m ple s  during train ing might n o t b e desi ra ble ,  which  limi ts the  use of per muta tion impo rtance . 

A Mod e l of Fa lse Po sit ive  Rat e sfo r S el e cti o nFre q uen cy {#sec :m od el}
= ==== = = = = ==== === ==== ===== ============= === ==== = === ===== =

Th e ma in  ques tion w e wan tto answer in th is s ection is  [* “W hat  i s the probab ili tythat afeature $f$  i s s el ected $k$ times in  a R an dom Fore st und e rth e  null hy po the sis? ” * ]{} w itht he  null hy pothes i sbe ing [*“ Th ere is  n o s tat istic a l re lation ship bet weenf eatures (covar i ates) and lab e ls ( re s pons e v ariables).” *]{} Answ erin g  t his quest ion w il l  a l low us to estimatefa lse po sitiv e rates for a  given thr e s h old on s elec t io n  frequency. Co nsequ ently, itw ill prov ide u s a prin cipled wa y to deter min e t hre sho l d sto split rele v a nt a nd  non-re lev ant fea tur es. 

O urst rategy is  to buil dan  a pp rox imate probabil is tic  m ode l ofs electi on fr eque nc yt hat  takesi nt o acco un tthedet ai ls of  the tra ining p rocedure,  su c h as  t he  featur e subsampling  s cheme, num be r o f tree s and thesize of the random feat u re subs eteachnode  optimize s o ver[^3 ].O nce th is mod el is  b uil t we ca n co mpu te  the expec t e d r ate o ffals e posit ives for a given t h res hold on selec tio n fr e q ue ncy an d  co ns e que n t ly determine th resholds b yl im iting this rat e.  We tak e a bot tom-u p  approa ch and st art by mo de ling t heprobabilit y of a f eature $f $  bein g  s elect edat a g iv ennodeand bu i ld- up th e mode lfrom t here. 
The rand omness in the feature s electi on pr oce ss is due  to the  random n atur e of fores t t rai ning.  As menti oned in  Se c tion\[se c :RF\] the r eare t wo  components t h atprovi det his ra ndom  nature: features ubsampling and  sam p l e s ubs a mpli ng  or bootstrapp ing .I n  our app ro ach we expl icitly f or m ulate  the f eature  subsam p l in g  and t hisisthe sourc e o ft he prob ab il i stic n atur eof our  model .  Wed o  not explicitlymodel t he sa m ple  subs am pling s c heme  and its e ffect on th e sele ctio n fre quency.  T here a retw o main rea s ons for t his.First,th e ef fec t of s ampl e subsa mpli ng  on  features e le c ti on ismuch  more  d iffi cult to e x plicitly  fo r mulate.  S eco n d , fors ma l l  datasetsboo tstra p p ing or sub s ampl i ng of th e data  might  not be use dat allwhe n traininga randomf ore st . Fe ature su bs ampl in g o nthe other handis usedall thet i me. Ho wev e r, a l though  w e do not mod el sa mp l e
 classification_and unsupervised_learning. On the contrary,_possible statistical_models_of the_other_measures need to_take into account_the exact form of_the objective function_(or_the loss function) and therefore, they cannot be generally applicable. Lastly, for small sample_sizes,_as also_noted_by_Archer and Kimes in [@Archer2008],_subsampling or bootstrapping of the_samples during_training might not be desirable, which limits the_use_of permutation importance.

A_Model of False Positive Rates for Selection Frequency {#sec:model}
=======================================================

The_main question we want to answer_in this section_is_[*“What_is the probability that_a feature $f$ is selected $k$_times in a Random Forest under_the null hypothesis?”*]{} with the null hypothesis_being [*“There is no statistical relationship_between features (covariates) and labels_(response variables).”*]{}_Answering this question will allow_us to estimate_false positive_rates for a_given threshold on selection frequency. Consequently,_it will provide_us a principled way to determine_thresholds_to split relevant_and_non-relevant_features.

Our strategy_is to build_an_approximate probabilistic_model_of selection frequency that takes into_account_the details of the training procedure, such_as the feature subsampling_scheme,_number of trees and_the size of the random_feature subset each node optimizes over[^3]._Once this_model is_built we can compute the expected rate of false positives for_a given threshold on selection frequency_and consequently determine thresholds_by limiting_this_rate. We take_a_bottom-up approach_and start by modeling the probability of_a feature_$f$ being selected at a given_node and build-up the_model_from there.

The randomness in the feature_selection process is due to the_random nature of forest training._As_mentioned_in Section \[sec:RF\] there are two_components that provide this random nature:_feature subsampling and_sample subsampling or bootstrapping. In our approach_we_explicitly formulate the feature subsampling and_this_is the source of the probabilistic_nature_of_our model. We do not_explicitly model the sample subsampling scheme_and its effect on the selection frequency. There are_two main reasons_for this. First, the effect_of_sample_subsampling on feature selection is much more difficult to explicitly_formulate. Second,_for small datasets_bootstrapping or subsampling of the data might not be used_at all when training a random forest._Feature subsampling on the other hand is used all the time._However, although we do not model sample