 non-relevant features even when there is a complex correlation structure between the features. Especially when there are actually relevant features, the threshold is able to limit the false positive rates on the order of the desired rate while keeping the false negative rates low.

Discussions
===========

Experimental analysis demonstrated that the proposed method is able to provide thresholds that can limit the false positive rates on the order of the desired level, especially in the presence of relevant features, and keep false negative rates low for high-dimensional problems. How to get rid of the remaining false positives and improve upon the false negatives are the topics of this discussion section.

The main source of higher false positive rates than the desired level is spurious statistics that violate the Assumption \[assumption\]. This is actually the case for any sort of feature selection method. As the dimension of the problem gets higher, in other words as the ratio $F/S$ increases, the risk of having stronger spurious statistical relationships between the label and some non-relevant features also increases. The remedy to tackle this issue is subsampling of the samples either with bagging or bootstrapping. In our experiments we have set the bagging ratio to $1/2$. Reducing this ratio even further improves on the problem of spurious statistics. However, in most problems there are not enough samples to have a bagging ratio lower than $1/2$. An alternative would be to use a wrapping algorithm based on bootstrapping such as the Stability Selection [@meinshausen2010stability].

The most prominent cause of false negative rates is the competition between relevant features. Each relevant feature will manifest a different empirical correlation with the label. Hence, their selection frequencies will be different. When a relevant feature ranks lower on the ranking of relevant features based on empirical correlations, the chances of it making it above the threshold will decrease. Wrapper algorithms, such as forward or backward selection methods, combined with bootstrapping schemes can remedy this issue. Alternatively, the recently proposed knock-out strategies [@konukoglu2013feature], which remove the most relevant features rather than removing the non-relevant ones, can also solve this issue.

One of the biggest advantages of the proposed method is that it is straightforward to integrate it into a wrapping algorithm, such as the forward or backward feature selection, or a bootstrapping scheme such as Stability Selection [@meinshausen2010stability]. The proposed method determines in a principled way a threshold on
non - relevant features even when there is a complex correlation structure between the feature. particularly when there are actually relevant features, the threshold is able to specify the false positive rate on the club of the desired pace while retain the assumed negative rates broken. 

 Discussions 
 = = = = = = = = = = = 

 Experimental psychoanalysis demonstrated that the project method is able to provide thresholds that can limit the assumed positive rates on the order of the desire level, especially in the presence of relevant features, and keep assumed negative rates low for high - dimensional problems. How to get rid of the remaining false positives and improve upon the false negatives are the topics of this discussion section. 

 The main source of higher false convinced rates than the desired degree is inauthentic statistics that violate the Assumption   \[assumption\ ]. This is actually the case for any sort of feature selection method. As the property of the problem gets higher, in other words as the ratio $ fluorine / S$ increases, the risk of having stronger inauthentic statistical relationships between the label and some non - relevant feature of speech also increases. The remedy to tackle this issue is subsampling of the sample either with bagging or bootstrapping. In our experiments we have set the sacking ratio to $ 1/2$. reduce this ratio even further better on the problem of bastardly statistics. However, in most problems there embody not enough sample to have a bagging proportion lower than $ 1/2$. An alternative would be to practice a wrapping algorithm based on bootstrapping such as the Stability Selection   [ @meinshausen2010stability ]. 

 The most prominent cause of delusive negative rate is the competition between relevant features. Each relevant feature will manifest a different empiric correlation with the label. Hence, their choice frequencies will be different. When a relevant feature ranks lower on the ranking of relevant features based on empirical correlation, the chances of it making it above the threshold will decrease. Wrapper algorithms, such as forward or backward selection method, combine with bootstrapping schemes can remedy this issue. Alternatively, the recently proposed knock - out strategies   [ @konukoglu2013feature ], which remove the about relevant features rather than removing the non - relevant ones, can also solve this issue. 

 One of the biggest advantages of the proposed method acting is that it is straightforward to integrate it into a wrapping algorithm, such as the forward or backward feature selection, or a bootstrapping scheme such as Stability Selection   [ @meinshausen2010stability ]. The propose method determines in a principled way a threshold on
 noj-relevant features even dhen there is a complee corremation sgructure between the featured. Wspecually when there are aztually rvlevant fwatuces, the thresholv is ablc to mlmit chx false positivg rates on tve order of tha aedired rate while keeping the false gegativr gates low.

Discufsiomf
===========

Expsginental analysis demonstrated fhat tht proposed method os able to provide threshopds hhat can limit the false posijjve eates on the order of uhz desired lgvel, especially in the presence uf rekevant feajbees, dnd keep faose nvgative rates low for high-dikensional probkemv. Hiw to get rid of the cemaining false posijives and hm'rove upon the false betativgs ara thd tooica pf this fisrussion secfion.

The maib source of higher gajwe positive rztes trag the desired level is spurious statistpcs fhat violate the Assumprion \[assumption\]. This id actuallr the case for any sort of feature selection methmd. As ghe dlmenruoj of the problem gets higher, in other words af tne ratio $F/S$ incrcases, the risk of nagimd stronger sporious stafistical relationsjips bejween rhe label and some non-relevant features qlso increasvs. Tye remedy to tacklz this issue is xubsakpling of the samples enther sith bagginh or bootagrapping. In our dxpvrimants we have set the baggigg ratio vo $1/2$. Rzducing ghis ratio even furtjer improves on the probpem oy spusious statlstics. However, in most problems there are not etouch samplzs to mave a bagging watio lower thcn $1/2$. An ajternxtive wouls be to use a wrap[ing algorithk based on bomtstrappyng wuch as the Rtability Selevtion [@meinshausen2010stavility].

The most prokingnf cause of falsz begative rates os ghe clm'etitymn between ralevxnt geatufes. Each reodvany feature will manifast z different empirivaj correlqtion wieh the label. Nence, their selectlon fcequenries woll be different. When a relevant reature rwnkf lower on tre rqnking of rekevant features based on empirical corrxlations, the chances of it making it above uhe threshold will decrease. Wrapper algorithms, wuch as forward ov backward selection msthods, comblned with bootstrapping schemes can remedy this issue. Alternatively, the rwcentlb [roposed knkck-oit stgatzgizs [@konukjglu2013heeture], which removv the most relevant features ratier than ramjving the non-relevant ones, csn also solve tfis issue.

One of the biggsst advamtages of the proposed method ix that it is straightforeard to untegdate in iito a wrapping algorothm, sbch as the foewarf or backward festmre selectlon, pr a bootstrapping scheme such as Wtabilutj Selection [@meinsiausen2010stabilltn]. The kroposed method determinss ih a prkncipled way s threshold on
non-relevant features even when there is a structure the features. when there are is to limit the positive rates on order of the desired rate while the false negative rates low. Discussions =========== Experimental analysis demonstrated that the proposed is able to provide thresholds that can limit the false positive rates on order the level, in the presence of relevant features, and keep false negative rates low for high-dimensional problems. How get rid of the remaining false positives and upon the false negatives the topics of this discussion The source of false rates the desired level spurious statistics that violate the Assumption \[assumption\]. This is actually the case for any sort of feature method. As of the gets in words as the increases, the risk of having stronger between the label and some non-relevant features also The remedy tackle this issue is subsampling of samples either with bagging or bootstrapping. In our we have set the bagging ratio to $1/2$. Reducing this ratio even further improves on of spurious statistics. However, most problems there not samples have bagging ratio than $1/2$. An alternative would be to use a wrapping algorithm on bootstrapping such as the Stability Selection [@meinshausen2010stability]. The most of negative rates is competition between relevant features. relevant will manifest a different with label. frequencies be When a relevant feature lower on the ranking of features based on empirical making it above the threshold will decrease. Wrapper such as forward or backward selection methods, with bootstrapping schemes can remedy this issue. Alternatively, the recently proposed knock-out [@konukoglu2013feature], which most relevant features rather than removing the non-relevant can also solve this One of the biggest advantages of the proposed method that is straightforward integrate it into wrapping algorithm, such the forward or selection, or bootstrapping such [@meinshausen2010stability]. The proposed method determines in principled a threshold on
 non-relevant features even whEn there is a ComplEx cOrrElAtioN strUcture between tHE feaTures. Especially when theRe are AcTUallY ReLevanT featurES, tHE ThrEsHoLd iS aBLe To limIt tHe false Positive raTes On The order of thE DeSired rate wHilE keeping the fAlsE negatIvE raTEs low.

disCussiOns
===========

ExpERimentAl analysiS dEMonstrATed that THE pRopoSed method is able to PRoVIde thresholds tHat can LiMIt THE faLse Positive raTeS on thE Order of THe DESIreD Level, especialLy in the presENce Of releVaNt fEAtures, And keEp FAlsE negative raTes lOw for high-DimensIOnal proBLems. How To get rId oF thE remAInInG faLsE PosITiVes ANd iMprove upOn ThE falsE negATIVEs arE thE topIcs of This discussioN seCtioN.

the Main sOurce Of hiGhEr falSe posiTive rAtEs than the desireD levEl is spuriOus StAtiStIcs thAT violaTe tHe ASsumptiOn \[assumPTioN\]. THIS Is Actually the case for AnY SOrT of featuRe seleCTiOn MEthod. As tHe DimEnsiON Of the ProbLEm Gets highEr, in otHEr WoRds as thE rAtio $F/S$ InCreAseS, the rISk of Having Stronger SpuriOUs statistical rELationships beTWeEN ThE LabeL anD some non-relEvanT FeatUres ALsO inCReaseS. The rEmEDy TO tackle this issue is sUbSampliNg of tHe samples eithEr with baggING Or bootstRappINg. iN our experimentS we haVe set the baGGing ratiO to $1/2$. ReDucing thIs ratio evEN Further iMprOveS on The PRObLem of spurious STAtisTiCs. HowevEr, iN most prOblEms TheRe aRe Not enough Samples tO hAvE a BaGgiNg ratIO lower thAn $1/2$. an aLtErnAtive WOuld be To use A wraPpInG AlgOrithm bASeD ON booTsTrAppiNg sUcH as thE StaBIliTy SelecTion [@meinsHauSEn2010stAbIlIty].

The mOst prominent cAuSe of false nEgAtiVe rateS IS the compEtition between relevant fEAtures. EAch RelevAnt fEature wilL maNifest A diFFerent EmpiriCal coRrElaTIOn witH THe LabEl. hence, their SELecTion fReQuenCies wilL be different. When a rELevAnt feature ranKs lOwer ON ThE raNKiNG of ReLEvaNT Features based on Empirical cOrRElAtions, the cHAncEs Of it makIng it abOve thE ThreshoLd will decRease. WrapPeR algORIthMs, such as foRward or bAckward seLEctioN MeThods, ComBined wItH boOtstrApping SCheMes caN remedY tHis issUe. AltErNatively, The recently proposed knocK-out stRategIes [@Konukoglu2013FeaTUre], Which remoVe thE most relevAnt FeaTures RatHEr thaN remOViNg tHE non-rElevANt ones, can ALsO soLVE tHis issue.

One OF THe bIggesT adVAntageS of tHe proposed method iS That it is straigHtfoRWArd To iNTegrAtE it into a wrappiNg aLgORIthm, such As The forward oR backwarD fEAture SelectIon, or a BootstrAPPiNG schemE sucH as stability selEcTIon [@meinShAuSEn2010stabIlitY]. THe propOsed meTHod dETErmines in a princiPled wAY A threSHolD on
 non-relevant features eve n when the re is  acom pl ex c orre lation structu r e be tween the features. Es pecia ll y  whe n  t hereare act u al l y  re le va ntfe a tu res,the  thresh old is abl e t olimit the fa l se  positiverat es on the or der  of th edes i red r ate  whil e keep i ng the  false ne ga t ive ra t es low. Di scus sions
=========== 
E x perimental ana lysisde m on s t rat edthat the p ro posed methodi sa b l e t o  provide thre sholds that can  limit  t hef alse p ositi ve rat es on the o rder  of the d esired level,e special ly inthe  pr esen c eof  re le v ant fe atu r es,  and kee pfa lse n egat i v e rate s l ow f or hi gh-dimensiona l p robl e ms.  Howto ge t ri dof th e rema ining  f alse positivesandimprove u pon  t hefa lse n e gative s a rethe top ics oft his  d i s c us sion section.

The  m a i nsource o f high e rfa l se posit iv e r ates t han t he d e si red leve l is s p ur io us stat is tics t ha t v iol ate t h e As sumpti on \[ass umpti o n\]. This is a c tually the ca s ef o ra ny s ort  of feature  sel e ctio n me t ho d.A s the  dime ns i on of the problem gets  h igher,  in o ther words as  the ratio $ F /S$ incr ease s ,t he risk of hav ing s tronger sp u rious st atist ical rel ationship s betweenthe  la bel  an d so me non-releva n t  fea tu res als o i ncrease s.The  re med yto tackle  this is su eis  s ubs ampli n g of the  s amp le s e ither with b aggin g or  b oo t str apping. In o ur e xp er imen tswe  have  set the  baggin g ratio t o $ 1 /2$.  R ed ucing t his ratio eve nfurther im pr ove s on t h e  problem  of spurious statistics .  Howeve r,in mo st p roblems t her e arenot enough  sampl es to  h ave a  bagg i n grat io  lower tha n $1/ 2$. A nalte rnative  would be to use a wra pping algorit hmbase d on  bo o ts t rap pi n g s u c h as the Stabil ity Select io n  [ @meinshaus e n20 10 stabili ty].

T he mo s t promi nent caus e of fals enega t i verates is t he compe tition be t weenr el evant  fe atures .Eac h rel evantf eat ure w ill ma ni fest a  diff er ent empi rical correlation withthe la bel.Hen ce, their  se l ect ion frequ enci es will be  di ffe rent.  Wh e n a r elev a nt  fe a turerank s  lower on th e r a n ki ng of relev a n t  fe ature s b a sed on  emp irical correlatio n s, the chances  ofi t  ma kin g  itab ove the thresh old  w i l l decrea se . Wrapper a lgorithm s, suchas for ward o r backw a r ds electi on m eth ods, comb ine dw ith boo ts tr a ppingsche me s canremedy this i ssue. Alternativ ely,t h e rec e ntl y pro po sed kno c k-ou t strategi es [@konuko glu201 3fea ture] , which  r emovethe  m ost releva n t feature s rat her tha nremo vin g thenon- r e levan t on es , c an also s o l ve th is iss ue.
Oneof  the  biggesta dvantage s o f  the pr op ose d method is t hat it isstr aight f o rward to i n tegr a te it in to a w rappin g algor i thm ,such as  th e forward o r backwar d  fe at ureselectio n,  oraboo ts trapping scheme  such as  Stabili t y  Selec tio n  [@m e inshau se n2010stabili ty].Th e  propos e d  me thod de t erm ine s i n apri ncip l ed way a threshold on
 non-relevant_features even_when there is a_complex correlation_structure_between the_features._Especially when there_are actually relevant_features, the threshold is_able to limit_the_false positive rates on the order of the desired rate while keeping the false_negative_rates low.

Discussions
===========

Experimental_analysis_demonstrated_that the proposed method is_able to provide thresholds that_can limit_the false positive rates on the order of_the_desired level, especially_in the presence of relevant features, and keep false_negative rates low for high-dimensional problems._How to get_rid_of_the remaining false positives_and improve upon the false negatives_are the topics of this discussion_section.

The main source of higher false positive_rates than the desired level is_spurious statistics that violate the_Assumption \[assumption\]. This_is actually the case for_any sort of_feature selection_method. As the_dimension of the problem gets higher,_in other words_as the ratio $F/S$ increases, the_risk_of having stronger_spurious_statistical_relationships between_the label and_some_non-relevant features_also_increases. The remedy to tackle this_issue_is subsampling of the samples either with_bagging or bootstrapping. In_our_experiments we have set_the bagging ratio to $1/2$._Reducing this ratio even further improves_on the_problem of_spurious statistics. However, in most problems there are not enough samples_to have a bagging ratio lower_than $1/2$. An alternative_would be_to_use a wrapping_algorithm_based on_bootstrapping such as the Stability Selection [@meinshausen2010stability].

The most_prominent cause_of false negative rates is the_competition between relevant features._Each_relevant feature will manifest a different_empirical correlation with the label. Hence,_their selection frequencies will be_different._When_a relevant feature ranks lower_on the ranking of relevant features_based on empirical_correlations, the chances of it making it_above_the threshold will decrease. Wrapper algorithms,_such_as forward or backward selection methods,_combined_with_bootstrapping schemes can remedy this_issue. Alternatively, the recently proposed knock-out_strategies [@konukoglu2013feature], which remove the most relevant features rather than_removing the non-relevant_ones, can also solve this_issue.

One_of_the biggest advantages of the proposed method is that it_is straightforward_to integrate it_into a wrapping algorithm, such as the forward or backward_feature selection, or a bootstrapping scheme such_as Stability Selection [@meinshausen2010stability]. The proposed method determines in a principled way_a threshold on