ae are also given that allow one to characterize the sensitivity of a planned experiment through the median significance of a given hypothesis under assumption of a different one, e.g., median significance with which one would reject the background-only hypothesis under assumption of a certain signal model. These exploit the use of an artificial data set, the “Asimov” data set, defined so as to make estimators for all parameters equal to their true values. Methods for finding the expected statistical variation in the sensitivity (error bands) are also given.

These tools free one from the need to carry out lengthy Monte Carlo calculations, which in the case of a discovery at $5 \sigma$ significance could require simulation of around $10^8$ measurements. They are are particularly useful in cases where one needs to estimate experimental sensitivities for many points in a multidimensional parameter space (e.g., for models such as supersymmetry), which would require generating a large MC sample for each point.

The approximations used are valid in the limit of a large data sample. Tests with Monte Carlo indicate, however, that the formulae are in fact reasonably accurate even for fairly small samples, and thus can have a wide range of practical applicability. For very small samples and in cases where high accuracy is crucial, one is always free to validate the approximations with Monte Carlo.

Acknowledgements {#acknowledgements.unnumbered}
================

The authors would like to thank Louis Fayard, Nancy Andari, Francesco Polci and Marumi Kado for fruitful discussions. We received useful feedback at the Banff International Research Station, specifically from Richard Lockhart and Earl Lawrence. One of us (E.G.) is obliged to the Benoziyo Center for High Energy Physics, to the the Israeli Science Foundation(ISF), the Minerva Gesellschaft and the German Israeli Foundation (GIF) for supporting this work. K.C. is supported by US National Science Foundation grant PHY-0854724. G.C. thanks the U.K. Science and Technology Facilities Council as well as the Einstein Center at the Weizmann Institute of Science, where part of his work on this paper was done.

[\[88\]]{}

S.S. Wilks, [*The large-sample distribution of the likelihood ratio for testing composite hypotheses*]{}, Ann. Math.Statist. [**9**]{} (1938) 60-2.

A. Wald, [*Tests of Statistical Hyp
ae are also given that allow one to characterize the sensitivity of a plan experiment through the medial significance of a given guess under assumption of a unlike one, e.g., median meaning with which one would reject the background - only guess under premise of a certain signal mannequin. These exploit the use of an artificial datum set, the “ Asimov ” data set, defined so as to make estimators for all parameters adequate to their true values. Methods for detect the expected statistical variation in the sensitivity (mistake bands) are besides given. 

 These cock free one from the need to carry out drawn-out Monte Carlo calculations, which in the case of a discovery at $ 5 \sigma$ significance could require simulation of around $ 10 ^ 8 $ measurement. They are are peculiarly utilitarian in cases where one needs to estimate experimental sensitivities for many points in a multidimensional parameter space (for example, for models such as supersymmetry), which would command generating a large MC sample for each item. 

 The approximations used are valid in the limit of a big data sample. Tests with Monte Carlo indicate, however, that the rule are in fact reasonably accurate even for fairly small samples, and thus can have a wide-eyed range of practical applicability. For very small samples and in sheath where high accuracy is all-important, one is constantly free to validate the approximations with Monte Carlo. 

 Acknowledgements { # acknowledgements.unnumbered } 
 = = = = = = = = = = = = = = = = 

 The authors would wish to thank Louis Fayard, Nancy Andari, Francesco Polci and Marumi Kado for fruitful discussions. We received useful feedback at the Banff International Research Station, specifically from Richard Lockhart and Earl Lawrence. One of us (E.G.) is obliged to the Benoziyo Center for High Energy Physics, to the the Israeli Science Foundation(ISF), the Minerva Gesellschaft and the German Israeli Foundation (GIF) for supporting this work. K.C. is supported by US National Science Foundation grant PHY-0854724. G.C.   thanks the U.K.   Science and Technology Facilities Council as well as the Einstein Center at the Weizmann Institute of Science, where share of his work on this newspaper was done. 

 [ \[88\ ] ] { } 

 S.S.   Wilks, [ * The large - sample distribution distribution of the likelihood ratio for testing composite hypotheses * ] { }, Ann.   Math. Statist.   [ * * 9 * * ] { } (1938) 60 - 2. 

 A.   Wald, [ * Tests of Statistical Hyp
ae wre also given that allod one to characjeeize tie sensjtivity uf a planned experiment throngh rhe mtbian significance of a given jypothesus uider assumption of a difnzrent lne, z.g., median signiflcance with which one wound rzject the background-only hypothesis tnder axskmption of a cgrtaim sighal model. These exploit the use of an artpficial data set, yhe “Asimov” data set, definef so as to make estimahors for alo pawqmeters equau to their true values. Methods for finding the expectdd stctistical vqruatlmn in the sxnsitinity (error bakcs) are also goven.

These toolx fcee ine from the need to rarry out lengthy Mogte Carlo cclculations, which in rhw casg of d dirxovdry av $5 \aigma$ digiificance ckuld requirw simulation of aroinq $10^8$ measurementa. They awe are particularly useful in cases whege ohe needs to estimate experimental sensitivitles for mwny points in a multidimensional parameter space (a.g., foc oodtlw sucf ad supersymmetry), which would require generatind a lsrge MC sample for each ppijt.

Yre approximatkons used are valid in the pimit os a lqrge data samlle. Tests with Monte Carlo undicate, howvver, that the formulae are in facc reaspnablu accurate even for faixly smzll samples, and thus zan have a wide fanne mf practical applicability. For very smakl sampues snd in cases whege hinv accuracy is cruclal, oue is always frfe to validate the approximatioix with Monte Cdrlm.

Acknowlzdgemekts {#acknowledgeients.unnumbereb}
================

The autkors wuuld like no thank Nouis Fayarq, Nancy Andarh, Francesco Pmlci and Marymi Jado fof fruitful discissions. Wv xeceived yseful feedback at thg Ganff Internatiinao Research Statoon, spqcpfirally xrom Richard Locyhafy and Earl Lawrekce. One of us (E.G.) is obliged to fhe Benoziyo Centet nor High Gnergy Phrsics, to the yhe Israeli Scienct Founvation(MSF), thr Mynerva Gesellschaft and the Gedman Israfli Foundation (DIF) nor fupporting this work. K.C. is supported by US Nationel Science Foundation geant PHY-0854724. G.C. thanks tke U.K. Science end Teshnology Xacilities Council aw well as the Einxtein Center at the Wsizmant Inshitute of Science, where part of his work on this paper was done.

[\[88\]]{}

S.S. Wilks, [*Thw largx-swmple distrjbutoon ox che likelyhoov catio for testing composite hypotheses*]{}, Ann. Math.Stetist. [**9**]{} (1938) 60-2.

A. Wand, [*Tests of Statistical Hyp
ae are also given that allow one the of a experiment through the hypothesis assumption of a one, e.g., median with which one would reject the hypothesis under assumption of a certain signal model. These exploit the use of artificial data set, the “Asimov” data set, defined so as to make estimators all equal their values. Methods for finding the expected statistical variation in the sensitivity (error bands) are also given. tools free one from the need to carry lengthy Monte Carlo calculations, in the case of a at \sigma$ significance require of $10^8$ measurements. They are particularly useful in cases where one needs to estimate experimental sensitivities for many points in a parameter space models such supersymmetry), would generating a large for each point. The approximations used the limit of a large data sample. Tests Monte Carlo however, that the formulae are in reasonably accurate even for fairly small samples, and can have a wide range of practical applicability. For very small samples and in cases accuracy is crucial, one always free to the with Carlo. {#acknowledgements.unnumbered} ================ authors would like to thank Louis Fayard, Nancy Andari, Francesco Polci Marumi Kado for fruitful discussions. We received useful feedback at International Station, specifically from Lockhart and Earl Lawrence. of (E.G.) is obliged to Center High the Israeli Foundation(ISF), the Minerva Gesellschaft the German Israeli Foundation (GIF) supporting this work. K.C. Science Foundation grant PHY-0854724. G.C. thanks the U.K. and Technology Facilities Council as well as Einstein Center at the Weizmann Institute of Science, where part of his on this done. [\[88\]]{} S.S. Wilks, [*The large-sample distribution of likelihood ratio for testing hypotheses*]{}, Ann. Math.Statist. [**9**]{} (1938) 60-2. A. Wald, [*Tests Statistical
ae are also given that allow onE to charactErize The SenSiTiviTy of A planned experiMEnt tHrough the median signifiCance Of A GiveN HyPotheSis undeR AsSUMptIoN oF a dIfFErEnt onE, e.g., Median sIgnificancE wiTh Which one woulD ReJect the bacKgrOund-only hypoTheSis undEr AssUMptioN of A certAin sigNAl modeL. These expLoIT the usE Of an artIFIcIal dAta set, the “Asimov” daTA sET, defined so as to Make esTiMAtORS foR alL parameterS eQual tO Their trUE vALUEs. METhods for findiNg the expectED stAtistiCaL vaRIation In the SeNSitIvity (error bAnds) Are also giVen.

TheSE tools fREe one frOm the nEed To cArry OUt LeNgtHy mOntE caRlo CAlcUlations, WhIcH in thE casE OF A DiscOveRy at $5 \Sigma$ Significance cOulD reqUIre SimulAtion Of arOuNd $10^8$ meaSuremeNts. ThEy Are are particulaRly uSeful in caSes WhEre OnE needS To estiMatE exPerimenTal sensITivItIES FoR many points in a multIdIMEnSional paRameteR SpAcE (E.g., for modElS suCh as SUPersyMmetRY), wHich woulD requiRE gEnErating A lArge MC SaMplE foR each POint.

the appRoximatiOns usED are valid in the LImit of a large dATa SAMpLE. TesTs wIth Monte CarLo inDIcatE, howEVeR, thAT the fOrmulAe ARe IN fact reasonably accuRaTe even For faIrly small sampLes, and thus CAN Have a widE ranGE oF Practical appliCabilIty. For very SMall sampLes anD in cases Where high ACCuracy is CruCiaL, onE is ALWaYs free to validATE the ApProximaTioNs with MOntE CaRlo.

ackNoWledgemenTs {#acknowLeDgEmEnTs.uNnumbERed}
================

The auThOrs WoUld Like tO Thank LOuis FAyarD, NAnCY AnDari, FraNCeSCO PolCi AnD MarUmi kaDo for FruiTFul DiscussIons. We recEivED useFuL fEedback At the Banff IntErNational ReSeArcH StatiON, SpecificAlly from Richard Lockhart ANd Earl LAwrEnce. ONe of Us (E.G.) is oblIgeD to the benOZiyo CeNter foR High enErgY pHysicS, TO tHe tHe israeli SciENCe FOundaTiOn(ISf), the MinErva Gesellschaft anD The german Israeli fouNdatION (GiF) fOR sUPpoRtINg tHIS work. K.C. is supporTed by US NatIoNAl science FouNDatIoN grant PhY-0854724. G.C. thaNks thE u.K. ScienCe and TechNology FacIlItieS cOunCil as well aS the EinsTein CenteR At the wEiZmann insTitute Of sciEnce, wHere paRT of His woRk on thIs Paper wAs donE.

[\[88\]]{}

S.s. Wilks, [*ThE large-sample distributioN of the LikelIhoOd ratio foR teSTinG compositE hypOtheses*]{}, Ann. matH.StAtist. [**9**]{} (1938) 60-2.

a. WaLD, [*TestS of STAtIstICal HyP
ae are also given that all ow one tochara cte riz ethesens itivity of a p l anne d experiment through t he me di a n si g ni fican ce of a gi v e n h yp ot hes is un der a ssu mptionof a diffe ren tone, e.g., m e di an signifi can ce with whic h o ne wou ld  re j ect t hebackg round- o nly hy pothesisun d er ass u mptiono f  a  cer tain signal model .  T h ese exploit th e useof an a rti fic ial data s et , the “Asimov ”  d a t a  se t , defined soas to makee sti mators  f ora ll par amete rs equ al to their  tru e values.  Metho d s for f i nding t he exp ect edstat i st ic alva r iat i on  in the  sensiti vi ty  (err or b a n d s ) ar e a lsogiven .

These tool s f reeo nefromthe n eedto  carr y outlengt hy  Monte Carlo ca lcul ations, w hic hinth e cas e  of adis cov ery at$5 \sig m a$si g n i fi cance could requir es i mu lation o f arou n d$1 0 ^8$ meas ur eme nts. T hey a re a r eparticul arly u s ef ul  in cas es  where  o nenee ds to esti mate e xperimen tal s e nsitivities fo r  many pointsi na mu l tidi men sional para mete r  spa ce ( e .g .,f or mo delssu c ha s supersymmetry), w hi ch wou ld re quire generat ing a larg e M C sample  for ea c h point.

Theappro ximationsu sed arevalid  in thelimit ofa large da tasam ple . T e s ts  with Monte C a r lo i nd icate,how ever, t hat  th e f orm ul ae are in  fact re as on ab ly  ac curat e  even fo rfai rl y s malls amples , and  thu sca n  ha ve a wi d er a ngeof  p ract ica lappli cabi l ity . For v ery small  sa m ples  a nd  in cas es where high  a ccuracy is  c ruc ial, o n e  is alwa ys free to validate the approxi mat ionswith  Monte Ca rlo .

Ack now l edgeme nts {# ackno wl edg e m ents. u n nu mbe re d}
======= = = === ====
T he a uthorswould like to than k  Lo uis Fayard, N anc y An d a ri , F r an c esc oP olc i and Marumi Kado  for fruit fu l  d iscussions .  We  r eceived  useful  feed b ack atthe Banff  Internat io nalR e sea rch Statio n, speci fically f r om Ri c ha rd Lo ckh art an dEar l Law rence. One  of u s (E.G .)  is ob liged  t o the Be noziyo Center for HighEnergy  Phys ics , to thethe Isr aeli Scie nceFoundation (IS F),  theMin e rva G esel l sc haf t  andtheG erman Isr a el i F o u nd ation (GIF) f o r s uppor tin g  thiswork . K.C. is support e d by US Nation al S c i enc e F o unda ti on grant PHY-0 854 72 4 .  G.C. th an ks the U.K.  Science  a n d Tec hnolog y Faci litiesC o un c il aswell  as  the Eins tei nC enter a tth e  Weizm annIn stitut e of S c ienc e ,  where part of h is wo r k  on t h ispaper  w as done . 

[\ [88\]]{}
S.S. Wilks,  [*The  lar ge-sa mple di st ributi onof  the likel i hood rati o for  testin gcomp osi te hyp othe s e s*]{} , An n.  Ma th.Statis t .  [ * *9 ** ] {}(193 8) 60 -2 .

A . Wald, [ * Tests of  St a tistica lHyp 
ae are_also given_that allow one to_characterize the_sensitivity_of a_planned_experiment through the_median significance of_a given hypothesis under_assumption of a_different_one, e.g., median significance with which one would reject the background-only hypothesis under assumption_of_a certain_signal_model._These exploit the use of_an artificial data set, the_“Asimov” data_set, defined so as to make estimators for_all_parameters equal to_their true values. Methods for finding the expected statistical_variation in the sensitivity (error bands)_are also given.

These_tools_free_one from the need_to carry out lengthy Monte Carlo_calculations, which in the case of_a discovery at $5 \sigma$ significance could_require simulation of around $10^8$ measurements._They are are particularly useful_in cases_where one needs to estimate_experimental sensitivities for_many points_in a multidimensional_parameter space (e.g., for models such_as supersymmetry), which_would require generating a large MC_sample_for each point.

The_approximations_used_are valid_in the limit_of_a large_data_sample. Tests with Monte Carlo indicate,_however,_that the formulae are in fact reasonably_accurate even for fairly_small_samples, and thus can_have a wide range of_practical applicability. For very small samples_and in_cases where_high accuracy is crucial, one is always free to validate the_approximations with Monte Carlo.

Acknowledgements {#acknowledgements.unnumbered}
================

The authors_would like to thank_Louis Fayard,_Nancy_Andari, Francesco Polci_and_Marumi Kado_for fruitful discussions. We received useful feedback_at the_Banff International Research Station, specifically from_Richard Lockhart and Earl_Lawrence._One of us (E.G.) is obliged_to the Benoziyo Center for High_Energy Physics, to the the_Israeli_Science_Foundation(ISF), the Minerva Gesellschaft and_the German Israeli Foundation (GIF) for_supporting this work._K.C. is supported by US National Science_Foundation_grant PHY-0854724. G.C. thanks the U.K. Science and_Technology_Facilities Council as well as the_Einstein_Center_at the Weizmann Institute of_Science, where part of his work_on this paper was done.

[\[88\]]{}

S.S. Wilks, [*The large-sample distribution of_the likelihood ratio_for testing composite hypotheses*]{}, Ann. Math.Statist. [**9**]{}_(1938)_60-2.

A. Wald,_[*Tests of Statistical Hyp