 functions on a convex domain. In this case one can use the expectation-maximization (EM) algorithm [@Vardi] as discussed in [@Zhang]. A problem is that due to stopping criteria and numerical precision, one cannot expect to find the exact optimum. We show in App. \[sect:bad estimation\] that one can compensate for this problem to maintain validity of the computed $p$-value.

Conclusion {#sect:conclusion}
==========

The degree of violation of LR in a Bell-type test is usually expressed in terms of the number of SDs of violation. This quantity cannot, however, be used to obtain valid $p$-values for rejecting LR by conventional means. It also fails to quantitatively compare the success of different experimental tests of LR and does not account for stability issues or memory effects in experiments. We solve these problems by providing a method—the PBR protocol—for determining valid $p$-values directly from the settings and outcomes in a sequence of trials. The PBR protocol does not rely on a predetermined Bell inequality, adapts to the actual experimental configuration, and is asymptotically optimal for independent and identically distributed trials. It therefore provides a standardized measure of success for experimental tests of LR. While the protocol remains valid if the experiment drifts over the sequence of trials, how well it performs depends on the nature of the drifts and how the protocol takes them into account. Another valid protocol that accounts for memory effects can be based on martingale bounds [@Gill1; @Gill2]. This protocol requires a Bell inequality that is fixed for the experiment. Given the Bell inequality, the martingale-based protocol has the advantage that it is computationally efficient with respect to number of settings, outcomes, and parties. The disadvantage is that it is suboptimal and does not provide a clear quantitative comparison of different experimental tests. Our simulations show that it is practical to apply the PBR protocol to data from typical experimental configurations, and that the running $p$-values can be used for tweaking an experiment in progress to find the experimentally accessible configuration that provides the highest violation of LR.

We thank D. N. Matsukevich for providing the experimental data for Ref. [@Monroe], and we are grateful for the editorial support of A. Migdall and K. Coakley. This paper is a contribution of the National Institute of Standards and Technology and is not subject to U.S. copyright.

 {#sect:appendix}

Statistical Concept
functions on a convex domain. In this case one can use the anticipation - maximization (EM) algorithm   [ @Vardi ] as discourse in   [ @Zhang ]. A problem is that due to stopping criteria and numeral precision, one cannot expect to discover the exact optimum. We show in App.   \[sect: regretful estimation\ ] that one can pay for this problem to sustain validity of the computed $ p$-value. 

 Conclusion { # sect: conclusion } 
 = = = = = = = = = = 

 The degree of violation of LR in a Bell - character test is usually expressed in price of the number of SDs of misdemeanor. This quantity cannot, however, be use to obtain valid $ p$-values for rejecting LR by conventional means. It also fail to quantitatively compare the success of different experimental tests of LR and does not account for stability consequence or memory effects in experiments. We solve these problems by providing a method — the PBR protocol — for determining valid $ p$-values directly from the settings and outcomes in a succession of test. The PBR protocol does not rely on a preset Bell inequality, adapts to the actual experimental shape, and is asymptotically optimal for independent and identically distributed trials. It consequently provides a exchangeable measure of success for experimental tests of LR. While the protocol remain valid if the experiment drifts over the sequence of trials, how well it performs depends on the nature of the drifts and how the protocol takes them into account. Another valid protocol that accounts for memory effects can be based on dolphin striker bounds   [ @Gill1; @Gill2 ]. This protocol want a Bell inequality that is fix for the experiment. Given the Bell inequality, the martingale - based protocol have the advantage that it is computationally efficient with respect to number of mise en scene, outcomes, and parties. The disadvantage is that it is suboptimal and does not provide a clear quantitative comparison of unlike experimental tests. Our simulations show that it is practical to apply the PBR protocol to datum from typical experimental configurations, and that the running $ p$-values can be used for tweak an experiment in progress to receive the experimentally accessible configuration that provides the highest violation of LR. 

 We thank D. N. Matsukevich for providing the experimental datum for Ref.   [ @Monroe ], and we are grateful for the editorial support of A. Migdall and K. Coakley. This paper is a contribution of the National Institute of Standards and Technology and is not subject to U.S. copyright. 

  { # faction: appendix } 

 Statistical Concept
 fujctions on a convex domaln. In this case one can use tge expecgation-maximization (EM) algorivhm [@Vqrdi] qs discussed in [@Zhang]. A problem ps that dye ti stopping rditeria and nhlerieao precision, ong cannot expact to find tha dxcct optimum. We show in App. \[sect:bad eseimatiom\] hhat one can cjmpemfate for this problem to maintain valisity of the computed $p$-value.

Conclusion {#sect:concpusiln}
==========

The degree of villation of OR ig a Bell-type gest is uslclly expresaed in terms of the number of SAs of violation. Tyis xuantity cainot, hjwever, be uscc to ottain vslid $p$-values fpr cejexting LR by conventioial means. It also fayls to qudncitatively compare thw wuccevs ox diwderdnt eepedimentwl vests of LR and does nit account for stabolynu issues or jemory esfects in experiments. We solve these prmblsms by providing a methid—the PBR protocol—for determinyng valid $p$-values directly from the settings and mutcojds nk x sfquence of trials. The PBR protocol does not rqmy ok a predetermineb Bell inequalitu, wdskts to the actoal expzdijental configuratiln, and ys astmptoticajly pptimal for independent and identically eistributed trials. It therefoxe profides a standardized measure of auccess for experimehgal tests of LR. Dhike the prouucol remains valiq if the xxpernment drkfts over ehe sequenfe of trials, how well it perflrks depends on the nature of the drifts anv how the projocml nakes thei intp account. Anoeher valid projocol thac accojnts for mvmory effxcts can be fased on marthjgale bounds [@Jill1; @Gill2]. Thiw pritocol fdquires a Bell inequaliny thar is fixed for the exkediment. Given thz Vell inequality, thd mwrninjale-bwved protocol has thd advavtage that lt ks cpmputationally effichent with respect to nimner of sejtings, ouecomes, and patties. The disadvanuage iv tiat it is suboptimal and does not provise a cleag qmantitative cjmpavisog of diffexent experimental tests. Our simulations whow that it is ptactical to apply the KBR protocol vo datw from ty[ical experimental cinfigurations, and that the running $p$-vamues cdn be used for tweaking an experiment in progress to find the experimentally axcessiulq configurafion that pxovndes thq hijhxst violation of KR.

We thank D. N. Matsukevich for 'roviding dhz experimental data for Ref. [@Momrue], and we are grateful for the editorjal suppprt of A. Migdall and K. Coakley. Yhis paper is a contribubion of tye Nafional Insvitute of Standards snd Teehnoloty and us nlt subject to U.S. clpyright.

 {#dect:eppzndix}

Statistical Concept
functions on a convex domain. In this can the expectation-maximization algorithm [@Vardi] as is due to stopping and numerical precision, cannot expect to find the exact We show in App. \[sect:bad estimation\] that one can compensate for this problem maintain validity of the computed $p$-value. Conclusion {#sect:conclusion} ========== The degree of violation LR a test usually expressed in terms of the number of SDs of violation. This quantity cannot, however, be to obtain valid $p$-values for rejecting LR by means. It also fails quantitatively compare the success of experimental of LR does account stability issues or effects in experiments. We solve these problems by providing a method—the PBR protocol—for determining valid $p$-values directly the settings in a of The protocol does not a predetermined Bell inequality, adapts to configuration, and is asymptotically optimal for independent and distributed trials. therefore provides a standardized measure of for experimental tests of LR. While the protocol valid if the experiment drifts over the sequence of trials, how well it performs depends nature of the drifts how the protocol them account. valid that accounts memory effects can be based on martingale bounds [@Gill1; @Gill2]. This requires a Bell inequality that is fixed for the experiment. Bell the martingale-based protocol the advantage that it computationally with respect to number outcomes, parties. that is and does not provide clear quantitative comparison of different tests. Our simulations show apply the PBR protocol to data from typical configurations, and that the running $p$-values can used for tweaking an experiment in progress to find the experimentally accessible that provides violation of LR. We thank D. N. Matsukevich providing the experimental data Ref. [@Monroe], and we are grateful for the editorial of Migdall and Coakley. This paper a contribution of National Institute of Technology and not to Statistical Concept
 functions on a convex domain. IN this case oNe can Use The ExPectAtioN-maximization (Em) AlgoRithm [@Vardi] as discussed iN [@ZhanG]. A PRoblEM iS that Due to stOPpING crItErIa aNd NUmEricaL prEcision, One cannot eXpeCt To find the exaCT oPtimum. We shOw iN App. \[sect:bad eStiMation\] ThAt oNE can cOmpEnsatE for thIS problEm to maintAiN ValidiTY of the cOMPuTed $p$-Value.

Conclusion {#seCT:cONclusion}
==========

The degRee of vIoLAtION of lR iN a Bell-type TeSt is uSUally exPReSSED in TErms of the numbEr of SDs of viOLatIon. ThiS qUanTIty canNot, hoWeVEr, bE used to obtaIn vaLid $p$-valueS for reJEcting Lr By conveNtionaL meAns. it alSO fAiLs tO qUAntITaTivELy cOmpare thE sUcCess oF difFERENt exPerImenTal teSts of LR and doeS noT accOUnt For stAbiliTy isSuEs or mEmory eFfectS iN experiments. We sOlve These probLemS bY prOvIding A Method—The pBR ProtocoL—for detERmiNiNG VAlId $p$-values directly fRoM THe Settings And outCOmEs IN a sequenCe Of tRialS. tHe PBR ProtOCoL does not Rely on A PrEdEterminEd bell inEqUalIty, AdaptS To thE actuaL experimEntal COnfiguration, anD Is asymptoticaLLy OPTiMAl foR inDependent anD ideNTicaLly dIStRibUTed trIals. IT tHErEFore provides a standaRdIzed meAsure Of success for eXperimentaL TESts of LR. WHile THe PRotocol remains Valid If the experIMent drifTs oveR the sequEnce of triALS, how well It pErfOrmS dePENdS on the nature oF THe drIfTs and hoW thE protocOl tAkeS thEm iNtO account. ANother vaLiD pRoToCol That aCCounts foR mEmoRy EffEcts cAN be basEd on mArtiNgAlE BouNds [@Gill1; @gIlL2]. tHis pRoToCol rEquIrEs a BeLl inEQuaLity thaT is fixed fOr tHE expErImEnt. GiveN the Bell inequAlIty, the martInGalE-based PROtocol haS the advantage that it is coMPutatioNalLy effIcieNt with resPecT to numBer OF settiNgs, outComes, AnD paRTIes. ThE DIsAdvAnTage is that IT Is sUboptImAl anD does noT provide a clear quanTItaTive comparisoN of DiffEREnT exPErIMenTaL TesTS. our simulations sHow that it iS pRAcTical to appLY thE PbR protoCol to daTa froM Typical ExperimenTal configUrAtioNS, And That the runNing $p$-valUes can be uSEd for TWeAking An eXperimEnT in ProgrEss to fINd tHe expErimenTaLly accEssibLe ConfigurAtion that provides the higHest viOlatiOn oF LR.

We thanK D. N. mAtsUkevich foR proViding the eXpeRimEntal DatA For ReF. [@MonROe], And WE are gRateFUl for the eDItOriAL SuPport of A. MigDALL anD K. CoaKleY. this paPer iS a contribution of tHE National InstiTute OF staNdaRDs anD TEchnology and is Not SuBJEct to U.S. cOpYright.

 {#sect:aPpendix}

STaTIsticAl ConcEpt
 functions on a convex dom ain. In th is ca seone  c an u se t he expectation - maxi mization (EM) algorith m [@V ar d i] a s  d iscus sed in[ @Z h a ng] .Apro bl e mis th atdue tostopping c rit er ia and numer i ca l precisio n,one cannot e xpe ct tofi ndt he ex act  opti mum. W e  showin App. \ [s e ct:bad estimat i o n\ ] th at one can compen s at e  for this prob lem to  m a in t a inval idity of t he  comp u ted $p$ - va l u e .
C onclusion {#s ect:conclus i on} 
===== == === 
The d egree  o f  vi olation ofLR i n a Bell- type t e st is u s ually e xpress edinterm s  o fthe  n u mbe r  o f S D s o f violat io n.  This  qua n t i t y ca nno t, h oweve r, be used to  ob tain val id $p $-val uesfo r rej ecting  LR b yconventional me ans.  It alsofai ls  to  q uanti t ativel y c omp are the  succes s  of  d i f f er ent experimental t es t s  o f LR and  doesn ot  a c count fo rsta bili t y  issu es o r  m emory ef fectsi nex perimen ts . We s ol vethe se pr o blem s by p roviding  a me t hod—the PBR pr o tocol—for det e rm i n in g  val id$p$-valuesdire c tlyfrom th e s e tting s and  o u tc o mes in a sequence o ftrials . The  PBR protocol  does notr e l y on a p rede t er m ined Bell ineq ualit y, adaptst o the ac tualexperime ntal conf i g uration,  an d i s a sym p t ot ically optima l forin depende ntand ide nti cal lydis tr ibuted tr ials. It  t he re fo reprovi d es a sta nd ard iz edmeasu r e of s ucces s fo rex p eri mentalt es t s  ofLR .Whil e t he  prot ocol rem ains va lid if th e e x peri me nt  drifts  over the seq ue nce of tri al s,how we l l  it perf orms depends on the nat u re of t hedrift s an d how the  pr otocol  ta k es the m into  acco un t.A n other v al idpr otocol tha t acc ounts  f or m emory e ffects can be base d  on  martingale b oun ds [ @ G il l1; @G i ll2 ]. Thi s protocol requir es a Bellin e qu ality that isfi xed for  the ex perim e nt. Giv en the Be ll inequa li ty,t h e m artingale- based pr otocol ha s  thea dv antag e t hat it  i s c omput ationa l lyeffic ient w it h resp ect t onumber o f settings, outcomes, a nd par ties.  Th e disadva nta g e i s that it  issuboptimal  an d d oes n otp rovid e ac le arq uanti tati v e compari s on  of d if ferent expe r i m ent al te sts .  Our s imul ations show thati t is practical  toa p ply  th e  PBR  p rotocol to dat a f ro m typicalex perimentalconfigur at i ons,and th at the  runnin g $p $ -value s ca n b e used fo r t we a king an  e xp e riment  inpr ogress  to fi n d th e experimentally a ccess i b le co n fig urati on  that p r ovid es the hig hest violat ion of  LR. 

Wethank D .N. Mat suk ev ich for pr o viding th e exp eriment al  dat a f or Ref . [@ M o nroe] , an dweare grate f u lf or  t h e e dito rialsu ppor t of A. M i gdall an d K .  Coakle y.  Th i s  paper is a  contribut ion  of t h e  NationalI nsti t ut e  of S tandar ds and  Techno l ogy  a nd is n ots u bject toU.S. copy r igh t. 

 { #sect:ap pe ndix }
St at istical Concept 
 functions_on a_convex domain. In this_case one_can_use the_expectation-maximization_(EM) algorithm [@Vardi] as_discussed in [@Zhang]. A_problem is that due_to stopping criteria_and_numerical precision, one cannot expect to find the exact optimum. We show in App. \[sect:bad_estimation\]_that one_can_compensate_for this problem to maintain_validity of the computed $p$-value.

Conclusion_{#sect:conclusion}
==========

The degree_of violation of LR in a Bell-type test_is_usually expressed in_terms of the number of SDs of violation. This_quantity cannot, however, be used to_obtain valid $p$-values_for_rejecting_LR by conventional means._It also fails to quantitatively compare_the success of different experimental tests_of LR and does not account for_stability issues or memory effects in_experiments. We solve these problems_by providing_a method—the PBR protocol—for determining_valid $p$-values directly_from the_settings and outcomes_in a sequence of trials. The_PBR protocol does_not rely on a predetermined Bell_inequality,_adapts to the_actual_experimental_configuration, and_is asymptotically optimal_for_independent and_identically_distributed trials. It therefore provides a_standardized_measure of success for experimental tests of_LR. While the protocol_remains_valid if the experiment_drifts over the sequence of_trials, how well it performs depends_on the_nature of_the drifts and how the protocol takes them into account. Another_valid protocol that accounts for memory_effects can be based_on martingale_bounds [@Gill1;_@Gill2]. This protocol_requires_a Bell_inequality that is fixed for the experiment._Given the_Bell inequality, the martingale-based protocol has_the advantage that it_is_computationally efficient with respect to number_of settings, outcomes, and parties. The_disadvantage is that it is_suboptimal_and_does not provide a clear_quantitative comparison of different experimental tests._Our simulations show_that it is practical to apply the_PBR_protocol to data from typical experimental_configurations,_and that the running $p$-values can_be_used_for tweaking an experiment in_progress to find the experimentally accessible_configuration that provides the highest violation of LR.

We thank_D. N. Matsukevich_for providing the experimental data_for_Ref. [@Monroe],_and we are grateful for the editorial support of A._Migdall and_K. Coakley. This_paper is a contribution of the National Institute of Standards_and Technology and is not subject to_U.S. copyright.

 {#sect:appendix}

Statistical Concept