 violation is sufficiently significant if $nS\geq c$ for some constant $c$, simulations show that the best choice of $c$ depends on the distribution of settings and outcomes in the experiment.

The third and simplest approach is to block the data from the trials. Instead of updating the log-$p$-value after every trial, we process data $h$ trials at a time. The first block is used only for estimating the setting and outcome distribution of future trials. That is, we set $R_k(x)=1$ for $k=0,\ldots,(h-1)$. Subsequently, we have $R_{mh+k}=R_{mh}$ for $k=1,\ldots,(h-1)$ and all $m$. Note that neither the validity nor the asymptotic optimality of the calculated $p$-values requires updating the PBRs after each trial. Choosing $h$ large enough ensures that the first block’s trials have sufficient information for obtaining reasonable estimates of the distribution. An additional advantage of blocking the trials is that we avoid unnecessarily invoking the computationally costly optimizations required for updating the PBRs. We standardized the choice of block size so that if the total number of trials to be analyzed is $N$, $h$ is the maximum of $\lceil N/1000\rceil$ and $\lceil\ln(2d)d\rceil$, where $d$ is the number of possible setting and outcome combinations in a trial. The first expression ensures that we do not lose too much log-$p$-value by using the first block only for learning the setting and outcome distribution. The second one is chosen so that if $q$ is uniform, the probability that every setting and outcome combination occurs is at least $1/2$.

We conclude this section with a note on implementing the PBR protocol. For monitoring an experiment and to adapt to changes in experimental configuration, the estimated setting and outcome distributions used in the PBRs should be based on recent trials only. This can be accomplished by windowing the trials with a window large enough to have statistically significant violation of LR (if there is violation), but small enough to avoid seeing significant changes in configuration. Our implementation of the local realism analysis engine uses a computationally simpler approach based on weighting the trials with exponentially decreasing weights in time determined by a configurable half-life. This feature was not used in the comparisons in Sec. \[sect:results\].

Effects of Suboptimal Estimates and LR Models {#sect
violation is sufficiently significant if $ nS\geq c$ for some constant $ c$, simulations picture that the dear choice of $ c$ depends on the distribution of settings and consequence in the experiment. 

 The third and simplest approach is to obstruct the data from the trials. alternatively of updating the log-$p$-value after every trial, we serve data $ h$ trials at a fourth dimension. The inaugural block is used only for estimating the setting and outcome distribution of future test. That is, we set $ R_k(x)=1 $ for $ k=0,\ldots,(h-1)$. Subsequently, we have $ R_{mh+k}=R_{mh}$ for $ k=1,\ldots,(h-1)$ and all $ m$. Note that neither the robustness nor the asymptotic optimality of the calculated $ p$-values requires updating the PBRs after each trial. choose $ h$ large enough ensures that the first block ’s trial have sufficient information for obtaining reasonable estimates of the distribution. An extra advantage of stuff the trials is that we avoid unnecessarily invoking the computationally costly optimizations required for updating the PBRs. We standardized the choice of pulley size so that if the total number of test to be analyze is $ N$, $ h$ is the maximum of $ \lceil N/1000\rceil$ and $ \lceil\ln(2d)d\rceil$, where $ d$ is the number of potential setting and result combinations in a trial. The inaugural expression ensures that we do not lose too much log-$p$-value by using the first blockage only for learning the setting and outcome distribution. The second one is chosen so that if $ q$ is uniform, the probability that every mise en scene and outcome combination occurs is at least $ 1/2$. 

 We conclude this section with a note on follow through the PBR protocol. For monitoring an experiment and to adapt to changes in experimental shape, the estimated context and outcome distributions use in the PBRs should be based on late trials only. This can be accomplished by windowing the trials with a window bombastic enough to have statistically meaning violation of LR (if there is irreverence), but small enough to avoid see significant change in configuration. Our implementation of the local realism analysis engine uses a computationally simpler approach establish on weighting the trials with exponentially decrease weights in time specify by a configurable half - life. This feature was not used in the comparison in Sec.   \[sect: results\ ]. 

 Effects of Suboptimal Estimates and LR Models { # sect
 villation is sufficiently rignificant if $uW\geq c$ for skme consgant $c$, simulations show that tye bewt choice of $c$ depends on the dpstributiin oh settings and onfcomes lu the cxpernmxnt.

The third anc simplest approach is tm cllck the data from the trials. Insteaq of upcahing the log-$p$-vwlue wfted every trial, we process data $h$ trjals at a time. The forst block is used only fog eshimating the settijg and outcime qustribution uf future urnals. That ia, we set $R_k(x)=1$ for $k=0,\ldots,(h-1)$. Subseqjentlv, we have $R_{nh+j}=R_{mj}$ for $k=1,\ldots,(i-1)$ and wll $m$. Note tmst neidher thr validity nor thx astmptotic optimality oh the calculated $p$-vajues requhrzs updating the PBRs qfrer edch driau. Chuosjnj $h$ large enkugh ensurss that the first block’s trialx rqve sufficienf infowmwtion for obtaining reasonable estimatev or the distribution. An aeditional advantage ov blockind the trials is that we avoid unnecessarily invokhng tie conpmtatkinwlly costly optimizations required for updatigf uhe PBRs. We standcrdized the choive og block size su that if the total number lf triajs to be analyeed ix $N$, $h$ is the maximum of $\lceul N/1000\rceil$ anb $\lxeil\ln(2d)d\rceil$, wherz $d$ is the nbmber pf poxsible setting and outcume dombinationd in a trjxl. The first expfesximn ensurtr that we do not jose too nuch log-$p$-vauue ny usigg the firdt block only for learnijg thg setthng and ouhcome distribution. The second oix is chosen sp dhan if $q$ is unifprm, the probafility that evgry settiug and outcome ckmbinatmon occurs if at least $1/2$.

We fonclude thiv sectiog wirh a note ov implementing yhe PBR pgouocol. For nonitoring an expevimenj znd to adapt to chqnges in experikengal clnhigurwdion, the esthmatdd rrttine and outcond dixtributions used in dhe LBRs should be basrd on recebt trialf only. This csn be accomplished by wmndowiig the tryals with a window large enougg to have stwtistically fignlficwnt violatnon of LR (if there is violation), but smalo enough to avoid weeing significant ehsnges in coifigurwtion. Our implementation of tye local realism snalysis engine uses z compgtatilnally simpler approach based on weighting the trials with exponentially eecreasing weights in yime gecerkined fy a ronfigurable half-kife. This feature was not used mn the com[axisons in Sec. \[sect:results\].

Effecys of Suboptimau Estimates and LR Modela {#sect
violation is sufficiently significant if $nS\geq c$ constant simulations show the best choice distribution settings and outcomes the experiment. The and simplest approach is to block data from the trials. Instead of updating the log-$p$-value after every trial, we data $h$ trials at a time. The first block is used only for the and distribution future trials. That is, we set $R_k(x)=1$ for $k=0,\ldots,(h-1)$. Subsequently, we have $R_{mh+k}=R_{mh}$ for $k=1,\ldots,(h-1)$ and $m$. Note that neither the validity nor the optimality of the calculated requires updating the PBRs after trial. $h$ large ensures the block’s trials have information for obtaining reasonable estimates of the distribution. An additional advantage of blocking the trials is that avoid unnecessarily computationally costly required updating PBRs. We standardized of block size so that if of trials to be analyzed is $N$, $h$ the maximum $\lceil N/1000\rceil$ and $\lceil\ln(2d)d\rceil$, where $d$ the number of possible setting and outcome combinations a trial. The first expression ensures that we do not lose too much log-$p$-value by first block only for the setting and distribution. second is so that $q$ is uniform, the probability that every setting and outcome combination is at least $1/2$. We conclude this section with a implementing PBR protocol. For an experiment and to to in experimental configuration, the and distributions PBRs be on recent trials only. can be accomplished by windowing trials with a window significant violation of LR (if there is violation), small enough to avoid seeing significant changes configuration. Our implementation of the local realism analysis engine uses a computationally approach based the trials with exponentially decreasing weights in time by a configurable half-life. feature was not used in the comparisons in Sec. Effects Suboptimal Estimates LR Models {#sect
 violation is sufficiently siGnificant iF $nS\geQ c$ fOr sOmE conStanT $c$, simulations sHOw thAt the best choice of $c$ depeNds on ThE DistRIbUtion Of settiNGs AND ouTcOmEs iN tHE eXperiMenT.

The thiRd and simplEst ApProach is to blOCk The data froM thE trials. InsteAd oF updatInG thE Log-$p$-vAluE afteR every TRial, we Process daTa $H$ Trials AT a time. THE FiRst bLock is used only for EStIMating the settiNg and oUtCOmE DIstRibUtion of futUrE triaLS. That is, WE sET $r_K(x)=1$ fOR $k=0,\ldots,(h-1)$. SubseQuently, we haVE $R_{mH+k}=R_{mh}$ fOr $K=1,\ldOTs,(h-1)$ and All $m$. NOtE ThaT neither the ValiDity nor thE asympTOtic optIMality oF the caLcuLatEd $p$-vALuEs ReqUiREs uPDaTinG The pBRs afteR eAcH triaL. ChoOSING $h$ laRge EnouGh ensUres that the fiRst BlocK’S trIals hAve suFficIeNt infOrmatiOn for ObTaining reasonabLe esTimates of The DiStrIbUtion. aN additIonAl aDvantagE of blocKIng ThE TRIaLs is that we avoid unnEcESSaRily invoKing thE CoMpUTationalLy CosTly oPTImizaTionS ReQuired foR updatINg ThE PBRs. We StAndardIzEd tHe cHoice OF bloCk size So that if The toTAl number of triaLS to be analyzed IS $N$, $H$ IS tHE maxImuM of $\lceil N/1000\rcEil$ aND $\lceIl\ln(2D)D\rCeiL$, Where $D$ is thE nUMbER of possible setting aNd OutcomE combInations in a trIal. The firsT EXPression EnsuREs THat we do not lose Too muCh log-$p$-valuE By using tHe firSt block oNly for leaRNIng the seTtiNg aNd oUtcOME dIstribution. ThE SEconD oNe is choSen So that iF $q$ iS unIfoRm, tHe ProbabiliTy that evErY sEtTiNg aNd outCOme combiNaTioN oCcuRs is aT Least $1/2$.

WE concLude ThIs SEctIon with A NoTE On imPlEmEntiNg tHe pBR prOtocOL. FoR monitoRing an expEriMEnt aNd To Adapt to Changes in expeRiMental confIgUraTion, thE EStimated Setting and outcome distriBUtions uSed In the pBRs Should be bAseD on recEnt TRials oNly. ThiS can bE aCcoMPLisheD BY wIndOwIng the triaLS WitH a winDoW larGe enougH to have statisticalLY siGnificant violAtiOn of lr (If TheRE iS VioLaTIon), BUT small enough to aVoid seeing SiGNiFicant chanGEs iN cOnfigurAtion. OuR implEMentatiOn of the loCal realisM aNalySIS enGine uses a cOmputatiOnally simPLer apPRoAch baSed On weigHtIng The trIals wiTH exPonenTially DeCreasiNg weiGhTs in time Determined by a configurabLe half-Life. THis Feature waS noT UseD in the comPariSons in Sec. \[sEct:ResUlts\].

EFfeCTs of SUbopTImAl ESTimatEs anD lR Models {#sECt
 violation is sufficiently  significa nt if  $n S\g eq  c$forsome constant$ c$,simulations show thatthe b es t  cho i ce  of $ c$ depe n ds o n t he  d ist ri b ut ion o f s ettings  and outco mes  i n the experi m en t.

The th ird  and simples t a pproac hist o blo ckthe d ata fr o m thetrials. I ns t ead of updatin g th e lo g-$p$-value after ev e ry trial, we p rocess  d a ta $ h$tri als at a t im e. Th e  firstb lo c k isu sed only forestimatingt hesettin gand outcom e dis tr i but ion of futu re t rials. Th at is, we set$ R_k(x)= 1$ for  $k =0, \ldo t s, (h -1) $. Sub s eq uen t ly,  we have  $ R_ {mh+k }=R_ { m h } $ fo r $ k=1, \ldot s,(h-1)$ andall  $m$ .  No te th at ne ithe rthe v alidit y nor  t he asymptotic o ptim ality ofthe  c alc ul ated$ p$-val ues  re quiresupdatin g  th eP B R safter each trial.Ch o o si ng $h$ l arge e n ou gh ensuresth atthef i rst b lock ’ strials h ave su f fi ci ent inf or mation  f orobt ainin g  rea sonabl e estima tes o f  the distribut i on. An additi o na l ad v anta geof blocking  the tria ls i s  t hat we av oid u nn e ce s sarily invoking the  c omputa tiona lly costly op timization s r equiredforu pd a ting the PBRs.  We s tandardize d  the cho ice o f blocksize so t h a t if the  to tal  nu mbe r of  trials to be a naly ze d is $N $,$h$ isthe  ma xim umof  $\lceilN/1000\r ce il $an d $ \lcei l \ln(2d)d \r cei l$ , w here$ d$ isthe n umbe rof pos sible s e tt i n g an dou tcom e c om binat ions ina trial . The fir ste xpre ss io n ensur es that we do  n ot lose to omuc h log- $ p $-valueby using the first bloc k  only f orlearn ingthe setti ngand ou tco m e dist ributi on. T he  se c o nd on e is  ch os en so that i f $ q$ is  u nifo rm, the  probability thate ver y setting and  ou tcom e co mbi n at i onoc c urs i s at least $1/2 $.

We con cl u de  this sect i onwi th a no te on i mplem e nting t he PBR pr otocol. F or  mon i t ori ng an expe riment a nd to ada p t toc ha ngesinexperi me nta l con figura t ion , the  estim at ed set tingan d outcom e distributions used in  the P BRs s hou ld be bas edo n r ecent tri alsonly. This  ca n b e acc omp l ished  byw in dow i ng th e tr i als witha  w ind o w  l arge enough t o  ha ve st ati s ticall y si gnificant violati o n of LR (if th erei s  vi ola t ion) ,but small enou ghto a void see in g significa nt chang es in co nfigur ation.  Our im p l em e ntatio n of  th e local r eal is m  analys is  e n gine u sesacomput ationa l ly s i m pler approach ba sed o n weigh t ing  thetr ials wi t h ex ponentiall y decreasin g weig htsin ti me dete rm ined b y a  c onfigurabl e  half-lif e. Th is feat ur e wa s n ot use d in t he co mpar is ons  in Sec.\ [ se c t: re s ult s\]. 

Eff ec ts o f Subopti m al Estim ate s  and LR  M ode l s  {#sec t 
 violation_is sufficiently_significant if $nS\geq c$_for some_constant_$c$, simulations_show_that the best_choice of $c$_depends on the distribution_of settings and_outcomes_in the experiment.

The third and simplest approach is to block the data from the_trials._Instead of_updating_the_log-$p$-value after every trial, we_process data $h$ trials at_a time._The first block is used only for estimating_the_setting and outcome_distribution of future trials. That is, we set $R_k(x)=1$_for $k=0,\ldots,(h-1)$. Subsequently, we have $R_{mh+k}=R_{mh}$_for $k=1,\ldots,(h-1)$ and_all_$m$._Note that neither the_validity nor the asymptotic optimality of_the calculated $p$-values requires updating the_PBRs after each trial. Choosing $h$ large_enough ensures that the first block’s_trials have sufficient information for_obtaining reasonable_estimates of the distribution. An_additional advantage of_blocking the_trials is that_we avoid unnecessarily invoking the computationally_costly optimizations required_for updating the PBRs. We standardized_the_choice of block_size_so_that if_the total number_of_trials to_be_analyzed is $N$, $h$ is the_maximum_of $\lceil N/1000\rceil$ and $\lceil\ln(2d)d\rceil$, where $d$_is the number of_possible_setting and outcome combinations_in a trial. The first_expression ensures that we do not_lose too_much log-$p$-value_by using the first block only for learning the setting and_outcome distribution. The second one is_chosen so that if_$q$ is_uniform,_the probability that_every_setting and_outcome combination occurs is at least $1/2$.

We_conclude this_section with a note on implementing_the PBR protocol. For_monitoring_an experiment and to adapt to_changes in experimental configuration, the estimated_setting and outcome distributions used_in_the_PBRs should be based on_recent trials only. This can be_accomplished by windowing_the trials with a window large enough_to_have statistically significant violation of LR_(if_there is violation), but small enough_to_avoid_seeing significant changes in configuration._Our implementation of the local realism_analysis engine uses a computationally simpler approach based on_weighting the trials_with exponentially decreasing weights in_time_determined_by a configurable half-life. This feature was not used in_the comparisons_in Sec. \[sect:results\].

Effects of_Suboptimal Estimates and LR Models {#sect