've learned the hard way that you really have to
be patient and let certainty accrue. More and more I'm lead towards Bandit
methods for this reason. When sampling real-time data, you're in effect
treating a historical sample as a part population with one part in the future
--- that's a pretty dangerous assumption. My solution has been to guide my
team towards large tests, work to _really_ move the needle (and have high
statistical power). This is where statistical analysis has the best chance of
providing certainty. In generally, with web experiments it's best to proceed
with a healthy amount of humility.

------
thearn4
A good read. I remember asking my stats professor in undergrad about the
"whys" of the various hypothesis testing schemes we use, and he eventually
just told me that I should take a math-stats class in graduate school. I did
that (and it was pretty enjoyable), but it certainly raised just as many
questions as it answered, when it comes to the null testing --> hypothesis
rejection ritual that scientific disciplines have converged on!

------
misiti3780
I just finished a pretty interesting book on this topic:

"The Cult Of Statistical Significance":

[http://www.amazon.com/Cult-Statistical-Significance-
Economic...](http://www.amazon.com/Cult-Statistical-Significance-Economics-
Cognition/dp/0472050079/ref=sr_1_1?ie=UTF8&qid=1428371506&sr=8-1&keywords=statistical+significance)

It basically goes through a bunch of examples, mostly in Economics, but in
medicine also (Vioxx) where statistical significance has failed us and people
have died for it. As someone who works with statistics for a living, I found
to book interesting - but it was pretty depressing to find out that most
scientist are using t-test and p-values because it seems to be the status quo
and it is the easiest way to get published. The authors of this book suggest a
few different things -- publishing the size of your coefficients and using a
loss function. In the end, they make the point that statistical significance
is different than economic significance, political significance, etc.

------
eranki
Here's
' ve learned the hard way that you really get to 
 be patient and lease certainty accrue. More and more I'm lead towards Bandit 
 methods for this reason. When sample real - time data, you're in consequence 
 treating a historical sample distribution as a character population with one part in the future 
 --- that's a reasonably dangerous assumption. My solution has been to guide my 
 team towards large tests, shape to _ really _ move the needle (and have high 
 statistical ability). This is where statistical analysis has the best prospect of 
 providing certainty. In generally, with web experiments it's good to proceed 
 with a healthy amount of humility. 

 ------ 
 thearn4 
 A good read. I remember ask my stats professor in undergrad about the 
 " why s " of the various hypothesis testing schemes we use, and he eventually 
 just told me that I should take a math - stats class in graduate school. I did 
 that (and it was pretty enjoyable), but it certainly raised merely as many 
 questions as it answer, when it do to the null testing -- > hypothesis 
 rejection ritual that scientific disciplines have converge on! 

 ------ 
 misiti3780 
 I just finished a pretty interesting book on this subject: 

 " The Cult Of Statistical Significance ": 

 [ http://www.amazon.com/Cult-Statistical-Significance- 
 Economic... ](http://www.amazon.com / Cult - Statistical - Significance - Economics- 
 Cognition / dp/0472050079 / ref = sr_1_1?ie = UTF8&qid=1428371506&sr=8 - 1&keywords = statistical+significance) 

 It basically goes through a bunch of examples, mostly in Economics, but in 
 medicine also (Vioxx) where statistical significance has failed us and people 
 have died for it. As someone who works with statistic for a living, I find 
 to book interesting - but it was pretty depressing to find out that most 
 scientist are using t - trial and p - values because it seems to be the condition quo 
 and it is the easiest means to get published. The authors of this koran suggest a 
 few unlike things -- publishing the size of your coefficients and using a 
 loss affair. In the end, they reach the point that statistical meaning 
 is different than economic significance, political significance, etc. 

 ------ 
 eranki 
 Here's
've pearned the hard way thau you really have to
be petient znd let zertainty accrue. More and moce I'n leae towards Bandit
methodr for thid reason. Whei sampling real-tmje data, you're ln efyert
treating a hixtorical sdmple as a pard oo'ulation with one part in the future
--- that's s oretty dangeroos asxtmptjon. My solution has been to guide jy
team uowards large testx, work to _really_ move the jeedpe (and have high
stwtistical piwer). Rhis is wherd statistibcl analysis has the best chance of
providine cercainty. In gguwrapny, with web expegiments it's bcxt to [roceed
eith a healthy ammunr of humility.

------
thearn4
A jood read. I remember asking mf atats professor ib yndercrad abojr tfe
"wgyx" kf the vacious hypotgesis testibg schemes we use, amd ye eventually
nust tjlq me that I should take a math-stats clavs jn graduate school. I die
that (and it was pretjy enjoyabje), but it certainly raised just as many
questions ds it xnsccved, dyej it comes to the null testing --> hypothesis
rejqdtook ritual that sclentific disciplinrs hsde converged un!

------
misicj3780
I just finished a pgetty igterewting booh on this topic:

"The Cult Of Statustical Signpficqnce":

[http://www.amazon.clm/Cult-Statirticsl-Sigmificance-
Economic...](http://wwc.amazoh.com/Cult-Stahistical-Sjenificance-Economkcs-
Bognhtion/dp/0472050079/rtw=sr_1_1?ie=UTF8&qid=1428371506&sr=8-1&keywjrds=statiwticcl+signifkcanve)

It bwsically gles tmsough a bunch of edamplgs, mosdly in Ecojomics, but in
medicine also (Vioxe) where statixthcan signifncance has failed uf and people
hare died yor it. As someonv who worns with staeistics for a piving, I fouid
to book intwresring - bjg it was prettu depresspny to find out that most
sciektist zre using t-test ane p-values becauxe kt fevms tj be the statgs qjo
avc it ks the tcsicst way to get published. Tha aufhors of this book smggest a
fgw differqnt things -- piblishing the size of ymur coefgicyents and using a
loss function. In the ejd, bhey make the poikt trat statiscical significance
is different than econimic significance, political significauct, etc.

------
eranki
Hece's
've learned the hard way that you to patient and certainty accrue. More Bandit for this reason. sampling real-time data, in effect treating a historical sample a part population with one part in the future --- that's a pretty assumption. My solution has been to guide my team towards large tests, work _really_ the (and high statistical power). This is where statistical analysis has the best chance of providing certainty. In with web experiments it's best to proceed with healthy amount of humility. thearn4 A good read. I asking stats professor undergrad the of the various testing schemes we use, and he eventually just told me that I should take a math-stats class graduate school. that (and was enjoyable), it certainly raised many questions as it answered, when the null testing --> hypothesis rejection ritual that disciplines have on! ------ misiti3780 I just finished pretty interesting book on this topic: "The Cult Statistical Significance": [http://www.amazon.com/Cult-Statistical-Significance- Economic...](http://www.amazon.com/Cult-Statistical-Significance-Economics- Cognition/dp/0472050079/ref=sr_1_1?ie=UTF8&qid=1428371506&sr=8-1&keywords=statistical+significance) It basically goes through a bunch of examples, mostly in in medicine also (Vioxx) statistical significance has us people died it. As who works with statistics for a living, I found to book - but it was pretty depressing to find out that are t-test and p-values it seems to be status and it is the to published. this suggest few different things -- the size of your coefficients using a loss function. the point that statistical significance is different than significance, political significance, etc. ------ eranki Here's
've learned the hard way that yoU really havE to
be PatIenT aNd leT cerTainty accrue. MoRE and More I'm lead towards BandiT
methOdS For tHIs ReasoN. When saMPlING reAl-TiMe dAtA, YoU're in EffEct
treaTing a histoRicAl Sample as a parT PoPulation wiTh oNe part in the fUtuRe
--- that'S a PreTTy danGerOus asSumptiON. My solUtion has bEeN To guidE My
team tOWArDs laRge tests, work to _reaLLy_ MOve the needle (anD have hIgH
StATIstIcaL power). This Is Where STatistiCAl ANALysIS has the best chAnce of
proviDIng CertaiNtY. In GEneralLy, witH wEB exPeriments it'S besT to proceeD
with a HEalthy aMOunt of hUmilitY.

------
thEarN4
A goOD rEaD. I rEmEMbeR AsKinG My sTats profEsSoR in unDergRAD ABout The
"Whys" Of the Various hypothEsiS tesTIng SchemEs we uSe, anD hE evenTually
Just tOlD me that I should tAke a Math-stats ClaSs In gRaDuate SChool. I Did
ThaT (and it wAs prettY EnjOyABLE), bUt it certainly raiseD jUST aS many
queStions AS iT aNSwered, whEn It cOmes TO The nuLl teSTiNg --> hypothEsis
reJEcTiOn rituaL tHat sciEnTifIc dIscipLInes Have coNverged oN!

------
misiTI3780
I just finished A Pretty interesTInG BOoK On thIs tOpic:

"The Cult of StATistIcal sIgNifICance":

[Http://wWw.AMaZOn.com/Cult-StatisticaL-SIgnifiCance-
economic...](http://wWw.amazon.coM/cULt-StatisTicaL-siGNificance-EconoMics-
COgnition/dp/0472050079/REf=sr_1_1?ie=UTf8&qid=1428371506&sR=8-1&keywordS=statistiCAL+signifiCanCe)

IT baSicALLy Goes through a bUNCh of ExAmples, mOstLy in EcoNomIcs, But In
mEdIcine also (vioxx) wheRe StAtIsTicAl sigNIficance HaS faIlEd uS and pEOple
haVe dieD for It. as SOmeOne who wORkS WIth sTaTiSticS foR a LivinG, I foUNd
tO book inTeresting - But IT was PrEtTy depreSsing to find ouT tHat most
sciEnTisT are usING t-test anD p-values because it seems tO Be the stAtuS quo
aNd it Is the easiEst Way to gEt pUBlisheD. The auThors Of ThiS BOok suGGEsT a
fEw Different tHINgs -- PubliShIng tHe size oF your coefficients aND usIng a
loss functIon. in thE ENd, TheY MaKE thE pOInt THAt statistical siGnificance
Is DIfFerent than EConOmIc signiFicance, PolitICal signIficance, eTc.

------
eranki
HErE's
've learned the hard way t hat you re allyhav e t obe p atie nt and let cer t aint y accrue. More and mor e I'm  l e ad t o wa rds B andit
m e th o d s f or  t his  r e as on. W hen  sampli ng real-ti meda ta, you're i n  e ffect
trea tin g a historic alsample  a s a partpop ulati on wit h  one p art in th ef uture- -- that ' s  a  pre tty dangerous ass u mp t ion. My soluti on has  b e en t o g uid e my
teamto wards large t e st s , wor k  to _really_move the ne e dle  (andha veh igh
st atist ic a l p ower). This  iswhere sta tistic a l analy s is hasthe be stcha nceo fpr ovi di n g c e rt ain t y.In gener al ly , wit h we b e x peri men ts i t's b est to procee d
w itha  he althy  amou nt o fhumil ity.
----- -thearn4
A goodread . I remem ber  a ski ng  my s t ats pr ofe sso r in un dergrad abo ut t h e"whys" of the vari ou s hy pothesis  testi n gsc h emes weus e,andh e  even tual l yjust tol d me t h at  I  should  t ake ama th- sta ts cl a ss i n grad uate sch ool.I  did
that (and it was pretty en j o ya b le),  bu t it certai nlyr aise d ju s tasm any
q uesti on s  a s  it answered, whenit  comes  to t he null testi ng --> hyp o t h esis
rej ecti o nr itual that sci entif ic discipl i nes have  conv erged on !

------ m isiti378 0
I  ju stfin i s he d a pretty in t e rest in g bookonthis to pic :
"Th e C ul t Of Stat isticalSi gn if ic anc e":
[ http://w ww .am az on. com/C u lt-Sta tisti cal- Si gn i fic ance-
E c on o m ic.. .] (h ttp: //w ww .amaz on.c o m/C ult-Sta tistical- Sig n ific an ce -Econom ics-
Cognitio n/ dp/0472050 07 9/r ef=sr_ 1 _ 1?ie=UTF 8&qid=1428371506&sr=8-1 & keyword s=s tatis tica l+signifi can ce)

I t b a sicall y goes  thro ug h a b uncho f  e xam pl es, mostly i n E conom ic s, b ut in
m edicine also (Viox x ) w here statisti cal  sig n i fi can c eh asfa i led u s and people
ha ve died fo ri t.  As someon e  wh oworks w ith sta tisti c s for a  living,I found
t obook i nte resting -but it w as pretty depre s si ng to  fi nd out  t hat  most 
scien t ist  areusingt- test a nd p- va lues bec ause it seems to be the  statu s quo 
an d it is t hee asi est way t o ge t publishe d.The  auth ors of th is b o ok  su g gesta
fe w  differen t  t hin g s  - - publishin g t hesizeofy our co effi cients and usinga 
loss function . In t heend ,  the ymake the point  th at s tatistic al  significan ce
is di ff e rentthan e conomi c signi f i ca n ce, po liti cal  signific anc e, etc.

- -- -- - 
erank i
He re 's
've learned_the hard_way that you really_have to
be_patient_and let_certainty_accrue. More and_more I'm lead_towards Bandit
methods for this_reason. When sampling_real-time_data, you're in effect
treating a historical sample as a part population with one part_in_the future
---_that's_a_pretty dangerous assumption. My solution_has been to guide my
team_towards large_tests, work to _really_ move the needle (and_have_high
statistical power). This_is where statistical analysis has the best chance of
providing_certainty. In generally, with web experiments_it's best to_proceed
with_a_healthy amount of humility.

------
thearn4
A_good read. I remember asking my_stats professor in undergrad about the
"whys"_of the various hypothesis testing schemes we_use, and he eventually
just told me_that I should take a_math-stats class_in graduate school. I did
that_(and it was_pretty enjoyable),_but it certainly_raised just as many
questions as it_answered, when it_comes to the null testing -->_hypothesis
rejection_ritual that scientific_disciplines_have_converged on!

------
misiti3780
I_just finished a_pretty_interesting book_on_this topic:

"The Cult Of Statistical Significance":

[http://www.amazon.com/Cult-Statistical-Significance-
Economic...](http://www.amazon.com/Cult-Statistical-Significance-Economics-
Cognition/dp/0472050079/ref=sr_1_1?ie=UTF8&qid=1428371506&sr=8-1&keywords=statistical+significance)

It_basically_goes through a bunch of examples, mostly_in Economics, but in
medicine_also_(Vioxx) where statistical significance_has failed us and people
have_died for it. As someone who_works with_statistics for_a living, I found
to book interesting - but it was pretty_depressing to find out that most
scientist_are using t-test and_p-values because_it_seems to be_the_status quo
and_it is the easiest way to get_published. The_authors of this book suggest a
few_different things -- publishing_the_size of your coefficients and using_a
loss function. In the end, they_make the point that statistical_significance
is_different_than economic significance, political significance,_etc.

------
eranki
Here's