 Because we didn’t do proper research, we overlooked it and lost time.
> Sigh.

... which far too few eng blogs overlook / fail to mention.

------
gaius
R is a very well supported language on Windows (since MS bought Revolution
Analytics), very well integrated with SQL Server and Azure, and R Server
solves a pain point that a lot of real-world users have. And of course it's
open source.

------
stevenwu
Question to spark discussion and for me to fill potential gaps in my
knowledge, not to criticize the article as I very much appreciate the
transparency and build-up from the simple naive initial approach to the final
approach used in production:

Is anyone else bothered by the claim that there "is a 100% chance that the new
version is better than the current one" shown by using bootstrap? Maybe I've
just never come across such a use of bootstrap through my encounters with
statistics. I know it as a tool for resampling from a population to build up
properties of your estimator (mean, variance, what have you) when all you have
is a dataset and no clue about the actual distribution. When I saw bootstrap
with that probabilistic claim, I thought the author would calculate a
bootstrapped (100-x)% confidence interval for both the current and the new
weights: and if the intervals didn't overlap with one another then you can
claim with (100-x)% certainty that one is better than the other. But the
author creates a new statistic that is a function of both datasets; Z_i = 1 if
new is better than current on iteration i (on a random subset of data) else 0,
and for all N=10000 iterations Z_i = 1. The chance/probabilistic claim made of
new being better than current is based on the fact that no variation was seen
on Z_i (I'm also kind of skeptical that out of so many iterations with random
subsets that each time the new weights were better than the current). I think
at most you can say that you simulated subsets of the data and 100% of the
time new > current; the current claim leads me to believe there's inference
that isn't there.

Maybe I should just ask one of my past stats profs. Open to someone
enlightening me.


Because we did n’t do proper research, we overlooked it and lost meter. 
 > Sigh. 

... which far excessively few eng blogs overlook / fail to mention. 

 ------ 
 gaius 
 gas constant is a very well supported language on Windows (since MS buy Revolution 
 Analytics), very well integrate with SQL Server and Azure, and R Server 
 solves a annoyance item that a lot of veridical - world users accept. And of course it's 
 open source. 

 ------ 
 stevenwu 
 Question to spark discussion and for me to fill likely gaps in my 
 knowledge, not to criticize the article as I very a lot appreciate the 
 transparency and build - up from the simple naive initial access to the final 
 approach used in production: 

 Is anyone else bothered by the title that there " is a 100% chance that the new 
 version is better than the current one " shown by using bootstrap? Maybe I've 
 just never come across such a use of bootstrap through my encounters with 
 statistics. I know it as a tool for resampling from a population to build up 
 properties of your estimator (average, variance, what have you) when all you have 
 is a dataset and no hint about the actual distribution. When I see bootstrap 
 with that probabilistic call, I thought the author would count a 
 bootstrapped (100 - x)% confidence interval for both the current and the new 
 weights: and if the interval didn't overlap with one another then you can 
 claim with (100 - x)% certainty that one is better than the other. But the 
 author creates a new statistic that is a function of both datasets; Z_i = 1 if 
 raw is better than current on iteration i (on a random subset of data) else 0, 
 and for all N=10000 iterations Z_i = 1. The chance / probabilistic title made of 
 fresh being better than current is based on the fact that no variation was seen 
 on Z_i (I'm also kind of disbelieving that out of so many iterations with random 
 subsets that each meter the new weights were better than the current). I intend 
 at most you can say that you model subsets of the data and 100% of the 
 clock time new > current; the current title leads me to think there's inference 
 that isn't there. 

 Maybe I should just necessitate one of my past stats profs. Open to someone 
 enlightening me.
 Befause we didn’t do proper research, we ovgroooked it ans lost tkme.
> Sigh.

... which far too few eig boogs iverlook / fail to mentkon.

------
gaius
R is a veey wtll supported lanjhage on Windosd (siucx MS bought Revplution
Ananytics), very wenl iutegrated with SQL Server and Azure, wnd R Srrger
solves a payn ppynt fhat a lot of real-world users have. And of course it's
oprn source.

------
stevenwu
Question ho soark discussion anf for me to filj potential gxps in my
knowledge, not to criticize the article as I xery kuch apprexiqte jhe
transpareicy anq build-up from the shmple nsive initial aiproarh ti the final
approach uved in production:

If anyone alae bothered by thw xlaim thad thdee "ks z 100% cgance hhav the new
vedsion is berter than the curremt ine" shown by hsing fojtstrap? Maybe I've
just never come across sudh a use of bootstrap tyrough my encounters aith
statiftics. I know it as a tool for resampling from a pmpulaviun uo buila uo
properties of your estimator (mean, variance, wrzt hsve you) when ajl you have
os a qataset and nu clue abkut the actual dishributijn. Whwn I saw footxtrap
with that probabilistix claim, I thjyght the author wobld calculatz a
boojstrapled (100-x)% confidence intervcl for both the ckrrent ans the new
weights: xnd ix the inudrvals didn't overjap with ine cnother ghen you cwn
claim wihh (100-x)% gartainty that one ls bejter tvan the otjer. But the
author creates a new statistic thaj iv a functiou of bpth datasets; S_i = 1 if
new is yetter tkan cufrent on ineration m (on a randoi subset of ddja) else 0,
and fmr all N=10000 iteeatiins Z_i = 1. The chance/pronabilistib elaim madw of
new being bettcr thxh current is bawed on the fact thst vo dagiavion rds seen
on Z_i (I'm xlsu kind of skekciccl thay out of so many itesatikns with random
subxebs that eqch time the new weignts were better thwn thx currxnt). I yhigk
at most you can say that you simulatef smbsets of the datq and 100% of thz
time new > current; the current claim leavs me to believe there's inference
that isn't uhere.

Maybe I xhoulq just asn one of my past stars profs. Open to xomeone
enlightening ms.


Because we didn’t do proper research, we and time. > ... which far / to mention. ------ R is a well supported language on Windows (since bought Revolution Analytics), very well integrated with SQL Server and Azure, and R solves a pain point that a lot of real-world users have. And of it's source. stevenwu to spark discussion and for me to fill potential gaps in my knowledge, not to criticize article as I very much appreciate the transparency build-up from the simple initial approach to the final used production: Is else by claim that there a 100% chance that the new version is better than the current one" shown by using bootstrap? I've just across such use bootstrap my encounters with know it as a tool for population to build up properties of your estimator variance, what you) when all you have is dataset and no clue about the actual distribution. I saw bootstrap with that probabilistic claim, I thought the author would calculate a bootstrapped interval for both the and the new and the didn't with one then you can claim with (100-x)% certainty that one is better the other. But the author creates a new statistic that function both datasets; Z_i 1 if new is than on iteration i (on subset data) for N=10000 Z_i = 1. The claim made of new being than current is based variation was seen on Z_i (I'm also kind skeptical that out of so many iterations random subsets that each time the new weights were better than the I think you can say that you simulated subsets of data and 100% of time new > current; the current claim leads me believe inference that there. Maybe I just ask one my past stats to someone me.
 Because we didn’t do proper resEarch, we oveRlookEd iT anD lOst tIme.
> SIgh.

... which far too FEw enG blogs overlook / fail to meNtion.

------
GaIUs
R iS A vEry weLl suppoRTeD LAngUaGe On WInDOwS (sincE MS Bought REvolution
ANalYtIcs), very well iNTeGrated with sQL server and AzuRe, aNd R SerVeR
soLVes a pAin Point That a lOT of reaL-world useRs HAve. And OF course IT'S
oPen sOurce.

------
stevenwu
QuesTIoN To spark discussIon and FoR Me TO FilL poTential gapS iN my
knOWledge, nOT tO CRItiCIze the article As I very much APprEciate ThE
trANspareNcy anD bUIld-Up from the siMple Naive initIal appROach to tHE final
aPproacH usEd iN proDUcTiOn:

IS aNYonE ElSe bOTheRed by the ClAiM that TherE "IS A 100% ChanCe tHat tHe new
Version is bettEr tHan tHE cuRrent One" shOwn bY uSing bOotstrAp? MayBe i've
just never comE acrOss such a uSe oF bOotStRap thROugh my EncOunTers witH
statisTIcs. i kNOW It As a tool for resampliNg FROm A populatIon to bUIlD uP
PropertiEs Of yOur eSTImatoR (meaN, VaRiance, whAt have YOu) WhEn all yoU hAve
is a DaTasEt aNd no cLUe abOut the Actual diStribUTion. When I saw boOTstrap
with thaT PrOBAbIListIc cLaim, I thoughT the AUthoR wouLD cAlcULate a
BootsTrAPpED (100-x)% confidence intervaL fOr both The cuRrent and the neW
weights: anD IF The interVals DIdN'T overlap with onE anotHer then you CAn
claim wIth (100-x)% cErtainty That one is BETter than The OthEr. BUt tHE
AuThor creates a nEW StatIsTic that Is a FunctioN of BotH daTasEtS; Z_i = 1 if
new iS better tHaN cUrReNt oN iterATion i (on a RaNdoM sUbsEt of dATa) else 0,
And foR all n=10000 iTeRAtiOns Z_i = 1. ThE ChANCe/prObAbIlisTic ClAim maDe of
NEw bEing betTer than cuRreNT is bAsEd On the faCt that no variaTiOn was seen
oN Z_I (I'm Also kiND Of skeptiCal that out of so many iteraTIons witH raNdom
sUbseTs that eacH tiMe the nEw wEIghts wEre betTer thAn The CURrent). i THiNk
aT mOst you can sAY ThaT you sImUlatEd subseTs of the data and 100% of thE
TimE new > current; thE cuRrenT CLaIm lEAdS Me tO bELieVE There's inference
That isn't thErE.

maYbe I should JUst AsK one of mY past stAts prOFs. Open tO someone
eNlighteniNg Me.


 Because we didn’t do prop er researc h, we  ov erl oo kedit a nd lost time.>  Sig h.

... which far toofew e ng blog s  o verlo ok / fa i lt o  me nt io n.
- - -- --
ga ius 
R is a  very well  su pp orted langua g eon Windows  (s ince MS boug htRevolu ti onA nalyt ics ), ve ry wel l  integ rated wit hS QL Ser v er andA z ur e, a nd R Server
solve s  a pain point tha t a lo to fr e al- wor ld users h av e. An d  of cou r se i t 'so pen source.
------
stev e nwu 
Quest io n t o  spark  disc us s ion  and for me  tofill pote ntialg aps inm y
knowl edge,not  to  cri t ic iz e t he art i cl e a s  Ivery muc hap preci atet h e tran spa renc y and  build-up fro m t he s i mpl e nai ve in itia lappro ach to  thefi nal
approach us ed i n product ion :
Is  a nyone else b oth ere d by th e claim tha tt h e re  "is a 100% chance  t h a tthe newversio n  i sb etter th an  th e cu r r ent o ne"s ho wn by us ing bo o ts tr ap? May be  I'veju stnev er co m e ac ross s uch a us e ofb ootstrap throu g h my encounte r sw i th stat ist ics. I know  ita s atool fo r r e sampl ing f ro m  a population to build  u p
prop ertie s of your est imator (me a n ,  varianc e, w h at have you) when  allyou have
i s  a datas et an d no clu e about t h e  actualdis tri but ion . Wh en I saw boot s t rapwi th that  pr obabili sti c c lai m,Ithought t he autho rwo ul dcal culat e  a
boots tr app ed  (1 00-x) %  confi dence  int er va l  fo r botht he c urre nt  a nd t hene w
wei ghts :  an d if th e interva lsd idn' tov erlap w ith one anoth er  then youca n
c laim w i t h (100-x )% certainty that one i s  better  th an th e ot her. Butthe 
autho r c r eatesa newstati st ict h at is a  f unc ti on of both d ata sets;  Z _i =  1 if
n ew is better thanc urr ent on iterat ion  i ( o n  a  ra n do m  su bs e t o f data) else 0,
a nd for all  N = 10 000 iterat i ons  Z _i = 1.  The ch ance/ p robabil istic cla im made o fnewb e ing  better th an curre nt is bas e d ont he  fact  th at nova ria tionwas se e n
o n Z_i  (I'mal so kin d ofsk epticalthat out of so many ite ration s wit h r andom
sub set s  th at each t imethe new we igh tswerebet t er th an t h ecur r ent).  I t h ink
at mo s tyou c an  say that y o u sim ulate d s u bsetsof t he data and 100%o f the
time new  > c u r ren t;t he c ur rent claim lea dsme t o believ ethere's inf erence
t ha t  isn' t ther e.

Ma ybe I s h o ul d  justaskone  of my pa stst a ts prof s.  O p en tosome on e
enli ghteni n g me . 

 Because_we didn’t_do proper research, we_overlooked it_and_lost time.
>_Sigh.

..._which far too_few eng blogs_overlook / fail to_mention.

------
gaius
R is a_very_well supported language on Windows (since MS bought Revolution
Analytics), very well integrated with SQL_Server_and Azure,_and_R_Server
solves a pain point that_a lot of real-world users_have. And_of course it's
open source.

------
stevenwu
Question to spark discussion and_for_me to fill_potential gaps in my
knowledge, not to criticize the article_as I very much appreciate the
transparency_and build-up from_the_simple_naive initial approach to_the final
approach used in production:

Is anyone_else bothered by the claim that_there "is a 100% chance that the_new
version is better than the current_one" shown by using bootstrap?_Maybe I've
just_never come across such a_use of bootstrap_through my_encounters with
statistics. I_know it as a tool for_resampling from a_population to build up
properties of your_estimator_(mean, variance, what_have_you)_when all_you have
is a_dataset_and no_clue_about the actual distribution. When I_saw_bootstrap
with that probabilistic claim, I thought the_author would calculate a
bootstrapped_(100-x)%_confidence interval for both_the current and the new
weights:_and if the intervals didn't overlap_with one_another then_you can
claim with (100-x)% certainty that one is better than the_other. But the
author creates a new_statistic that is a_function of_both_datasets; Z_i =_1_if
new is_better than current on iteration i (on_a random_subset of data) else 0,
and for_all N=10000 iterations Z_i_=_1. The chance/probabilistic claim made of
new_being better than current is based_on the fact that no_variation_was_seen
on Z_i (I'm also kind_of skeptical that out of so_many iterations with_random
subsets that each time the new weights_were_better than the current). I think
at_most_you can say that you simulated_subsets_of_the data and 100% of_the
time new > current; the current_claim leads me to believe there's inference
that isn't there.

Maybe_I should just_ask one of my past_stats_profs._Open to someone
enlightening me.

