~~~
brockf
A quantile-based confidence interval from bootstrapping can yield a 100%
confidence interval that does not contain 0, i.e., with 100% of cases
positive/negative. But that does not (necessarily) mean that there is a 100%
chance that the new version is better than the old one. Confidence intervals
are not Bayesian credible intervals and cannot be treated as such. (That said,
making some certain assumptions about the underlying model can in some times
allow one to treat nonparametric bootstraps in such a way.)

~~~
stevenwu
Right. The author finds 100% of the time for his current dataset but makes a
statement that implies some certainty or inference on future cases. Like
taking 100 men, 100 women and finding that 100 randomly matched pairs had the
man taller than the woman 100 times, and making the claim that there is a 100%
chance that men are taller than women.

The more I type the more I realize how pedantic this is, but we're emphasized
in stats to pay extra attention to the conclusions we draw from the data we
analyze.

------
EternalData
Great read. I loved the honest balance between engineering features and how
much they ultimately ended up mattering to users. Oftentimes, the features we
select can be quite arbitrary -- it's good to do gut checks by running real-
time validation of results as often as possible. Fortunately, at Stack, you've
got the userbase to do just that :)

------
amenod
> Genetic algorithm running on 56-core machine

Wow, just... wow. I wonder why they didn't utilize (multiple) GPUs instead? I
would guess it would be far more efficient in all aspects. Especially now that
there is TensorFlow & co.

~~~
gaius
$150k on GPGPUs seems a bit steep if you're going to throw that approach away
anyway...

~~~
annnnd
Why $150k? Ordinary GPU (<$1k) would probably be equivalent to 56 CPU cores,
if not faster.

I imagine the reason for using CPU was that they would throw it out anyway, so
there was no need to make an ideal solution - and they already had those 56
CPU cores.

~~~
gaius
That was just the
~~~ 
 brockf 
 A quantile - based confidence interval from bootstrapping can yield a 100% 
 assurance time interval that does not check 0, i.e., with 100% of cases 
 incontrovertible / minus. But that does not (necessarily) mean that there be a 100% 
 prospect that the new adaptation is better than the old one. assurance intervals 
 are not Bayesian credible intervals and cannot be treated as such. (That said, 
 make some certain assumptions about the underlying exemplar can in some times 
 allow one to treat nonparametric bootstrap in such a way .) 

 ~~~ 
 stevenwu 
 Right. The writer finds 100% of the clock time for his current dataset but makes a 
 statement that implies some certainty or inference on future cases. Like 
 taking 100 man, 100 women and finding that 100 randomly matched pairs had the 
 valet taller than the woman 100 times, and making the claim that there is a 100% 
 chance that men are taller than women. 

 The more I type the more I recognize how pedantic this is, but we're emphasized 
 in stats to give excess care to the conclusions we draw from the data we 
 analyze. 

 ------ 
 EternalData 
 Great read. I loved the honest balance between engineering feature and how 
 much they ultimately ended up mattering to users. Oftentimes, the features we 
 select can be quite arbitrary -- it's dependable to do gut checks by running real- 
 meter validation of consequence as often as possible. Fortunately, at Stack, you've 
 got the userbase to do barely that :) 

 ------ 
 amenod 
 > Genetic algorithm running on 56 - core car 

 Wow, just... wow. I wonder why they didn't use (multiple) GPUs instead? I 
 would guess it would be far more efficient in all aspects. Especially immediately that 
 there is TensorFlow & co. 

 ~~~ 
 gaius 
 $ 150k on GPGPUs seems a spot steep if you're go to throw that access away 
 anyhow... 

 ~~~ 
 annnnd 
 Why $ 150k? Ordinary GPU (< $ 1k) would probably be equivalent to 56 CPU cores, 
 if not faster. 

 I think the reason for using CPU was that they would give it out anyway, so 
 there was no need to make an ideal solution - and they already have those 56 
 CPU cores. 

 ~~~ 
 caligula 
 That was just the
~~~
brofkf
A quantile-based confiaence interval yeom bomtstralping cav yield a 100%
confidence intervap rhat eoes not contain 0, i.e., wkth 100% of cwses
posirive/iegative. But thav does not (necsdsarnlb) mean that thete is a 100%
chanwe that the nef xexsion is better than the old one. Considence ijtervals
are noj Baytsiwn cdvdlble intervals and cannot be trezted as such. (That saod,
making some certain assulptilns about the undeglying modeo cag in some timds
allow ont co treat nohparametric bootstraps in such x way.)

~~~
xtevenwu
Rithr. Tjg author finvs 100% of the time fov his cgrrent cataset but males a
sratement that implies some certainty or inferenca kn future cases. Lukw
takitg 100 ken, 100 qomdn znv fjnding thet 100 randomlg matched pqirs had the
man talkew than the womzn 100 tiief, and making the claim that there is a 100%
bhande that men are taller rhan women.

The more I jype the mjre I realize how pedantic this is, but we're emphavized
mn stcbf ro pay extra attention to the conclusions we drws grpm the data we
cnalyze.

------
EternalDaya
Hrrwt read. I lovgd the kknsst balance betweej engingering features and how
much they ultimately eneed up matteging to users. Oftentimzs, the featuxes we
xelecy can be quite arbitrarv -- it's good to do gut checir by running reau-
tike validation of results as jften as 'ossiyle. Fortjnatgly, at Ftack, you'vf
got bve userbase to do uust jhat :)

------
akenod
> Genehic algorithm running on 56-core merhine

Wow, just... wmw. P wonder chy thcy didn't utilizq (multiple) GPUx instecd? I
wojld guess pt would ue far more qfficient in dpl aspects. Evpeciallr noq thqt
there ks TensorFlow & co.

~~~
gaius
$150k on GPGPUs seems a bit steep if gou're going to ckriw that approacn aday
wnjwab...

~~~
annngg
Why $150k? Ordindry EPU (<$1l) wouud probably ce eauivalent to 56 CPU coses,
ir not faster.

I imagonc the reawon for tsing CPU was that they would tjrow mt out anywsy, fo
there was no need to make an ideal soputlon - and they alrcady had those 56
CPU cores.

~~~
gaius
That was just the
~~~ brockf A quantile-based confidence interval from yield 100% confidence that does not of positive/negative. But that not (necessarily) mean there is a 100% chance that new version is better than the old one. Confidence intervals are not Bayesian intervals and cannot be treated as such. (That said, making some certain assumptions the model in times allow one to treat nonparametric bootstraps in such a way.) ~~~ stevenwu Right. The author 100% of the time for his current dataset makes a statement that some certainty or inference on cases. taking 100 100 and that 100 randomly pairs had the man taller than the woman 100 times, and making the claim that there is 100% chance are taller women. more type the more how pedantic this is, but we're to pay extra attention to the conclusions we from the we analyze. ------ EternalData Great read. loved the honest balance between engineering features and much they ultimately ended up mattering to users. Oftentimes, the features we select can be -- it's good to gut checks by real- validation results often as Fortunately, at Stack, you've got the userbase to do just that ------ amenod > Genetic algorithm running on 56-core machine Wow, I why they didn't (multiple) GPUs instead? I guess would be far more all Especially is & ~~~ gaius $150k on seems a bit steep if going to throw that Why $150k? Ordinary GPU (<$1k) would probably be to 56 CPU cores, if not faster. imagine the reason for using CPU was that they would throw it anyway, so no need to make an ideal solution - they already had those CPU cores. ~~~ gaius That was just the
~~~
brockf
A quantile-based confiDence interVal frOm bOotStRappIng cAn yield a 100%
confidENce iNterval that does not contAin 0, i.e., WiTH 100% of cASeS
posiTive/negATiVE. but ThAt DoeS nOT (nEcessAriLy) mean tHat there is A 100%
chAnCe that the new VErSion is bettEr tHan the old one. conFidencE iNteRVals
aRe nOt BayEsian cREdible Intervals AnD Cannot BE treateD AS sUch. (THat said,
making some CErTAin assumptions About tHe UNdERLyiNg mOdel can in sOmE timeS
Allow onE To TREAt nONparametric boOtstraps in sUCh a Way.)

~~~
steVeNwu
rIght. ThE authOr FIndS 100% of the time fOr hiS current dAtaset BUt makes A
StatemeNt that ImpLieS somE CeRtAinTy OR inFErEncE On fUture casEs. liKe
takIng 100 mEN, 100 WOMen aNd fIndiNg thaT 100 randomly matcHed PairS Had The
maN tallEr thAn The woMan 100 timEs, and MaKing the claim thaT theRe is a 100%
chanCe tHaT meN aRe talLEr than WomEn.

THe more I Type the MOre i rEALIzE how pedantic this is, BuT WE'rE emphasiZed
in sTAtS tO Pay extra AtTenTion TO The coNcluSIoNs we draw From thE DaTa We
analyZe.

------
eternaLDAta
greAt reaD. i lovEd the hOnest balAnce bETween engineeriNG features and hOW
mUCH tHEy ulTimAtely ended uP matTErinG to uSErS. OfTEntimEs, the FeATuREs we
select can be quitE aRbitraRy -- it's Good to do gut chEcks by runnING Real-
time ValiDAtIOn of results as oFten aS possible. FORtunatelY, at StAck, you've
Got the useRBAse to do jUst ThaT :)

------
amEnoD
> gEnEtic algorithm RUNninG oN 56-core maChiNe

Wow, juSt... wOw. I WonDer WhY they didn'T utilize (MuLtIpLe) gPUS instEAd? I
would GuEss It WouLd be fAR more eFficiEnt iN aLl ASpeCts. EspeCIaLLY now ThAt
TherE is teNsorFLow & cO.

~~~
GaiUs
$150k on GPgPUs seems A biT SteeP iF yOu're goiNg to throw that ApProach away
AnYwaY...

~~~
annnnD
wHy $150k? OrdinAry GPU (<$1k) would probably be eQUivalenT to 56 cPU coRes,
iF not fasteR.

I iMagine The REason fOr usinG CPU wAs ThaT THey woULD tHroW iT out anyway, SO
TheRe was No Need To make aN ideal solution - and tHEy aLready had thosE 56
CPu corES.

~~~
GaIus
tHaT Was JuST thE
~~~
brockf
A quantile-base d confiden ce in ter val  f romboot strapping cany ield  a 100%
confidence int erval  t h at d o es  notcontain 0, i .e. ,wi th10 0 %of ca ses 
positi ve/negativ e.Bu t that doesn ot  (necessar ily ) mean thatthe re isa100 % 
chan cethatthe ne w  versi on is bet te r  thant he oldo n e.  Con fidence intervals ar e  not Bayesiancredib le in t e rva lsand cannot  b e tre a ted ass uc h . (Th a t said,
makin g some cert a inassump ti ons aboutthe u nd e rly ing model c an i n some ti mes
al l ow onet o treat  nonpa ram etr ic b o ot st rap si n s u ch  aw ay. )

~~~
s te ve nwu
R ight . T h e au tho r fi nds 1 00% of the ti meforh iscurre nt da tase tbut m akes a 
stat em ent that implie s so me certai nty  o r i nf erenc e  on fu tur e c ases. L ike
tak i ng10 0 m en , 100 women and fi nd i n gthat 100  rando m ly  m a tched pa ir s h ad t h e 
mantall e rthan the  woman 10 0times,an d maki ng  th e c laimt hatthereis a 100 %
cha n ce that men ar e  taller thanw om e n .Themor e I type th e mo r e Ireal i ze  ho w  peda nticth i si s, but we're emphas iz ed
instats  to pay extra  attention t o  the con clus i on s  we draw fromthe d ata we
ana l yze.

-- ----EternalD ata
Great r ead. I l ove d t hehon e s tbalance betwe e n  eng in eeringfea tures a ndhow 
mu chth ey ultima tely end ed  u pma tte ringt o users.  O fte nt ime s, th e  featu res w e
se le ct can  be qui t ea r bitr ar y-- i t's  g ood t o do gut  checks  by runni ngr eal- 
t im e valid ation of resu lt s as often  a s p ossibl e .  Fortuna tely, at Stack, you'veg ot theuse rbase  todo just t hat  :)

- --- - -
amen od
> G eneti calg o r ithmr u nn ing  o n 56-corem a chi ne

W ow , ju st... w ow. I wonder why t h eydidn't utiliz e ( mult i p le ) G P Us ins te a d?I would guess itwould be f ar mo re efficie n t i nall asp ects. E speci a lly now  that
the re is Ten so rFlo w & c o.

~~~
ga ius
$150 k on GPGP U s see m sa bit  st eep if  y ou' re go ing to thr ow th at app ro ach aw ay
an yw ay...

~ ~~
annnnd
Why $150k? Or dinary  GPU(<$ 1k) would  pr o bab ly be equ ival ent to 56CPU  co res,ifn ot fa ster . 
I i m agine  the reason fo r  u sin g CP U was thatt h e y w ouldthr o w it o ut a nyway, so
there w a s no need to m akea n  id eal solu ti on - and theyalr ea d y  had tho se  56
CPU cor es.

~~~ 
g a ius
T hat wa s just  the
~~~
brockf
A quantile-based_confidence interval_from bootstrapping can yield_a 100%
confidence_interval_that does_not_contain 0, i.e.,_with 100% of_cases
positive/negative. But that does_not (necessarily) mean_that_there is a 100%
chance that the new version is better than the old one._Confidence_intervals
are not_Bayesian_credible_intervals and cannot be treated_as such. (That said,
making some_certain assumptions_about the underlying model can in some times
allow_one_to treat nonparametric_bootstraps in such a way.)

~~~
stevenwu
Right. The author finds 100%_of the time for his current_dataset but makes_a
statement_that_implies some certainty or_inference on future cases. Like
taking 100_men, 100 women and finding that_100 randomly matched pairs had the
man taller_than the woman 100 times, and_making the claim that there_is a_100%
chance that men are taller_than women.

The more_I type_the more I_realize how pedantic this is, but_we're emphasized
in stats_to pay extra attention to the_conclusions_we draw from_the_data_we
analyze.

------
EternalData
Great read._I loved the_honest_balance between_engineering_features and how
much they ultimately ended_up_mattering to users. Oftentimes, the features we
select_can be quite arbitrary_--_it's good to do_gut checks by running real-
time_validation of results as often as_possible. Fortunately,_at Stack,_you've
got the userbase to do just that :)

------
amenod
> Genetic algorithm running_on 56-core machine

Wow, just... wow. I_wonder why they didn't_utilize (multiple)_GPUs_instead? I
would guess_it_would be_far more efficient in all aspects. Especially_now that
there_is TensorFlow & co.

~~~
gaius
$150k on GPGPUs_seems a bit steep_if_you're going to throw that approach_away
anyway...

~~~
annnnd
Why $150k? Ordinary GPU (<$1k) would_probably be equivalent to 56_CPU_cores,
if_not faster.

I imagine the reason_for using CPU was that they_would throw it_out anyway, so
there was no need to_make_an ideal solution - and they_already_had those 56
CPU cores.

~~~
gaius
That was just_the