 lucrative research time.

~~~
drjesusphd
Precious? Yes. Lucrative? Hardly.

------
tango12
Maybe it can be thought of as a tooling problem? Say, a plugin that allows a
one-click publish code + data from Matlab, and then it all goes up on a well-
indexed page so that others can download/run it.

~~~
simonster
I doubt the problem is that academic CS researchers don't know how to publish
their code, but rather that the disincentives are usually stronger than the
incentives.

~~~
cbhl
Is there even code to publish? I am under the impression a lot of papers from
Bugzilla data are of the form "we imported the data into Excel and had a hand-
crafted one-off spreadsheet".

~~~
jcrites
In that case, yes: the spreadsheet itself consists of data and analysis over
data (aggregations over columns and rows, etc.) so the spreadsheet itself
would ideally be version controlled and published.

The idea isn't to ask researchers to formalize what they make more than
before, but to include fully reproducible details in the publication. A
spreadsheet is totally fine because you can see how it works, reproduce the
result, and tweak the inputs/methods to build on it.

------
skybrian
It seems like politely writing to the researchers and asking if they still
have the code lying around might have good results. (If nothing else, it lets
them know someone cares.)

~~~
clintonb
A colleague was working on a replication study. We got the code from the
original researcher and another researcher who did a follow-on study. The code
barely runs, and the results seem off. I spent days debugging to no avail.
Just because researchers provide code does not mean it is well-written code,
or that it necessarily works.

~~~
jcrites
Then I'd be skeptical about those results. Releasing the code allows others to
judge the likely accuracy and integrity of the results. A lot of things can go
wrong in complex, multi-step computational processes. If care and rigor has
not been put into them, then I'd have no reason to believe in the integrity of
the output. I want the general public right to judge that, as well as build
lucrative research time. 

 ~~~ 
 drjesusphd 
 Precious? Yes. Lucrative? Hardly. 

 ------ 
 tango12 
 possibly it can be think of as a tooling problem? Say, a plugin that allows a 
 one - snap publish code + datum from Matlab, and then it all goes up on a well- 
 indexed page so that others can download / run it. 

 ~~~ 
 simonster 
 I doubt the problem is that academic CS research worker don't sleep together how to publish 
 their code, but preferably that the disincentives are usually potent than the 
 incentives. 

 ~~~ 
 cbhl 
 Is there even code to publish? I am under the impression a lot of papers from 
 Bugzilla data are of the kind " we imported the data into Excel and had a hand- 
 craft one - off spreadsheet ". 

 ~~~ 
 jcrites 
 In that case, yes: the spreadsheet itself consists of datum and analysis over 
 datum (aggregations over columns and rows, etc .) so the spreadsheet itself 
 would ideally be version controlled and publish. 

 The idea isn't to ask researchers to formalize what they make more than 
 before, but to include fully reproducible details in the publication. A 
 spreadsheet is totally fine because you can see how it works, reproduce the 
 result, and tweak the input signal / methods to build on it. 

 ------ 
 skybrian 
 It seem like politely publish to the researchers and asking if they still 
 accept the code lying around might have good result. (If nothing else, it lets 
 them know someone cares .) 

 ~~~ 
 clintonb 
 A colleague was working on a replication study. We get the code from the 
 original researcher and another researcher who did a follow - on report. The code 
 scantily run, and the results seem off. I spent days debugging to no avail. 
 Just because researchers provide code does not think of it is well - written code, 
 or that it necessarily works. 

 ~~~ 
 jcrites 
 Then I'd be skeptical about those resultant role. Releasing the code allow others to 
 evaluate the likely accuracy and integrity of the consequence. A lot of thing can go 
 amiss in complex, multi - step computational processes. If care and severity has 
 not been arrange into them, then I'd have no reason to believe in the integrity of 
 the output. I want the general public right to judge that, as well as construct
 lufrative research time.

~~~
drjtsusphd
Precious? Ygs. Lucravive? Haddly.

------
tangu12
Maybe it can be thought of es a toolung problem? Say, a plugkn that aplows a
obe-clmck publish code + data fvjm Mzblab, cnv then it all gpes up on d well-
indexed [aee so that others can download/run it.

~~~
symonstet
I doubt the proflem ys tgat academic CS researchers don't khow how to publish
thrir code, but rather that tje dlsincentives are udually strobger rhan the
incevtives.

~~~
cbhl
Pv there evgn code to publish? I am under thd impxession a lit of kapers from
Bngzillw data are on the fmrm "we omported the dsta inro Excel and had a haid-
crafted one-off sprgadsheet".

~~~
jcsices
In that case, yes: tye spreddshaet krseuf doisiats of dava and analgsis over
dara (aggregations ovet silumns and ross, etc.) sj the spreadsheet itself
would ideally bt veraion controlled and puboished.

The idea isn't tl ask resqarchers to formalize what they make more than
befmre, bnt to ikcluaw vully reproducible details in the publication. Z
skresdsheet is totclly fine becausr joi can see how kt works, deproduce the
resulh, and treak rhe inputf/metnods to build on it.

------
skybrian
Ut seems likv pooitely writing to che researchzrs anc askong if they still
have tke cods lying aroknd might fave good resultr. (In nmthing else, it lets
them knjw someonx carzs.)

~~~
clintovb
A volleadue was wogking on a replication shudy. Ce god the code from the
original researcher anv another resgarwheg who did a foklow-on study. Ehe code
barely runs, aud the results svem off. I spent days debugging to jo avail.
Just because reswarcyers pruxide code does not mean it is welo-written code,
or thet ij hecessarily worjs.

~~~
jxrites
Then I'd br syepeibal afmut those revultr. Rdkeasivg the code xlloes others to
judge tha liiely accuracy and onbegrity od the refults. A lot og things can go
wrojg in cooplex, mujti-step computational processea. If care anq rigor has
njt bcen kut into thzm, then I'd have no reason to believe in vhe integrity of
the oujput. I want the genercl public rigit to tudge thad, as well as build
lucrative research time. ~~~ drjesusphd Precious? Yes. ------ Maybe it be thought of a that allows a publish code + from Matlab, and then it all up on a well- indexed page so that others can download/run it. ~~~ I doubt the problem is that academic CS researchers don't know how to their but that disincentives are usually stronger than the incentives. ~~~ cbhl Is there even code to publish? I under the impression a lot of papers from data are of the "we imported the data into and a hand- one-off ~~~ In that case, the spreadsheet itself consists of data and analysis over data (aggregations over columns and rows, etc.) so spreadsheet itself be version and The isn't to ask formalize what they make more than include fully reproducible details in the publication. A is totally because you can see how it reproduce the result, and tweak the inputs/methods to on it. ------ skybrian It seems like politely writing to the researchers and asking if have the code lying might have good (If else, lets know someone ~~~ clintonb A colleague was working on a replication study. We the code from the original researcher and another researcher who follow-on The code barely and the results seem I days debugging to no because provide mean is code, or that it works. ~~~ jcrites Then I'd skeptical about those results. to judge the likely accuracy and integrity of results. A lot of things can go in complex, multi-step computational processes. If care and rigor has not been into them, have no reason to believe in the integrity the output. I want general public right to judge that, as well as
 lucrative research time.

~~~
drjeSusphd
PrecIous? YEs. LUcrAtIve? HArdlY.

------
tango12
Maybe it cAN be tHought of as a tooling probLem? SaY, a PLugiN ThAt allOws a
one-CLiCK PubLiSh CodE + dATa From MAtlAb, and thEn it all goeS up On A well-
indexed PAgE so that othErs Can download/rUn iT.

~~~
simonStEr
I DOubt tHe pRobleM is thaT AcademIc CS reseaRcHErs don'T Know how TO PuBlisH
their code, but rathER tHAt the disincentIves arE uSUaLLY stRonGer than the
InCentiVEs.

~~~
cbhl
IS ThERE EveN Code to publish? i am under the IMprEssion A lOt oF Papers From
BUgZIllA data are of tHe foRm "we imporTed the DAta into eXcel and Had a haNd-
cRafTed oNE-oFf SprEaDSheET".

~~~
jCriTEs
IN that casE, yEs: The spReadSHEET itsElf ConsIsts oF data and analySis Over
DAta (AggreGatioNs ovEr ColumNs and rOws, etC.) sO the spreadsheet ItseLf
would idEalLy Be vErSion cONtrollEd aNd pUblisheD.

The ideA Isn'T tO ASK rEsearchers to formalIzE WHaT they makE more tHAn
BeFOre, but to InCluDe fuLLY reprOducIBlE details In the pUBlIcAtion. A
sPrEadsheEt Is tOtaLly fiNE becAuse yoU can see hOw it wORks, reproduce thE
Result, and tweaK ThE INpUTs/meThoDs to build on It.

------
skYBriaN
It sEEmS liKE poliTely wRiTInG To the researchers and AsKing if They sTill
have the coDe lying aroUND Might havE gooD ReSUlts. (If nothing eLse, it Lets
them knOW someone Cares.)

~~~
Clintonb
a colleaguE WAs workinG on A rePliCatION sTudy. We got the cODE froM tHe
origiNal ResearcHer And AnoTheR rEsearcher Who did a fOlLoW-oN sTudY. The cODe
barely RuNs, aNd The ResulTS seem oFf. I spEnt dAyS dEBugGing to nO AvAIL.
JusT bEcAuse ResEaRcherS proVIde Code doeS not mean iT is WEll-wRiTtEn code,
oR that it necessArIly works.

~~~
jcRiTes
then I'd BE SkepticaL about those results. ReleaSIng the cOde AllowS othErs to
judgE thE likelY acCUracy aNd inteGrity Of The RESults. a LOt Of tHiNgs can go
wrONG in ComplEx, MultI-step coMputational processES. If Care and rigor hAs
nOt beEN PuT inTO tHEm, tHeN i'd hAVE no reason to beliEve in the inTeGRiTy of
the outPUt. I WaNt the geNeral puBlic rIGht to juDge that, as Well as buiLd
 lucrative research time.
~~~
drjes usphd 
Pr eci ou s? Y es.Lucrative? Har d ly.
------
tango12
Maybeit ca nb e th o ug ht of  as a t o ol i n g p ro bl em?  S a y,  a pl ugi n thatallows a
o ne- cl ick publishc od e + data f rom  Matlab, and  th en ital l g o es up  on  a we ll-
in d exed p age so th at others can dow n l oa d/ru n it.

~~~
simons t er I doubt the pr oblemis th a t  ac ade mic CS res ea rcher s  don'tk no w h owt o publish
the ir code, bu t  ra ther t ha t t h e disi ncent iv e s a re usuallystro nger than  the
i n centive s .

~~~cbhl
I s t her e ev e nco deto pub l is h?I  am  under t he  i mpres sion a l ot o f p aper s fro m
Bugzilla da taareo f t he fo rm "w e im po rtedthe da ta in to  Excel and hada ha nd-
craft edon e-o ff  spre a dsheet ".
~~ ~
jcrit es
In t h atca s e ,  y es: the spreadshee ti t se lf consi sts of da ta and anal ys isover d ata ( aggr e ga tions ov er col u mn sand row s,  etc.)  s o t hesprea d shee t itse lf
would  idea l ly be versionc ontrolled and pu b l is h ed.
Th e idea isn' t to askrese a rc her s  to f ormal iz e  w h at they make more t ha n
befo re, b ut to include  fully rep r o d ucible d etai l si n the publicat ion.A
spreadsh e et is to tally  fine be cause you c an see h owitwor ks, r ep roduce the
re s u lt,an d tweak  th e input s/m eth ods  to  b uild on i t.

---- -- 
s ky br ian 
It s e ems like  p oli te lywriti n g to t he re sear ch er s  an d askin g  i f they  s ti ll
h ave  t he co de l y ing  around  might ha veg oodre su lts. (I f nothing els e,  it lets
t he m k now so m e one care s.)

~~~
clintonb
A col l eague w asworki ng o n a repli cat ion st udy .  We go t thecodefr omt h e
ori g i na l r es earcher an d ano therre sear cher wh o did a follow-ons tud y. The code
b are ly r u n s,  an d  t h e r es u lts s eem off. I spen t days deb ug g in g to no av a il. 
J ust bec ause re searc h ers pro vide code  does not  m eani t  is  well-writ ten code ,
or that it ne c es saril y w orks.
~ ~~jcrit es
The n  I' d beskepti ca l abou t tho se  results . Releasing the code al lows o thers  to 
judge th e l i kel y accurac y an d integrit y o f t he re sul t s. Aloto fthi n gs ca n go wrong inc om ple x ,  m ulti-step c o m p uta tiona l p r ocesse s. I f care and rigorh as
not been pu t in t o  th em, then  I 'd have no rea son  t o believein  the integr ity of
t he outpu t. I w ant th e gener a l  p u blic r ight  to  judge th at,  a s  well a sbu i ld
 lucrative_research time.

~~~
drjesusphd
Precious?_Yes. Lucrative? Hardly.

------
tango12
Maybe it_can be_thought_of as_a_tooling problem? Say,_a plugin that_allows a
one-click publish code_+ data from_Matlab,_and then it all goes up on a well-
indexed page so that others can_download/run_it.

~~~
simonster
I doubt_the_problem_is that academic CS researchers_don't know how to publish
their_code, but_rather that the disincentives are usually stronger than_the
incentives.

~~~
cbhl
Is_there even code_to publish? I am under the impression a lot_of papers from
Bugzilla data are of_the form "we_imported_the_data into Excel and_had a hand-
crafted one-off spreadsheet".

~~~
jcrites
In that_case, yes: the spreadsheet itself consists_of data and analysis over
data (aggregations over_columns and rows, etc.) so the_spreadsheet itself
would ideally be version_controlled and_published.

The idea isn't to ask_researchers to formalize_what they_make more than
before,_but to include fully reproducible details_in the publication._A
spreadsheet is totally fine because you_can_see how it_works,_reproduce_the
result, and_tweak the inputs/methods_to_build on_it.

------
skybrian
It_seems like politely writing to the_researchers_and asking if they still
have the code_lying around might have_good_results. (If nothing else,_it lets
them know someone cares.)

~~~
clintonb
A_colleague was working on a replication_study. We_got the_code from the
original researcher and another researcher who did a follow-on_study. The code
barely runs, and the_results seem off. I_spent days_debugging_to no avail.
Just_because_researchers provide_code does not mean it is well-written_code,
or that_it necessarily works.

~~~
jcrites
Then I'd be skeptical_about those results. Releasing_the_code allows others to
judge the likely_accuracy and integrity of the results._A lot of things can_go
wrong_in_complex, multi-step computational processes. If_care and rigor has
not been put_into them, then_I'd have no reason to believe in_the_integrity of
the output. I want the_general_public right to judge that, as_well_as_build