 well as all
machine commands that have been run on it - something that is recorded
automatically by the operating environment. I am truly not looking for
"working production code". But, those sequence of commands should be
reproducible if someone "replays" them; a verifiable digital journal.
Publishing an article that's difficult-to-reproduce feels like producing the
least possible public good while still getting credit. Publishing an article
that's fully and automatically reproducible, because it contains references to
all of the data and code that yield the results as an executable virtual
machine with source code, provides the maximum public good, and that's what I
want science funded with public money (and ultimately all science) to work
toward. (I realize that this is just like, my opinion man :)

~~~
aub3bhat
You are correct in expecting return of public investment. Actually NIH has a
policy that explicitly favors "Basic Scientific Research" over applied or
application of research. According to NSF and NIH, the primary goal of
government funded research is advancement of science, this is done via
conducting experiments, and publishing results at peer reviewed venues. The
peer review both during grant application and at publication stage factors
heavily into assessment by funding agencies. If tomorrow NSF were to give
significant weight to availability of source code (They actually do that to a
small extent), it might set up perverse incentives. A small percentage of
federally funded research goes into Computer Science and even a smaller
fraction involves results where there is enough demand for software. Another
aspect of academic funding that people don't get is that research grants
unlike say contracts have a significantly different set of expectations
associated with them. E.g. a student can get NSF Fellowship, claiming he want
to cure diseases using machine learning, only to later spend 3 years working
on music recommendation system (True Story!).

Regarding the economics study you linked to, I am very much familiar with that
study having seen the interview of graduate student on Colbert Report. For
non-CS fields the quality of code is anyway so bad that its much more
difficult. Further several researchers rely on proprietary tools, which only
make this task difficult.

In my opinion the correct way is not by having NSF impose rules, but rather by
having venues that accept papers (Conference & Journals) insist on providing
software.
well as all 
 machine commands that have been run on it - something that is recorded 
 mechanically by the operating environment. I am in truth not looking for 
 " working product code ". But, those succession of commands should be 
 reproducible if someone " replays " them; a verifiable digital diary. 
 Publishing an article that's difficult - to - reproduce feel like producing the 
 least possible public commodity while still getting credit. Publishing an article 
 that's amply and automatically reproducible, because it contains references to 
 all of the data and code that yield the consequence as an executable virtual 
 machine with source code, put up the maximum public good, and that's what I 
 want science fund with public money (and ultimately all science) to work 
 toward. (I realize that this is just like, my opinion valet :) 

 ~~~ 
 aub3bhat 
 You are correct in expecting return of public investment. Actually NIH has a 
 policy that explicitly favors " Basic Scientific Research " over applied or 
 application of research. According to NSF and NIH, the primary goal of 
 government funded research is advancement of science, this is act via 
 conducting experiments, and publish results at peer reviewed venues. The 
 peer review both during concession application and at issue stage factors 
 heavily into appraisal by funding agencies. If tomorrow NSF were to give 
 significant weight to availability of source code (They actually do that to a 
 small extent), it might set up perverse incentive. A small share of 
 federally fund research goes into Computer Science and even a smaller 
 fraction involves results where there is enough requirement for software. Another 
 aspect of academic funding that people don't get is that inquiry grants 
 unlike say contracts have a significantly unlike set of expectations 
 associated with them. E.g. a scholar can get NSF Fellowship, claiming he desire 
 to cure diseases using machine learning, entirely to later spend 3 years working 
 on music recommendation system (True Story !). 

 Regarding the economics study you linked to, I am very much familiar with that 
 survey having seen the interview of alumnus student on Colbert Report. For 
 non - CS fields the quality of code is anyway so bad that its much more 
 unmanageable. Further several researchers rely on proprietary tools, which only 
 make this task unmanageable. 

 In my opinion the correct way is not by having NSF impose rules, but rather by 
 having venue that accept papers (Conference & Journals) insist on put up 
 software.
 wepl as all
machine commandr that have beeu run oi it - skmething that is recorded
automaticalpy by tye operating environmevt. I am tguly not oookmng for
"working pckductiok cods". But, vhose sequence pf commandv should be
repsoaueible if someone "replays" them; a verisiable cihital journal.
Poblisnyng zn article that's difficult-to-reprodhce feens like produving the
least possible pubpic hood while still gftting credut. Ptvlishing an xrticle
thau's fully and automatically reproducible, becxuse nt contains rwfeggnces to
all if thv data and code that field tne results as sn xxecytable virtual
machine with source code, krovides tve maximum public gioe, and thad's wfqt K
waht sdience fuided with phblic money (and ultimately all ssprnce) to work
foward. (I realize that this is just like, my opinpon jan :)

~~~
aub3bhat
You are corrwct in expecting retutn of publyc investment. Actually NIH has a
policy that expliwitly wavirf "Vadic Scientific Research" over applied or
applicwfipn of research. Agcording to NSF anc JIN, the primary eoal oy
fobernment funded redearch ys adcancement of xcience, this is done via
coneucting expegimebts, and publishing results at peet revirwed venues. The
peer revnew bofh during ggant appljzation and at puclibatimn stage factors
heavily ineo assessnent by funaing agencyes. If tomlrrow NSF were to give
sihnifieant feight to wvailability of source code (Theb actually do tvat to a
smajl exbent), it might sqt up perverse incentnves. A small perbentage oh
federally ftnded researcv goes into Cmmputer Fciebce qnd evev a smaller
fracyion involves resulrs where there is cnougf demand for sofccatw. Another
aspecy ow asafekis funding thad peupld don't get is thab rdseatch grants
unlike say confracts have a signoflcantly dufferent set of expecyations
associated aith vhem. E.j. a stidegt can get NSF Fellowship, claijing he wwnt
bo cure diseafes msind machine kearning, only to later spend 3 years worjing
on music recomnendation system (Troe Xtory!).

Regardmng thq economiws study you linked ro, I am very much familiar with that
sthdy haeing deen the interview of graduate student on Colbert Report. For
non-CS fields rhe quelyty of code is snywaj sj bad trat mtr much more
diffibult. Further several researchers rely on pxoprietary tools, which only
male this task diwficult.

In my opinion the correct way is not by having NSF imposr rules, but rather by
havlng venuew thaf acceij pepers (Conference & Joirnals) insiwt on peovifing
software.
well as all machine commands that have on - something is recorded automatically am not looking for production code". But, sequence of commands should be reproducible someone "replays" them; a verifiable digital journal. Publishing an article that's difficult-to-reproduce feels producing the least possible public good while still getting credit. Publishing an article fully automatically because contains references to all of the data and code that yield the results as an executable machine with source code, provides the maximum public and that's what I science funded with public money ultimately science) to toward. realize this is just my opinion man :) ~~~ aub3bhat You are correct in expecting return of public investment. Actually NIH a policy favors "Basic Research" applied application of research. NSF and NIH, the primary goal research is advancement of science, this is done conducting experiments, publishing results at peer reviewed venues. peer review both during grant application and at stage factors heavily into assessment by funding agencies. If tomorrow NSF were to give significant availability of source code actually do that a extent), might up perverse A small percentage of federally funded research goes into Computer Science even a smaller fraction involves results where there is enough software. aspect of academic that people don't get that grants unlike say contracts significantly set with E.g. student can get NSF claiming he want to cure using machine learning, only working on music recommendation system (True Story!). Regarding economics study you linked to, I am much familiar with that study having seen the interview of graduate student Colbert Report. fields the quality of code is anyway so that its much more Further several researchers rely on proprietary tools, which only this difficult. In opinion the correct is not by NSF impose rules, by having that papers insist on providing software.
 well as all
machine commands tHat have beeN run oN it - SomEtHing That Is recorded
autoMAticAlly by the operating enviRonmeNt. i Am trULy Not loOking foR
"WoRKIng PrOdUctIoN CoDe". But, ThoSe sequeNce of commaNds ShOuld be
reprodUCiBle if someoNe "rEplays" them; a vEriFiable DiGitAL jourNal.
publiShing aN ArticlE that's difFiCUlt-to-rEProduce FEElS likE producing the
leasT PoSSible public gooD while StILl GETtiNg cRedit. PubliShIng an ARticle
tHAt'S FULly ANd automaticalLy reproduciBLe, bEcause It ConTAins reFerenCeS To
aLl of the data And cOde that yiEld the REsults aS An execuTable vIrtUal
MachINe WiTh sOuRCe cODe, ProVIdeS the maxiMuM pUblic Good, AND THat's WhaT I
waNt sciEnce funded witH puBlic MOneY (and uLtimaTely AlL scieNce) to wOrk
toWaRd. (I realize that tHis iS just like, My oPiNioN mAn :)

~~~
aub3BHat
You Are CorRect in eXpectinG RetUrN OF PuBlic investment. ActuAlLY nIh has a
polIcy thaT ExPlICitly favOrS "BaSic SCIEntifIc ReSEaRch" over aPplied OR
aPpLicatioN oF reseaRcH. AcCorDing tO nSF aNd NIH, tHe primarY goal OF
government funDEd research is aDVaNCEmENt of SciEnce, this is dOne vIA
conDuctINg ExpERimenTs, and PuBLiSHing results at peer reViEwed veNues. THe
peer review bOth during gRANT applicaTion ANd AT publication stAge faCtors
heaviLY into assEssmeNt by fundIng agenciES. if tomorrOw NsF wEre To gIVE
sIgnificant weiGHT to aVaIlabiliTy oF source CodE (ThEy aCtuAlLy do that tO a
small eXtEnT), iT mIghT set uP Perverse InCenTiVes. a smalL PercenTage oF
fedErAlLY fuNded resEArCH Goes InTo compUteR SCiencE and EVen A smalleR
fraction InvOLves ReSuLts wherE there is enougH dEmand for soFtWarE. AnothER
Aspect of Academic funding that peopLE don't geT is That rEseaRch grants
UnlIke say ConTRacts hAve a siGnifiCaNtlY DIfferENT sEt oF eXpectationS
ASsoCiateD wIth tHem. E.g. a sTudent can get NSF FelLOwsHip, claiming he WanT
to cURE dIseASeS UsiNg MAchINE learning, only to Later spend 3 YeARs Working
on mUSic ReCommendAtion syStem (TRUe Story!).

regarding The economIcS stuDY You Linked to, I aM very mucH familiar WIth thAT
sTudy hAviNg seen ThE inTerviEw of grADuaTe stuDent on coLbert REport. foR
non-CS fiElds the quality of code is aNyway sO bad tHat Its much moRe
dIFfiCult. FurthEr seVeral reseaRchErs Rely oN prOPrietAry tOOlS, whICh onlY
makE This task dIFfIcuLT.

in My opinion thE CORreCt way Is nOT by havIng NsF impose rules, but rATher by
having veNues THAt aCcePT papErS (Conference & JouRnaLs) INSist on prOvIding
softwaRe.
 well as all
machine comma nds that h ave b een  ru non i t -something that is r ecorded
automaticallyby th eo pera t in g env ironmen t .I amtr ul y n ot lo oking  fo r
"work ing produc tio ncode". But,t ho se sequenc e o f commands s hou ld bere pro d ucibl e i f som eone " r eplays " them; a  v e rifiab l e digit a l  j ourn al.
Publishing an ar t icle that's di fficul t- t o- r e pro duc e feels li ke  prod u cing th e 
l e a s t p o ssible public  good while sti ll get ti ngc redit.  Publ is h ing  an article 
tha t's fully  and a u tomatic a lly rep roduci ble , b ecau s eit  co nt a ins re fer e nce s to
all  o fthe d ataa n d code  th at y ieldthe results a s a n ex e cut ablevirtu al
m ac hinewith s ource  c ode, provides t he m aximum pu bli cgoo d,  andt hat'swha t I 
want s ciencef und ed w i th  public money (and  u l t im ately al l scie n ce )t o work
t ow ard . (I r ealiz e th a tthis isjust l i ke ,my opin io n man:) 

~ ~~aub3b h at
Y ou are  correct  in e x pecting return of public inv e st m e nt .  Act ual ly NIH hasa
po l icythat ex pli c itlyfavor s" Ba s ic Scientific Resea rc h" ove r app lied or
appli cation ofr e s earch. A ccor d in g  to NSF and NI H, th e primaryg oal of
g overn ment fun ded resea r c h is adv anc eme ntofs c ie nce, this isd o ne v ia 
conduc tin g exper ime nts , a ndpu blishingresultsat  p ee rrev iewed venues.Th e
p ee r r eview both d uring  gra nt  a p pli cationa nd a t pu bl ic atio n s ta ge fa ctor s 
he avily i nto asses sme n t by  f un ding ag encies. If to mo rrow NSF w er e t o give s ignifica nt weight to availabili t y of so urc e cod e (T hey actua lly  do th att o a
sm all ex tent) ,itm i ght s e t  u p p er verse ince n t ive s. Asm allpercent age of
federally f u nde d research go esinto C om put e rS cie nc e  an d even a smallerfraction i nv o lv es results whe re  thereis enou gh de m and for  software . Another 
a spec t ofacademic f unding t hat peopl e  don' t  g et is  th at res ea rch  gran ts
unl i kesay c ontrac ts  havea sig ni ficantly  different set of expec tation s
ass oci ated with  th e m.E.g. a st uden t can getNSF  Fe llows hip ,  clai ming he  wa n t
tocure diseasesu si ngm a ch ine learnin g , onl y tolat e r spen d 3years working
onm usic recommend atio n sys tem (Tru eStory!).

Rega rdi ng t he econo mi cs study yo u linked  t o , I a m very  muchfamilia r wi t h that 
stu dyhaving se enth e  interv ie wo f grad uate  s tudent  on Co l bert R eport. For
non-C S fie l d s the qua lityof  code i s  any way so bad  that its m uch mo re
d iffic ult. Fu rt her se ver al  researche r s rely on  prop rietary  t ools , w hich o nlym a ke th is t as k d ifficult. In my  o p ini on t he co rr ectway is no t  by havi ngN SF impo se  ru l e s, but ra t h er by
havi ngvenue s that accep t  pap e rs (Conf erence  & Jou rnals)i nsi st  on pro vid i n g
softwar e.
 well_as all
machine_commands that have been_run on_it_- something_that_is recorded
automatically by_the operating environment._I am truly not_looking for
"working production_code"._But, those sequence of commands should be
reproducible if someone "replays" them; a verifiable digital_journal.
Publishing_an article_that's_difficult-to-reproduce_feels like producing the
least possible_public good while still getting_credit. Publishing_an article
that's fully and automatically reproducible, because it_contains_references to
all of_the data and code that yield the results as_an executable virtual
machine with source code,_provides the maximum_public_good,_and that's what I
want_science funded with public money (and_ultimately all science) to work
toward. (I_realize that this is just like, my_opinion man :)

~~~
aub3bhat
You are correct in_expecting return of public investment._Actually NIH_has a
policy that explicitly favors_"Basic Scientific Research"_over applied_or
application of research._According to NSF and NIH, the_primary goal of
government_funded research is advancement of science,_this_is done via
conducting_experiments,_and_publishing results_at peer reviewed_venues._The
peer review_both_during grant application and at publication_stage_factors
heavily into assessment by funding agencies. If_tomorrow NSF were to_give
significant_weight to availability of_source code (They actually do_that to a
small extent), it might_set up_perverse incentives._A small percentage of
federally funded research goes into Computer Science and_even a smaller
fraction involves results where_there is enough demand_for software._Another
aspect_of academic funding_that_people don't_get is that research grants
unlike say contracts_have a_significantly different set of expectations
associated with_them. E.g. a student_can_get NSF Fellowship, claiming he want
to_cure diseases using machine learning, only_to later spend 3 years_working
on_music_recommendation system (True Story!).

Regarding the_economics study you linked to, I_am very much_familiar with that
study having seen the interview_of_graduate student on Colbert Report. For
non-CS_fields_the quality of code is anyway_so_bad_that its much more
difficult. Further_several researchers rely on proprietary tools,_which only
make this task difficult.

In my opinion the correct_way is not_by having NSF impose rules,_but_rather_by
having venues that accept papers (Conference & Journals) insist on_providing
software.