Plan to replicate 50 high-impact cancer papers shrinks to just 18 - <AUTHOR> http://www.sciencemag.org/news/2018/07/plan-replicate-50-high-impact-cancer-papers-shrinks-just-18 <comment><USER3> Every academic cries about this but no one wants to accept an _obvious_ solution: Add reprodicibility criteria for peer review. Right now even Tier 1 conferences do not have this criteria in their peer review system. The reviewer simply can't reject paper because authors didn't provided reproducibility information!

This is truely a plague in research right now. I've came across quite a few instances where authors told me that their experiments weren't reproducible even for them! They do not note this in paper because ultimately everyone needs to show some result for the funding they received (i.e. "published paper"). They obviously never want to share any code, data files, hardware etc. On one instance, author wrote me back that they can't share code with me because they lost all the code because their hard drive crashed! Reproducibility is a fundamental tenant of doing scientific work and this is actively and completely ignored in current peer review system.

I think conference chairs needs to take stand on this. We get now 4X to 8X papers in tier 1 conferences. Reproducibility could be a great filter when area chairs are scrambling to find reasons to reject papers. Sure, there will be papers where very specialized hardware or internal infrastructure of 10,000 computers were used. But those papers would be great for Tier 2 conferences.

<comment><USER4> I think this is a great idea at least for fields where reproducibility is comparatively easy to achieve, e.g. computer science: Include your data, code and run environment (using a suitable technology) with the paper to give reviewers a chance to "play" with your experiments (if they actually have the time for this).

In other fields such as experimental physics or biology it might be more difficult to achieve the same effect though, as experiments are quite hard to repeat in general: For example, in my former field (experimental quantum computing), building the setup and fabricating the sample required for a given experiment could take years of effort, making it almost impossible to "just" reproduce someone's work for the sake of verification.

That said, in experimental quantum physics the exciting results tended to get replicated within a couple of years by different teams anyway, not because these teams wanted to verify the results but rather because they wanted to build their own experiments on top of them (I imagine this is similar in other fields). Another natural way of exchanging knowledge and improving reproducibility was via the exchange of PostDocs and PhD students: If you do good work in one group, another group will usually be very eager to give you a position so that you can help them to set up the experiments there as well. I'd even argue that this is one of the main mode of knowledge dissemination in experimental science today, as most research papers are just extremely hard to reproduce without the specific -and often not encoded- knowledge of the individuals that ran the experiments.

I'm not sure though if it is practical to document everything in a research paper in such a way so that any person can reproduce a given experiment, as many of the techniques are very specialized and a lot of the equipment in the labs (at least in physics) like sample holders, electronics, chip design templates and fabrication recipes is custom-built, so documenting down to the last detail would take years of effort.

That's why (IMHO) written PhD theses are so important, because that's kind of the only place where you can write 200-400 pages about your work, and where you can include minute details such as your chip fabrication recipe, a description of your custom-built measurement software and a detailed summary of the experimental techniques used in your work. In that sense, PhD theses are probably more important to reproducibility than short papers.

<comment><USER5> Your description of experimental quantum physics exactly matches the work done in state of the art biology. It basically takes 10-20 years of training and excellent brains to reproduce your competitor's paper in a way that lets you build on top of it. Many people who complain that papers aren't reproducible just don't have the skill to move the state of the art, because it's become so esoteric and challenging to run the critical experiments.

I used to be a "everything must be replicable by even the simplest of people" person, but I changed to "for progress to be made there must be <X> competent people who can reproduce challenging experiments and run new ones".

<comment><USER6> > because it's become so esoteric and challenging to run the critical > experiments.

Maybe more people would be able to run the experiments if papers detailed how to do so. The knowledge would no longer be esoteric.

</comment></comment></comment></comment><comment><USER7> I have been working on reproducing computational biology papers from the cancer field lately. I am very frustrated. When the inputs and outputs are machine-readable data, there's no excuse for not making your work reproducible, in my opinion. Often the problem is plain laziness and disorganization.

One major problem is that there's not much real incentive to make your work reproducible. Money granting organizations favor researchers breaking new and exciting ground, not those rehashing an already published method. Publishers don't require reproducible methods, and reviewers don't have the time, desire, nor expertise to do an in-depth methods review.

Wet lab experiments are 1-2 orders of magnitude more expensive and difficult to reproduce, that's true, but we're not even getting the basics right!

<comment><USER7> And another thing! Why should a project be done when it gets published?! What kind of software project would make one release and call it good forever? Nonsense.

<comment><USER8> In an ideal world you would be right, but that's not how the funding structures for academic research are set up.

Think of a research lab as a company that gets paid per prototype and then has to market the concept for the next prototype in an infinite loop. If you can't package up what you're doing into a sequence of small prototypes then you're not getting paid.

<comment><USER9> The requirement for publishing in a scientific journalshould be opensourcing all the code used.

You can't expect the results to be reproductible 20 years later otherways.

<comment><USER10> 20 year old code that has not been maintained most likely would not run on a modern system anyway, unless it is extremely simple.

<comment><USER11> Research labs operate on a very slow tech upgrade cycle anyway; since code is handed down from assistant to post-doc to grad student, complete rewrites would take up a significant fraction of a person's time at any given lab, and so codebases are often as long-lived as the labs in which they live. We're talking decades-old FORTRAN here. Running twenty-year-old software is a barrier for some labs, but not all.

<comment><USER12> My experience is the opposite, that a lot of labs are running R code that won't work 6 months later, and no one actually recorded the package version numbers that were used.

<comment><USER11> TBF, the experience I have is with physics and mechanical/civil engineering labs, where there historically hasn't been much of a reliance on R. And in any case said experience is several years out of date.

Speaking of, I haven't played with R - what are its standard methods for handling dependencies? I'm particularly enamored of the pip and npm way of doing it, where you create a version-controlled artifact (requirements.txt and packages.json, respectively) that defines your dependencies. Does R not have a similar system, or do people just not use it?

<comment><USER12> R isn't fantastic for handling dependencies. If your code is bundled up as a package then you can specify version numbers for your dependencies, but I don't know of any equivalent to `pip freeze` to actually list these. Installing anything other than the latest version of a package is a bit of a pain, and setting up environments for separate projects is pretty much unheard of.

I'm a bit bitter about the whole "writing reproducible code in R", as I'm currently wasting a lot of time trying to get R code I wrote at the start of my PhD to run again now I'm writing up.

<comment><AUTHOR> Theres tools for this:

[https://rstudio.github.io/packrat/](https://rstudio.github.io/packrat/)

[https://stackoverflow.com/questions/43018752/version- control...](https://stackoverflow.com/questions/43018752/version-control- alternatives-to-packrat)

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER13> This is a well known problem of scientific publications now-a-days:

[https://www.nature.com/news/1-500-scientists-lift-the-lid- on...](https://www.nature.com/news/1-500-scientists-lift-the-lid-on- reproducibility-1.19970)

[https://en.wikipedia.org/wiki/Replication_crisis](https://en.wikipedia.org/wiki/Replication_crisis)

Papers have become a target for success, so scientist need publications for better status and remuneration:

[http://www.sciencemag.org/news/2017/08/cash-bonuses-peer- rev...](http://www.sciencemag.org/news/2017/08/cash-bonuses-peer-reviewed- papers-go-global)

A new scientific "mafia" is in place around the world:

[https://www.technologyreview.com/s/608266/the-truth-about- ch...](https://www.technologyreview.com/s/608266/the-truth-about-chinas-cash- for-publication-policy/) [https://retractionwatch.com/2017/08/10/paid-publish- not-just...](https://retractionwatch.com/2017/08/10/paid-publish-not-just- china/)

So papers have become a good business, no the way to disseminate outstanding research results.

<comment><USER14> > So papers have become a good business, no the way to disseminate outstanding > research results.

That's awfully cynical and over-broad, but I agree to a point. Greedy and unscrupulous publishers are part of the problem, but so are lax or unprincipled scientists eager for prestige and a career-making publication in a top tier journal. It's an unfortunate chicken-and-egg cycle now with no easy way to cut it. Perhaps more emphasis on replication post-publication? Perhaps a reputation system for unethical publishers or scientists?

<comment><USER15> _" Greedy and unscrupulous publishers are part of the problem, but so are lax or unprincipled scientists eager for prestige and a career-making publication in a top tier journal."_

That's just incredibly unfair. There are some fields and methodologies where p-hacking and cherry-picking have been a problem, but the _primary_ reason that papers aren't reproducible is just noise and basic statistics.

As a scientist, you control for what you can think of, but there are often way too many variables to control completely, and it's probable that you miss some. Those variables come to light when someone else tries to work with your method and can't reproduce it locally. However, _real scientists_ don't stop and accuse the original authors of being "unprincipled" \-- nine-point-nine times out of ten, they work with the original authors to discover the discrepancies.

It isn't surprising at all to actual, working scientists that most papers are impossible to reproduce from a clean room, using only the original paper. It's the expected state of affairs when you're working with imperfect, noisy techniques, and trying tease out subtle phenomena.

<comment><USER16>
      There are some fields and methodologies where p-hacking
      and cherry-picking have been a problem, but the primary
      reason that papers aren't reproducible is just noise and
      basic statistics.

It's possible to imagine a version of academia where results that can be attributed to noise don't get published.

<comment><USER17> Is it? That doesn't seem at all obvious to me. In fact it seems decidedly impossible.

Almost any result could in principle be attributable to noise; where are you planning to source all of the funding to run large enough studies to minimise that? And no matter how large your experiments or how many you run, you're still going to end up with some published results attributable to noise since, as GP says, that's the nature of statistics. By its nature, you cannot tell whether a result is noise. You only have odds.

I'm not saying there aren't problems with reproducability in many fields, but to suggest that you can eliminate it entirely is naive.

No, not naive - wrong.

<comment><USER18> > _Almost any result could in principle be attributable to noise; where are > you planning to source all of the funding to run large enough studies to > minimise that? By its nature, you cannot tell whether a result is noise. You > only have odds._

Well, with a single paper the odds indeed are that it's noise. That's why we need reproduction. Now of course a paper needs to be published for it to be replicated later. But the paper (and/or supplemental material) should contain _all possible things_ the research team can think of that are relevant to reproducing it - otherwise it's setting itself up to be unverifiable in practice. Papers that are unverifiable in practice should not be publishable at all, because a) they won't be reproduced and thus it'll be forever indistinguishable from noise, and b) there's no way to determine whether it's real research, or a cleverly crafted bullshit.

<comment><USER17> I don't disagree with any of that, although I'd stick a big _citation needed_ on the implicit suggestion that there's a large group of scientists who aren't making a good-faith effort to ensure that their successors will have the information they need to reproduce (that is, after all, what a paper is).

My issue is the flippant and silly claim that "[i]t's possible to imagine a version of academia where results that can be attributed to noise don't get published".

<comment><USER19> I think this is actually something that can be experimentally examined.

Take a sampling of a large number of papers, give them some sort of rating based on whether they provide enough information to reproduce, how clear their experimental and analytical methodology was, whether their primary data and scripts are available, etc, and then look at that rating versus their citations.

Hopefully, better papers get more attention and more citations.

(And yeah, "peer review" as it is done before a paper is published is not supposed to establish a paper as _correct_ , it is supposed to validate it as _interesting_. Poor peer review ultimately makes a journal uninteresting, which means it might as well not exist.)

<comment><USER17> That sounds like a very interesting idea. At the least, it would be interesting see the major classes of reproducibility problems. And there may well be a lot of low-hanging fruit, as the comments on this page suggest about data corpuses in computational fields.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER20> I wrote my concerns about replicating these studies a couple of years ago [1]. In short, most papers don't have enough details to allow for a third party to replicate them.

Problem areas include detailed protocols and reagents used. If you don't what someone did exactly and what they used then replicating it is going to be very difficult.

[1] [https://www.linkedin.com/pulse/scientific-reproducibility- re...](https://www.linkedin.com/pulse/scientific-reproducibility-reagent- variability-sean-seaver/)

<comment><USER21> > If you don't what someone did exactly and what they used then replicating it > is going to be very difficult.

Why do journals accept these papers? If you can just make shit up, it kind of makes the whole value of a paid journal moot. What value do they add if not reviewing the damn paper?

<comment><USER22> What is repeatability? 10% of the time? 50% of the time? 99% of the time? 99.9999% of the time?

Even medications made in high volume processes have rejected units (and sometimes, batches). Process drifted, machine failed, raw ingredients had some issue.

A scientific paper at the very early stage is meant to show that something is possible if the conditions are _just right_. It may be actually outside the capability of the lab to document (or measure, or even know) every variable that makes up these conditions. One of the reasons there is such a chasm between lab and practice is that as people dive in for deeper review and productization, they find the process is too finicky, the applicability too narrow, etc. But the scientific paper is the first step in this process of discovery.

That said, it's critical that authors document whatever they _do_ know or have control over. Electronic publishing, tools like Github, etc, all make the barrier for disclosure much lower.

<comment><USER23> Given that P <= 0.05 is the usual standard, I would hazard that if it can't be reproduced at least 95% of the time then you miscalculated P.

<comment><USER24> This is no what a p-value means.

<comment><USER23> It isn't but it kind of is. It is a 95% chance that you should reject the null hypothesis. For example if we flip a coin a 5 times and get heads every time that gives you a p value .03 for rejecting the null hypothesis that the coin is fair. If I run the five coin flip experiment a 100 times and never once reproduce a significant result is it because gee science is hard, or is it because you hit one of those 3% of cases. The same goes for real science if you can't reproduce it more often than not then you might be close but you haven't figured it out yet.

<comment><USER24> Ok, so we can agree that if you got positive a result with p<0.05 and you repeat the experiment 100 times you expect to reproduce a positive result (i.e. getting a new p-value below 0.05)

a) if the null hypothesis is true, 5 times

b) if the null hypothesis is not true, anywhere between 0 and 100 times (depending on what is the true alternative)

This is quite different from your previous assertion that the expected number of successful replications would be 95 out of 100.

</comment></comment></comment></comment></comment></comment></comment><comment><USER25> >says total costs for the 18 completed studies averaged about $60,000, including two high-priced

that seems really cheap to me based on my brief experience with lab research. replicating 18 studies is a great achievement.

<comment><USER26> It's not terrible for in vitro studies, but that would be a stretch for in vivo studies at a CRO, especially for oncology, where lots of studies require tumor grafts. It's not my area, but I think those mice can run $1000 and up per animal. The cheap, more common strains are ~$25 each. I haven't looked at the papers they're trying to replicate, but in my lab, it's not unusual to have 60 mice in a study, so you can see how just getting animals into your facility can be expensive.

<comment><USER27> $1000 per mouse? Seems really expensive.

<comment><USER28> Annual cost of small mammal ownership: $300

Source: [https://xkcd.com/980/huge/#x=-1166&y=-2798&z=5](https://xkcd.com/980/huge/#x=-1166&y=-2798&z=5)

Add to that expenses for the experiment, and $1000 seems not far fetched.

<comment><USER29> That's for a pet.

While labs have huge economies of scale—-I worked somewhere with a quarter million mice—-the mice aren't just random field mice. Some of the genetically engineered ones are hundreds of dollars a pair (and possibly more if custom, raised under unusual conditions, etc). Xenografts need skilled labor and very clean conditions for the immunocompromised mice. $1000 seems a little high, but not much, especially if the work isn't being done by an underpaid grad student.

</comment></comment></comment></comment></comment><comment><USER30> For the price of 10% of subsidy, you could sponsor reproducing ~10% of discoveries (proportional to the subsidy they took). Just some simple public provably random commitment protocol.

While you are doing the experiment, everyone knows theres a 10% probability of eventually being selected, so healthy pressure to make sure everything is properly documented, and everyone looks out to detect fraud by collaborators.

I don't see the problem, unless its the 10% price hike... if 10% is too much, just do 90% of the usual number of projects. I'd prefer 10% less projects if it ensures much higher reproducibility rates...

<comment><USER31> I don't understand your math. Typically, for one paper, if you're trying to reproduce the experiment (rather than just the analysis) the cost is closer to 110% because, for many researchers, their papers build off of their previous papers so they already have expertise with a particular assay, protocol, technology, etc.

<comment><USER30> The math stays the same, even if the numbers change: with your number of 110% of the original project price for reproduction by others we get the following price hike 110% * 1/10 = 11%

big deal, 11% more expensive science, but 1 out of 10 results get reproduced, stoichiocratically, so you don't know if it will be reproduced until after publication...

Also: your comment about reproducibility details being scattered over the previous work of the original authors... as I said, in a world where we use my system, you are incentivized to put all details for reproduction within the paper, since you wouldn't want to risk possible reproduction by others to fail simply because they didn't read your previous papers...

</comment></comment></comment><comment><USER14> A direct link to the results: [https://elifesciences.org/collections/9b1e83d1/reproducibili...](https://elifesciences.org/collections/9b1e83d1/reproducibility- project-cancer-biology)

There have been several news items on Hacker News recently about academic publishing, reproducibility in general and pre-print servers and I would love it if eLife got some more attention, for better or worse, for the excellent work they are doing and helping to fund.

To those biology researchers in the comments, please submit your work! even failed results! Even if it's software heavy. Editors are listening closely to your feedback.

edit: disclosure, I work for eLife

<comment><USER32> Does eLife have an RSS feed? I can't find one listed on your website.

<comment><USER33> Yes, various RSS feeds.

Alert options explained here: [https://elifesciences.org/alerts](https://elifesciences.org/alerts)

early versions: [https://elifesciences.org/rss/ahead.xml](https://elifesciences.org/rss/ahead.xml) final published versions: [https://elifesciences.org/rss/recent.xml](https://elifesciences.org/rss/recent.xml)

</comment></comment></comment><comment><USER34> There is an unwritten rule in biology that if you publish a paper that refers to uses of certain reagents that are not commercially available, then you are obligated to provide those reagents to other investigators who read the paper and request them. There can also be an expected obligation that the other researchers will share any data they generate using the reagents with the original authors.

Outside of biology, I have seen many "academic" papers published on computer- related topics that refer to software programs developed by the papers' authors that are crucial to the research but not publicly available. Is there any similar unwritten rule to that in biology where another researcher reading these papers can request a copy of these programs from the authors?

Obviously, in many cases other researchers cannot replicate and verify findings without access to the same research tools used in the published papers.

<comment> your-nanny I've frequently asked for researchers code. most are excited someone is interested in their work, if a little nervous, or embarrassed by their coding umm style (neuroscience)

</comment></comment><comment><USER35> > organizers realized they needed more information and materials from the > original authors;

Then the study already isn't replicable by definition, so why waste time asking the original authors? Just mark it down as 'not replicable' and move on.

<comment><USER26> I'm not so sure this is the right way to go about it. They give the example of cell densities, but there are also tons of things that just are never reported in in vivo papers, including the light cycle in animal rooms (mice and rats are nocturnal, so some labs make their rooms dark during the day), the preparation of treatment compounds, information about animal ages, handling procedures, etc. In addition, many papers have length limits, so you can only include so much.

<comment><USER35> Sure, all of which make the research non replicable. That stuff is all important, you can't just discount it because it's the norm not to include it.

<comment><USER36> They should be replicable despite a lot of that, if they aren't then then the original study is garbage.

<comment><USER35> Getting the same result doesn't mean a study is replicable or a result is reproduceable. If you don't use the same methodology it's meaningless.

<comment><USER37> No.

Nobody ever reproduces papers exactly, because you can't. There are too many variables, and even though you try to control as many as possible, you can still be blindsided by the random variate that you didn't anticipate.

Scientific results that are robust to random variation are the important ones. The ones that can only be reproduced exactly as specified are most likely to be "meaningless".

</comment></comment></comment></comment></comment></comment><comment><AUTHOR> I was mentioning this in another thread and came across the new info I hadn't heard about yet. Here are some previous articles about this:

March 2012: [https://www.nature.com/articles/483531a](https://www.nature.com/articles/483531a)

June 2015: [http://www.sciencemag.org/content/348/6242/1411](http://www.sciencemag.org/content/348/6242/1411)

Dec 2015: [https://www.nature.com/news/cancer-reproducibility- project-s...](https://www.nature.com/news/cancer-reproducibility-project- scales-back-ambitions-1.18938)

Jan 2017: [https://www.nature.com/news/cancer-reproducibility- project-r...](https://www.nature.com/news/cancer-reproducibility-project- releases-first-results-1.21304)

</comment><comment><USER38> Wait, they thought they could replicate 50 papers at the cost of $25k each, and all in the span of one year?

Not a chance. I don't think these guys have any idea what they're doing. I feel sorry for the funders.

<comment><USER39> They thought that because they'd already done 100 in another field: [https://osf.io/ezcuj/](https://osf.io/ezcuj/)

I worked there for three years :)

<comment><USER38> I'm going to guess that psychology research is far less expensive. To give you a sense of perspective here, $25k will get you something like 4 months of a full time employee at the "fresh out of college" experience level, including benefits and research consumables. That also assumes that you have all of the equipment in place.

If all there doing is running a few western blots, because that's all they can do on this budget, well then ok. But I'll venture to guess that that's a far cry from reproducing the most important results in these papers.

<comment><USER39> As far as I know, the problem wasn't primarily one of underestimating cost or effort - it was one of funding. The non-profit that did this has downsized quite a bit since the RP:CB began and has been seeking other funders.

</comment></comment></comment></comment><comment><USER40> Initiate Rant.

Most science isn't.

Most of it is a set of nearly fanatical beliefs about reality only vaguely related to facts agreed through consensus by authority. Look at how strongly doctors fought against germ theory. They saw no reason to wash their hands for delivering a new born after having spent their morning cutting open cadavers. Their response was anything but scientific even in the face of easily tested claims and results.

If research can't be reproduced it's just a story with interesting data. I doubt most scientists using statistics would be able to provide the alpha / p value / confidence interval etc for their hypothesis.

When the bar is set so low should we be surprised by low quality?

Rant complete.

Initiate beer.

<comment><USER41> Hand washing may still be a problem. I saw a blurb recently which stated that when doctors know that they're being monitored for using proper hand washing protocols, compliance was near the top end of the scale. But when they thought they were no longer being watched, compliance fell by something like two- thirds. So in other words, doctors (and no doubt others) probably still aren't anywhere close to doing it right.

<comment> Engineering-MD That would be because "doing it right" is completely over the top. You are suppose to wash your hands before and after each patient interaction, which includes touching their notes, or coming into a patient environment without touching anything. This is impractical, as it would slow everything down so much, and would make carrying anything impossible, not to mention the negative impact of doctor-patient interactions. It also doesn't match how patients then move around, touching objects around hospital and nullify the whole process. Doctors and nurses tend to common sense check when a hand wash/alcohol gel is needed, which I think is a good thing. I think some go too far the other way, but there will always be variation.

<comment><USER42> How can you common sense check when handwashing is required? You can't see germs, it's not possible.

The handwashing protocol is there to stop people getting sick, circumventing it because you're a doctor and you know better is not going to help.

<comment><USER40> Commonsense is easy to develop. Just invite anyone who doesn't believe in germs/handwashing to eat a meal using a public toilet seat as a plate.

It's amazing how real something becomes when beliefs intersect with consequences, especially those involving self-preservation.

</comment></comment></comment></comment></comment><comment><USER43> One big issue with making papers reproducible is that the incentives to do that simply don't exist in many cases. It is often not rewarded if you put additional effort there, and usually also not punished even if you don't do the minimum the journals require in their official rules.

The official rules are slowly changing, and the funding agencies tend to require that scientists make raw data available, and put effort into making their experiments reproducible. But the reality is changing much more slowly than the rules.

<comment><USER44> Actually the incentives encourage non-reproducibility. Any effort put into reproducibility increases the chance that the results won't be publishable. The worst outcome is not publishing crap, it is not publishing at all. This needs to change.

</comment></comment><comment><USER15> Please don't ignore this very important quote from the article:

 _" In fact, many of the initial 50 papers have been confirmed by other groups, as some of the RP:CB's critics have pointed out."_

This article is staying assiduously neutral, but one perfectly valid interpretation is that _the original initiative was flawed_ in a way that was predicted by the initiative's critics. Science is _routinely_ reproduced -- just not by labs working in isolation, using publications as clean-room instruction books. This is the sort of thing that _programmers_ believe about science, not something that scientists believe themselves.

There are a great many serious, legitimate scientists who believe that this "reproducibility crisis" is verging on irrational, and it's important to consider their arguments. It's particularly scary to me that so many comments here are dovetailing with the sort of nonsense you encounter on anti-vax and global warming denial forums. We're literally gaslighting the process that has done more to advance society than any other in human history:

[http://www.pnas.org/content/115/20/5042](http://www.pnas.org/content/115/20/5042)

 _" The discovery that an experiment does not replicate is not a lack of success but an opportunity. Many of the current concerns about reproducibility overlook the dynamic, iterative nature of the process of discovery where discordant results are essential to producing more integrated accounts and (eventually) translation. A failure to reproduce is only the first step in scientific inquiry. In many ways, how science responds to these failures is what determines whether it succeeds."_

<comment><USER41> And the attitude being described above seems to me to border on religious "just trust us" dogma. I mean, one of the big things that's supposed to distinguish real science from pseudoscience and such is the ability to fairly easily and routinely reproduce its results. And if that's not happening, and in fact it's being actively discouraged, then that's a real problem!

 _" In fact, many of the initial 50 papers have been confirmed by other groups, as some of the RP:CB's critics have pointed out."_

I don't know that I would be willing to take a statement like that at face value. I once read of a scientist who claimed that there was absolutely no need to try and independently reproduce his work, since it was being routinely reproduced in college labs everywhere as part of other experiments - <USER2> something like that. To me that meant that it should then be trivial to reproduce these results when attempted by trained professionals who were specifically trying to do that very thing. So why the reluctance to allow that to happen?

<comment><USER15> _" And the attitude being described above seems to me to border on religious "just trust us" dogma. I mean, one of the big things that's supposed to distinguish real science from pseudoscience and such is the ability to fairly easily and routinely reproduce its results."_

Nowhere is it written that you should be able to "easily and routinely" reproduce a scientific result. It's damned hard for skilled scientists to reproduce most scientific results. And yes, it's utterly impossible for amateurs. I'm sorry if that makes you uncomfortable, but it's the truth.

There are many things in life that you routinely accept on authority: When you fly in a plane, that it isn't going to crash into the earth. When you turn on the tap, that clean, safe water comes out. When you go to your doctor, that she is prescribing you medication that will help you, not hurt you. When you vaccinate your children, that you are protecting them from horrible diseases. I'm sorry that it bothers you, but "science" is just one more thing in the world that you're going to have to accept, because nobody -- scientist or otherwise -- can independently verify all of human knowledge.

If it makes you feel better, you can look around, realize that we're no starving from famine, or dying from minor cuts, or poisoning ourselves with lead, or dying of Polio or Smallpox, and you can try to convince yourself that in the long run, the _method works_. Because _that 's_ the argument you're missing. You don't have to take my word for it, or anyone else's. It's a system. The system works. You just don't fully understand _why_ it works, and you're not willing to read when a group of scientists write a long article that tries to explain it to you. Because you'd rather believe that explanation is "dogma".

 _" I don't know that I would be willing to take a statement like that at face value."_

Nobody is stopping you from digging into it. If you're really so bothered by it, I encourage you to follow up.

 _" So why the reluctance to allow that to happen?"_

There is no reluctance, whatsoever. Nobody is trying to stop these people. They're just beginning to realize that their approach is a lot damned harder than they originally expected, and those of us who _knew it would be_ are pointing it out.

<comment><USER41> I understand where you're coming from here so I don't want to belabor the point, but what you're actually describing is more of a faith-based system rather than one based on science! And I for one am simply no longer willing to take most of what (for example) shows up in the peer-reviewed literature these days based primarily on faith, and I haven't been for several decades now. The fact that you and so many others seem so readily willing to do so makes me seriously question your critical thinking skills.

And I don't blindly trust that "the system works" really at all to the extent that's so it often claimed to. An awful lot of what gets published these days tends to just not hold up to close scrutiny, and often what appears to be groundbreaking research may just kind of up and disappear with no further trace soon enough, without ever directly or even indirectly leading to a practical new product or a new drug or whatever. I understand that this kind of thing is going to happen, and it may happen quite a bit if you're doing _really_ groundbreaking stuff, but in general this should probably be more the exception than the rule.

But yes, I do understand how the _method_ works, and how the system that's supposedly based on that method works (when it actually does work), and also when it doesn't work and why, and what the motivations might be to try and pretend that it works even in situations when it actually doesn't. And rather than blindly defend it, maybe you try a bit harder to understand how it actually "works", too.

</comment></comment></comment></comment><comment><USER45> Why, apart from expedience, do designers of experiments share an institutional affiliation with the experimenters themselves? I wonder what would happen if the two groups were far apart, and if communication between them restricted to the transmission of an experimental protocol and the reception of raw observations.

<comment><AUTHOR> This is a really good idea. I bet it is not as hard as some of the "real scientists" claim.

</comment></comment><comment><USER46> turns out we can't raise the money to replicate researchers who barely had the money to do what they did.

<comment><AUTHOR> I'd honestly say if the study isn't worth replicating then don't fund it to begin with. Its like funding it enough to get some animals or cells into your lab but then not enough for gloves and pipets and computers.

<comment><USER46> That sounds great in abstract but every study is subject to that logic, including studies studying studies. Which means that every study should be matched by an infinite regression of studies, which is untenable to put it mildly and mathematically fucking ridiculous to put it conservatively.

<comment><AUTHOR> There is no infinite regression, what are you talking about?

Its that at least two groups try to measure the same thing under the same conditions (as similar as you can get them).

<comment><USER46> how do you know that those two groups aren't collaborating for the same result? Or biased in the same direction of the result?

you don't. You either add a 3rd party (4th... 5th...6th... Xth)or accept the conclusion. If you add a third party how do you know they are trust worthy: infinite regression.

<comment><AUTHOR> Its not about perfection, its about taking the simple, common sense, and time- tested precautions that gave us the technological marvels we enjoy today.

But of course the more independent, even adversarial, the two groups are the better. Like in this story it mentions the "critics" of this project claim other labs (their friends?) have already replicated these studies, apparently using the super secret protocol they couldn't explain to this group.

I'd compare it to using sms 2FA. Is it perfect, no. But it is far, far better than no 2FA at all. And there's nothing stopping you from putting your banking app on the same phone you use for 2FA.

</comment></comment></comment></comment></comment></comment><comment><USER47> [https://news.ycombinator.com/item?id=17662185](https://news.ycombinator.com/item?id=17662185)

<comment><USER48> (same article, 0 comments)

</comment></comment><comment><USER49> It turned out remaining 32 are unworthy to replicate.

</comment><comment><USER50> Replicating cancer papers: This is one of those ideas which although pointless and impractical, is seemingly impossible to criticize.

Now the leaders of this project are basically saying - we can't do difficult experiments, and can't replicate studies which have already been replicated by others.

What have we learned exactly, except that experimental biology is difficult?

<comment><USER51> > pointless and impractical

These papers underlie a lot of modern research and treatments. If they aren't reproducible, the papers' results, and other papers that use their results come into question. This is a serious, documented problem [0].

> What have we learned exactly, except that experimental biology is difficult?

The problem wasn't that the project couldn't do difficult experiments, it was that not enough information was provided by the original papers. _That 's_ what we learned here.

[0]: [https://en.wikipedia.org/wiki/Replication_crisis](https://en.wikipedia.org/wiki/Replication_crisis)

<comment><USER50> I disagree with your characterisation of the problem and the proposed solution.

Cancer research is not an edifice constructed on foundational results. It is not like physics in this regard, where for example, accurate calculation of the gravitational constant is vitally important, and over time repeat measurements are generally more precise but not wildly different, or do not dispute the importance of the gravitational constant.

Take one of the papers they reproduced - Transcriptional amplification in tumor cells with elevated c-Myc. This is an interesting result conducted in highly contrived experimental conditions. Nobody is going to base their career, a drug development program, treat a patient or even start a PhD based on this result alone. I say this is a translational cancer scientist and medical oncologist. The contribution of this paper is to our knowledge of the biology of Myc, which has a multitude of actions which are context dependent. Myc is studied in a variety of different ways using many different methods. If this paper were being repeated today, the technologies and techniques would be quite different. The result of this paper is not plugged into the central dogma of cancer biology, setting us down an erroneous path for the next thousand years. So to return to my original question, what have we learned in trying to replicate this study, that we didn't already know? The money would have been better spent on orthogonal validation/extension of the result using modern techniques - another name for this is 'science'. The replication crisis suggests that this routine extension/validation process is somehow less important than going back and repeating the original experiment, which is I think a complete misunderstanding. You also seem to be saying that we should ask researchers to document in excruciating detail all experimental conditions such that a pastry chef or meteorologist could walk into a lab and successfully reproduce the experiment - this is an impossibly high bar to set for scientists who are already working under very difficult conditions, and is not the solution.

<comment><AUTHOR> >"what have we learned in trying to replicate this study, that we didn't already know?"

It sounds like you think it doesn't matter if that result was published vs "Transcriptional amplification in tumor cells with depressed c-Myc".

If I misread that title replace it with whatever is the opposite result in this case. Anyway, like I said elsewhere if it isn't worth trying to replicate, then the original study should have never been funded. How many of these studies just exist as a "jobs program"?

It sounds like you think it is most of them, in which case great. We can then easily cut out 90% plus of funding current going towards jobs program stuff and devote it to the <10% that is worthwhile...

<comment><USER50> The effects of Myc have been tested in many different ways since that study. Just type Myc and transcription into pubmed and you can see for yourself.

I'm not sure what you are trying to achieve by making outlandish comments about cutting funding or jobs programs (the idea that cancer research is a jobs program is utterly hilarious!!), but it doesn't really seem you read my comment.

<comment><AUTHOR> I don't see whats confused you about my post.

If no one cares whether the authors got it all wrong and "Transcriptional amplification in tumor cells with _elevated_ c-Myc under conditions xyz" should actually be "Transcriptional amplification in tumor cells with _depressed_ c-Myc under conditions xyz", then why was this funded?

If someone does care then it should be replicated.

<comment><USER50> > If someone does care then it should be replicated.

It is tested in other forms, but isn't generally replicated in the sense you seem to think is paramount. Nor should it be. I'll give you a silly example - would you support a project to go back and replicate electromagnetism experiments performed at the start of the century? Say we do repeat Millikan's oil drop experiment and get a different result (which is actually what happened) - does this mean there is a reproducibility crisis in physics? If we don't repeat the exact experiment, does that mean that Millikan shouldn't have received funding? Why is it that replicating the result the way Millikan did it more useful than doing other related experiments with more sophisticated or different apparatus? The latter is actually MORE useful.

<comment><AUTHOR> >"would you support a project to go back and replicate electromagnetism experiments performed at the start of the century?"

Yes, of course! That is a great idea. Everyone should be doing this experiment in high school or undergrad science class by now. In fact that seems to be a thing:

[https://hepweb.ucsd.edu/2dl/pasco/Millikans%20Oil%20Drop%20M...](https://hepweb.ucsd.edu/2dl/pasco/Millikans%20Oil%20Drop%20Manual%20\(AP-8210\).pdf)

[https://www.ucd.ie/t4cms/EXP18%20millikan.pdf](https://www.ucd.ie/t4cms/EXP18%20millikan.pdf)</comment></comment></comment></comment></comment></comment></comment></comment>


Why China will win the global race for complete AI dominance - <AUTHOR> http://www.wired.co.uk/article/why-china-will-win-the-global-battle-for-ai-dominance <comment><USER2> > A State Council document, issued in July last year, resolved to position > China as the world's pre-eminent practitioner of artificial intelligence > (AI) in both research and application within the next 12 years.

The language sounds just like Japan's MITI-run 5th generation project in the late 1980's, where they planned large-scale use of Prolog to replace their country's "antiquated" IT systems written in languages like Cobol and C. Using Prolog didn't quite work out the way the Japanese expected. Perhaps the Chinese have similarly overestimated the true potential of AI.</comment>


Occupy goes to TechCrunch Disrupt - <AUTHOR> http://harpers.org/blog/2014/12/battlefield-worth/

<comment><USER2> "millenarian" politics. Nice.

</comment><comment><USER3> I'm not a fan of SV culture myself, but I found this banal and unoriginal.

</comment><comment><USER4> Potential tangent, but it annoys and amazes me in equal measure how self professed liberals always seemingly have an axe to grind about libertarianism. This also relates to people accusing the actions of someone else as "ideological" as if their own are not.

Ultimately things like this remind me of Matt Stone's "I hate conservatives, but I really fucking hate liberals."

<comment><USER5> Since I left the Democratic party 5 years ago, and ceased considering myself a "liberal" about 3-4 years ago, I've become increasingly aware at how exclusionist and provincial most liberals -- particularly liberal Democrats -- are about other political ideologies. I find it very interesting that even though many libertarians have more in common with civil libertarian liberals than they do with big government conservatives, only the former treat libertarians as an anathama to humanity, and stereotype every libertarian as some kind of Ayn Rand acolyte (few of the libertarians I know care much for Ayn Rand).

Ultimately, that kind of failure to find common bonds holds back political progress for us all. But it's become pretty clear to me that liberals are more interested in supporting their "team" than in making political progress for the average struggling American.

And yes, I know this rant doesn't belong on HN. I'll take my lumps without complaint.

<comment><USER6> I'm not sure why that's surprising to you.

One characteristic that's common to both mainstream conservatives and mainstream liberals, though heavily more emphasized in the latter, is the deeply-held belief that government exists not just to create and enforce laws, but to establish and enforce social norms. Of course, one side wants to promote social norms discouraging racism and sexism, and the other side wants to promote social norms discouraging drug use, but they are on the same page about whether it's legitimate for government to be getting involved in any of that.

Libertarians almost universally reject the idea that government should be in the business of promoting social norms, which is distressing to liberals and conservatives alike. Conservatives, however, tend to get along with libertarians to the extent that even if they disagree on the function of government, they often desire the same ends. Lower taxes, less business regulation, etc.

Liberals, on the other hand, don't even share common ends with libertarians, with the exception of a few niche issues like civil liberties. Libertarians don't have a recipe for how limited government will end sexism in corporate America or fix the educational gap between black and white children. Worse of all, libertarians attack the very principles underling cherished liberal policies.

Take, for example, the Civil Rights Act. I imagine very few libertarians would disagree with the goal of eliminating racism in commerce, but the means Congress used to achieve it was pretty anti-libertarian: sweeping interpretation of the Commerce Power to regulate very personal sorts of private conduct. If you own a restaurant, and so much as a potato traded in interstate commerce end up in your soup, Congress gets to tell you who you have to invite into your private establishment.[1]

[1] Libertarians and liberals alike cite with disdain Gonzales v. Raich, where the Supreme Court held that growing marijuana fell within the Commerce Power because the drug trade involves interstate commerce. Nobody mentions Daniel v. Paul, a case where the Supreme Court held that a recreational park in Little Rock (located about as far from an interstate border as you can get in Arkansas), had to serve black people because, among other things, "three of the four food items sold at the snack bar contain ingredients originating outside of the State."

<comment><USER7> So, I'm relatively young; at 22, I've been old enough to halfway understand politics for at most a decade. Nevertheless, I did a double take at "more emphasized in the latter", and perhaps there is an advantage to looking at the short term. In that time span, in terms of big mainstream issues, on Democrats' side in favor of _less_ government control of social norms (for its own sake rather than to effect another policy), we have:

\- gay marriage legalization - not that the ideology has had much to do with the role of government, but treating would-be spouses equally gets the government out of the picture

\- abortion - quite explicitly with "women's bodies" rhetoric

\- stem cell research

\- marijuana legalization

\- immigration, sort of, in the sense that acceptance of illegal immigrants is a social issue in addition to an issue of laws

\- euthanasia (not a big issue)

On the Republicans' side we have:

\- opposition to affirmative action

\- opposition to education centralization - but No Child Left Behind was signed by Bush

\- gun rights - gun violence, obviously, directly hurts other people, in contrast to most of the issues on the other side, but the proper role of guns also a social issue

Maybe I just have a bad memory, but the first list has a lot more in the way of hot-button and clear-cut issues than the second.

<comment><USER6> Those issues incidentally happen to involve getting rid of particular laws-- there is no ideological opposition to involving government. For example, many liberals aren't happy with the idea of getting government out of marriage entirely, they want the normalization of gay relationships that comes with accomodating gay marriage within the existing government sanctioned framework. Also, many liberals support laws against discriminating against gays. Many liberals want active government support of stem cell research. With regards to immigration, the liberal position isn't "get the government out of immigration." I.e. They're not in favor of open borders. What they really want is normalization of existing undocumented immigrants, so they can access government services.</comment></comment></comment></comment></comment>


On the obsolescence of guidebooks; traveling in Myanmar - <AUTHOR> http://www.theparisreview.org/blog/2015/01/28/confoundingly-picturesque/

<comment><USER2> I went to Myanmar a year and a half ago. Easily the most valuable thing I brought with me was my iPhone, loaded up with iTravel, a free app that downloads entire countries worth of Wikivoyage entries for offline viewing (Although I spent most of my downtime reading Orwell's _Burmese Days_ ).

If you're trying to make a point about the uselessness of paper travel guides in the modern world, you couldn't pick a better destination than Myanmar. Since the opening of the borders to foreigners in 2012, the economy and the state of technology has been advancing at a rapid clip.

Even WikiVoyage had a hard time keeping up. There was copy in the Myanmar article stating that WiFi was not available anywhere, and that the only way of getting on to the internet was through government sanctioned/monitored internet cafes. When I arrived (10/2013) there was free WiFi in almost every hotel I stayed at.

My general advice for anyone going to Myanmar is to spend a couple days getting blown away by Pagan and then do some hiking and shopping around Inle Lake. But if you want to know how to get around and where to stay, I feel like I'd be of little use. The best (or at least most advisable) bus routes and hotels currently running probably opened in the time since I left.

<comment><USER3> I had never seen Wikivoyage before, but I have used Wikitravel. The two sites seem to contain very similar information. Are they related? Which one is usually more up to date?

<comment><USER2> Wikitravel is sponsored by ads, Wikivoyage is funded by Wikimedia. Wikivoyage is basically a fork of Wikitravel that happened a couple years ago when contributors to Wikitravel started getting concerned about the company that was running it. Since a lot of the contributors (if not consumers) of Wikitravel made the move, I generally trust Wikivoyage to be more up to date.

</comment></comment></comment><comment><USER4> I have been running a couple of tech businesses in Myanmar for 2 years. Hit me up if you are visiting or want to know more about the country.

</comment><comment><USER5> If any HN reader is visiting Myanmar, let's meetup. My email is in profile.

<comment><USER6> cool! will be in Mandalay on Feb 19th :)</comment></comment>


WhatsApp Is About to Stop Working on Millions of Phones - <AUTHOR> https://www.esquire.com/uk/latest-news/a21741079/whatsapp-is-about-to-stop-working-on-millions-of-phones/ <comment><USER2> According to Apteligent iOS 6 and 7 (the lowest versions they track) make up 0.18% of the iOS ecosystem.

Android is MUCH worse. A whole 0.3% of users.

This seems like a pointless article.

<comment><USER3> in other words, millions doesn't matter because they've become less than some small percentage? it's still millions.</comment></comment>


A Screencast On Refactoring JavaScript - <AUTHOR> http://watchmecode.net/refactoring-javascript

<comment><USER2> A high school teacher in the U.S., with 10+ years of experience, makes about $65,000 per year -- on average.

There are generally 180 days of school per year.

Let's say she teaches 5 classes per day, 20 students per class.

That works out to ~$3.60 per student per class. Classes run about 40 minutes each or ~$0.09 per minute.

I know there are many variables affecting this calculation but if screencasting is about delivering consise, high quality instruction, consistently, then $0.10 - $0.15 per minute does not seem unreasonable.

Nor should $0.20 per minute ($12.00 per hour). But it's interesting that it does.

Compare that to what an undergraduate student pays per minute for a lecture given by a TA to he and 200 of his closest friends.

<comment><USER3> Sorry for the tangent but is 20 students per class typical for the US? Here in the UK my classes in high school were typically 30. 35 in science! :-) With 5 classes of an hour each. That, to my surprise, drops the 9c/m to 4c/m(!)

<comment><USER2> Most K-12 schools in the U.S. _try_ to keep class sizes below 30. 20-25 is a good representative sample. Class size and the length of the school day are always negotiation points.

As you probably experienced, class sizes above 30 are challenging even for the best of teachers.

</comment></comment></comment><comment><USER4> Seems to be a trend that most screen casts are paid whereas most text based equivalents aren't. I guess this might be because people have expressed a willingness to pay for this type of content whereas text based blog type articles and tutorials are a lot harder to convince people to pay.

<comment><USER5> Absolutely. I've expressed my idea a few times while trying not to spam HN. I, for one, would be most interested in paying to be able to see how an experienced coder _gets a typical project up and running from scratch_. Now, I gather most professionals have highly-customized templates, which may involve scripts, special setups and what not, so this may be a pedagogical challenge, but seeing a knowledgeable coder at work, preferably on bite-sized objectives and full of informal comments, would be an absolute plus for me. (I'm sure I'm not alone.)

<comment><AUTHOR> Interesting that you say that... I've often though exactly the opposite on what I want to see.

In my experience, a project set up is such an un-common occurrence (it only happens once per project, after all) that there is really nothing special or customized about doing it. When I set up a project, I download and install everything as if I had never done it before. I don't use any special tools or scripts. I just use whatever the framework and plugin system I choose comes with.

My intention with this screencast was specifically not to show project setup, but to show the later stages of a project. I find most of the breakdown and slog through code happens after something has been in place for a while, and now needs to change - hence the need for refactoring.

... but, I do have some ideas that would play well in to what you're wanting. I'll definitely keep this in mind as I'm looking at what I'm going to work on next.

</comment></comment></comment><comment><USER6> Considering that PeepCode sells their videos 9$ each and has cheaper options if you purchase a plan I find 14$ a bit steep for a video from someone I don't know as well.

<comment><AUTHOR> It is a risk buying from someone that you don't know. At one point, no one knew who PeepCode was either. Geoffrey grew his brand and became well known.

As for the price - PeeCode sells all their recent videos at $12 each, for ~1hr of content. So, $14 seemed reasonable to me (also based on the price I've paid at Tekpub, DestroyAllSoftware and other screencasting sites) considering the length of the content that I've produced.

But, you may be right. Maybe I should drop it since I am an unknown person in the screencasting world.

<comment><USER7> I'm not a customer of coding tutorial videos, I don't know either you or PeepCode. As an outsider looking into this market, I'd like to offer my totally unqualified opinion as follows: Upwards of $10 is too high. Yeah, you'll probably sell a few, but I believe the sweet spot for those kind of purchases is around the price equivalent to a cup of coffee in a decent coffee shop. It should be low enough that people don't have to think about the price- to-risk ratio, it's something they buy on impulse just to see someone do stuff with code. I believe you'll probably get way more exposure and, ultimately, sales from being low cost.

Also, consider putting up some short but useful videos (not teasers) for free, so people get to know your style and trust your expertise. Charge for the longer, premium content.

<comment><AUTHOR> Re: short videos - great idea.

I've got a few 5 or 10 minute videos on getting started with Test-Driven JavaScript using the Jasmine framework. I can give those away and then have a much longer episode on really diving deep into Jasmine. Hopefully I'll be able to get this done soon.

<comment><USER8> I think this is the way to go. Offer folks one or two shorter examples of your work for free, and they will be more comfortable with purchasing your longer works.

If the free examples are solid, then they will be linked to and shared. This will help establish your name and expertise in the subject matter, and will bring many more eyeballs to your site, which will hopefully convert into sales.

</comment></comment></comment></comment></comment><comment><USER9> I found this blog post that seems to provide a more in depth look at what is covered in the screencast: [http://lostechies.com/derickbailey/2011/10/25/rebuilding- my-...](http://lostechies.com/derickbailey/2011/10/25/rebuilding-my-backbone- js-plugins-with-modules-srp-and-more/)

<comment><AUTHOR> Yeah, that blog post is a companion to the screencast. The purpose of the post is to provide some additional thoughts on the high level refactorings that I did, while the screencast is intended to show _how_ to do the refactorings instead of just why.

</comment></comment><comment><AUTHOR> FYI - I had been incorrectly advertising the screencast as 120 minutes. It's actually 1:21 - 1 hour, 20 minutes. I'm terribly sorry about that. I've corrected the site.

</comment><comment><AUTHOR> FYI - I've knocked the price down to $9 for the next few days. :)

</comment><comment><USER10> Just FYI - price is on $9 for next 3 1/2 days :).</comment>