We Should Not Accept Scientific Results That Have Not Been Repeated - <AUTHOR> http://nautil.us/blog/we-should-not-accept-scientific-results-that-have-not-been-repeated <comment><USER2> Define repetition.

It's not as simple as that, for all sciences - once again an article on repeatability seems to have focused on medicinal drug research (it's usually that or psychology), and labelled the entire "Scientific community" as 'rampant' with " statistical, technical, and psychological biases".

How about, Physics?

The LHC has only been built once - it is the only accelerator we have that has seen the Higgs boson. The confirmation between ATLAS and CMS could be interpreted as merely internal cross-referencing - it is still using the same acceleration source. But everyone believes the results, and believes that they represent the Higgs. This isn't observed once in the experiment, it is observed many, many times, and very large amounts of scientists time are spent imagining, looking for, and measuring, any possible effect that could cause a distortion or bias to the data. When it costs billions to construct your experiment, sometimes reproducing the exact same thing can be hard.

The same lengths are gone to in order to find alternate explanations or interpretations of the result data. If they don't, they know that some very hard questions are going to be asked - and there will be hard questions asked anyway, especially for extraordinary claims - look at e.g. DAMA/LIBRA which for years has observed what looks like indirect evidence for dark matter, but very few people actually believe it - the results remain unexplained whilst other experiments probe the same regions in different ways.

Repetition is good, of course, but isn't a replacement for good science in the first place.

<comment><USER3> We don't need to 'define repetition', we need to foster a culture that a) accepts repetition and b) does not accept something for a fact just because it's in a journal. Right now, (a) is not even acceptable; nobody will publish a replication study. Ofc, LHC is impossible to replicate, but the vast majority of life science studies are.

I should think this is mostly needed in life sciences. Other, more 'exact' sciences seem to not have this problem.

<comment><USER4> _> we need to foster a culture that a) accepts repetition_

Do we?

I don't think we do. I think we need to foster a culture of honesty and rigor. Of good science. Which is _decidedly_ different from fostering a culture of "repetition" for its own sake.

Paying for the cost of mountains upon mountains of lab techs and materials that it would require to replicate every study published in a major journal just _isn 't_ a good use of ever-dwindling science dollars. Replicate where it's not far off the critical path. Replicate where the study is going to have a profound effect on the direction of research in several labs. But don't just replicate because "science!"

In fact, one could argue that the increased strain on funding sources introduced by the huge cost of reproducing a bunch of stuff would increase the cut-throat culture of science and thereby decrease the scientist's natural proclivity toward honesty.

 _> and b) does not accept something for a fact just because it's in a journal_

Again, it's entirely unclear what you mean here.

It's impossible to re-verify every single paper you read (I've read three since breakfast). That would be like re-writing every single line of code of every dependency you pull into a project.

And I'm pretty sure literally no scientist takes a paper's own description of its results at face value without reading through methods and looking at (at least) a summary of the data.

Taking papers at face value is really only a problem in science _reporting_ and at (very) sub-par institutions/venues.

I don't care about the latter, and neither should you.

WRT the former, science reporters often grossly misunderstand the paper anyways. All the good reproducible science in the world is of zero help if science reporters are going to bastardize the results beyond recognition anyways...

<comment><USER5> > I don't think we do. I think we need to foster a culture of honesty and > rigor. Of good science. Which is decidedly different from fostering a > culture of "repetition" for its own sake.

No one is proposing repetition for its own sake. The point of repetition is to create rigor, and you _can 't do rigorous science without repetition_.

> Paying for the cost of mountains upon mountains of lab techs and materials > that it would require to replicate every study published in a major journal > just isn't a good use of ever-dwindling science dollars. Replicate where > it's not far off the critical path. Replicate where the study is going to > have a profound effect on the direction of research in several labs. But > don't just replicate because "science!"

I could see a valid argument for only doing science that will be worth replicating, because if you don't bother to replicate you aren't really proving anything.

<comment><USER4> _I could see a valid argument for only doing science that will be worth replicating, because if you don 't bother to replicate you aren't really proving anything._

Exactly. A lot of the science _I 've_ done should not be replicated. If someone told me they wanted to replicate it, I would urge them not to. Not because I have something to hide. But because some other lab did something strictly superior that should be replicated instead. Or because the experiment asked the wrong questions. Or because the experiment itself could be pretty easily re-designed to avoid some pretty major threats.

The problem is that is that hindsight really is 20/20\. It's kind of impossible to _ONLY_ do good science. So it's important to have the facility to recognize when science (including your own) isn't good -- or is good but not as good as something else -- and is therefore not worth replicating.

I guess the two key insights are:

1\. Not all science is worth replicating (either because it's too expensive or for some other reason).

2\. Replication doesn't necessarily reduce doubt (particularly in the case of poorly designed experiments, or when the experiment asks the wrong questions).

<comment><USER6> This is a really good post which contributes to the conversation. Why make it on a throwaway account? We need more of this here!

</comment></comment></comment></comment></comment></comment><comment><USER7> I see a lot of people commenting here that there's no incentive to repeat previous research because it's not useful for getting grants, etc. This kind of true but I think it misses something important.

At least in life sciences (can't comment on other fields), it's not that scientists _don 't_ repeat each other's results. After all, if you're going to invest a significant fraction of your tiny lab budget on a research project, you need to make sure that the basic premise is sound, so it's not uncommon that the first step is to confirm the previous published result before continuing. And if the replication fails, it's obviously not a wise idea to proceed with a project that relies on the prior result. But that work never makes it into a paper.

If the replication succeeds, great! Proceed with the project. But it's time- consuming and expensive to make the reproduction publication worthy, so it will probably get buried in a data supplement if it's published at all.

If the replication fails, it's even more time-consuming and expensive to convincingly demonstrate the negative result. Moreover, the work is being done by an ambitious student or postdoc who is staring down a horrible job market and needs novel results and interesting publications in order to have a future in science. Why would someone like that spend a year attacking the work of an established scientist over an uninteresting and possibly wrong negative result, and getting a crappy paper and an enemy out of it in the end, instead of planning for their own future?

If enough people fail to replicate a result, it becomes "common knowledge" in the field that the result is wrong, and it kind of fades away. But it's not really in anyone's interest to write an explicit rebuttal, so it never happens.

<comment><USER8> This is why I'm always skeptical of people zealously claiming "all GMOs are perfectly safe! It's been _proven_!"

Yeah, proven by expensive studies that were funded by the company making the GMO. Who is going to pay for another study to try to disprove it?

There's a reason we sprayed DET all over our vegetables for years before it was banned: there was no scientific studies proving that it was harmful, even though it clearly was harmful in hindsight.

Science is not instant, and there's no way someone can claim that some brand- new GMO is "perfectly safe", without any long-term studies on its effects over 10, 20, 30 years of exposure. That's just not possible. And yet you try to explain it to these science zealots and they just brush you off as being "anti-science".

<comment><USER9> Do you mean DDT?

Anyway, beyond the 'philosophy of science' issue of whether you can prove something, there is good affirmative evidence that existing GMOs are safe for numerous reasons.

First, there's no mechanistic reason to think they would be dangerous. T-DNAs are not somehow magically toxic to humans; everything you eat is riddled with millennia of t-dnas, old viruses, transposon blooms, etc. etc.

The technologies themselves should be safe as well. BT is well understood and embraced by the organic community as an applied natural pesticide, so you would need to find evidence that the localization somehow makes it toxic. Glyphosate resistance is also unlikely to have any effect _a priori_ because it affects a metabolic pathway completely absent in animals.

Argue all you like about how nothing can be 'perfectly safe', sure, but there's no reason to think that GMOs are dangerous, and people have looked quite hard. ______

Finally, just look at the Seralini pile-of-bullshit for evidence that there's plenty of incentive to publish anything critical of GMOs. No one is sitting on career-making evidence.

[https://en.wikipedia.org/wiki/S%C3%A9ralini_affair](https://en.wikipedia.org/wiki/S%C3%A9ralini_affair)

<comment><USER8> > First, there's no mechanistic reason to think they would be dangerous.

That's like saying "there's no reason to think peanuts would be dangerous" since humans eat them all the time. And yet, they are deadly to some humans. No one knows why.

But they do know that food allergies are far more prevalent in the U.S. than in other countries. And now, suddenly, people in Africa and China are starting to exhibit food allergies that the U.S. has had for a while. So what have we started shipping over to them that's causing these allergies? Who will fund that study?

<comment><USER10> What you've hinted at here is fascinating to me.

Do you have some references?

In case it's not clear, I'd like to read articles about food allergies that have been common in USA for some time now becoming common in countries that are coming out of 2nd world status and into 1st world.

</comment></comment></comment></comment></comment><comment><USER11> I'm a physician and I've been suggesting this to my colleagues for a few years, only to be met with alienated stares and labeled cynical.

The doctors and doctors-in-training I work with have altruistic motives, but place too much stock in major medical studies. They also frequently apply single-study findings to patient care, even to patients that would've been excluded from that study (saw this a lot with the recent SPRINT blood pressure trial).

And don't even get me started on the pulmonary embolism treatment studies. What a clinical mess that is.

It's frustrating.

<comment><USER12> but in medicine, what is the practical alternative?

how do _you_ incorporate these findings? ignore them?

if so, it's probably bad for your patients. the only thing worse than a single-study finding is a zero-study finding.

<comment><USER11> I was simply suggesting what we all learn in medical school and residency: to appropriately evaluate clinical studies. Just don't think most doctors do.

Let me give you an example of how I approach things. The guidelines for acute pancreatitis recommend using a fluid called LR instead of NS for volume resuscitation. This is based on an single study that included 10 patients and simply noted slightly better lab numbers; there was no difference in clinical outcome. Lots of problems with that study, right (small, underpowered, confounders, validity issues, etc)? However, there's no major disadvantage for using LR in those patients (unless hyperkalemia is a concern), so I use it since it might have a benefit.

This is a very simple example. It gets much more complicated than that.

"Probably" is one of favorite words in medicine, btw :).

<comment><USER13>
        "Probably" is one of favorite words in medicine, btw :).

Right as it should. If somebody answers my question by "It depends", then I know I'm in good company!

</comment></comment></comment></comment><comment><USER14> > Nowadays there's a certain danger of the same thing happening (not repeating > experiments), even in the famous field of physics. I was shocked to hear of > an experiment done at the big accelerator at the National Accelerator > Laboratory, where a person used deuterium. In order to compare his heavy > hydrogen results to what might happen with light hydrogen, he had to use > data from someone else's experiment on light, hydrogen, which was done on > different apparatus. When asked why, he said it was because he couldn't get > time on the program (because there's so little time and it's such expensive > apparatus) to do the experiment with light hydrogen on this apparatus > because there wouldn't be any new result. And so the men in charge of > programs at NAL are so anxious for new results, in order to get more money > to keep the thing going for public relations purposes, they are destroying-- > possibly--the value of the experiments themselves, which is the whole > purpose of the thing. It is often hard for the experimenters there to > complete their work as their scientific integrity demands.

\-- Richard Feynman, "Surely You're Joking, Mr. Feynman", pp. 225-226

<comment><USER15> Current top comment [https://news.ycombinator.com/item?id=12186295](https://news.ycombinator.com/item?id=12186295) , specifically rebutted 30 years before the comment was written.

<comment><USER4> This _isn 't_ a rebuttal of the linked comment.

The linked comment doesn't state that it would be a waste of time to replicate on a hypothetical LHC clone.

Rather, the linked comment states that we can accept the Higgs result with reasonable confidence _even though_ it's currently infeasible to replicate that experiment.

Feynman's issue was also qualitiatively different -- the scientist was _comparing_ results from two different instruments. The people in charge of one of the instruments wouldn't allow the scientist to run both experiments on a single instrument. In fact, from context, it's not even clear to me Feynmann would have insisted on re-running the original experiment if the scientist were not using a different accelerator for the second one. Anyways, in the Higgs case, there's no potential for a "comparing readings from instrument A to readings from instrument B" type bug.

More to the point, and FWIW, I somehow doubt Feynman would insist on building a second LHC for the sole purpose of replicating the Higgs measurement. But I guess we have to leave that to pure speculation.

</comment></comment></comment><comment><USER16> My first reaction to this headline was "duh." _Of course_ we should hold off on accepting scientific claims (i.e., predictions about the natural world) that to date have been verified only by the same person making those claims!

My next reaction was, "wow, it's a sad state of affairs when a postdoctoral research fellow at Harvard Medical School feels he has to spell this out in a blog post." It implies that even at the prestigious institution in which he works, he is coming across people who treat science like religion.

<comment><USER17> Of course you as a reader of said claims confirm at the very least that they've been independently reproduced, right?

(If so, this shouldn't be news.)

<comment><USER18> Unfortunately, there is no easy way to do it. Confirmation studies are not easily accepted by impactful journals/conferences, thus nearly nobody bothers to do them. Even if there is one, it can be surprisingly hard to find it.

As a point of anecdata: my wife's master thesis was a confirmation study of using LLDA for face recognition. I remember seeing it included in some book by the university press. I gave up Googling for it after 5 minutes.

<comment><USER19> There needs to be better scientific protocol. More linking through data instead of annoying cites. I think anyway.

</comment></comment></comment></comment><comment><USER20> We shouldn't "accept" or "reject" results at all.

It's not a binary option. One poor experiment might give us some evidence something is true. A single well reviewed experiment gives us more confidence. Repeating the results similarly does. As does the reputation of the person conducting the experiment and the way in which it was conducted.

It's not a binary thing where we decide something is accepted or rejected, we gather evidence and treat it accordingly.

<comment><USER21> So many scientists I talk to don't have a basic understanding of philosophy of science. I don't necessarily blame them--I understand why "philosophy" as an academic field is seen as a soft, speculative, and pretentious field compared to the rigor of science, but as Daniel Dennett said, "There is no such thing as philosophy-free science; there is only science whose philosophical baggage is taken on board without examination."

These days, if you ask a scientist "So how do we prove something is true using science?" they'll be able to recite Popper's falsificationism as if it's a fundamental truth, not a particular way of looking at the world. But the huge gap between the particular theory that people get taught in undergrad--that science can't actually prove anything true, just disprove things to approach better hypotheses--and the real-world process of running an experiment, analyzing data, and publishing a paper is unaddressed. The idea that there's a particular bar that must be passed before we accept something as true is exactly what got us into this mess in the first place! There's a naive implicit assumption in scientific publishing that a p-value < 0.05 means something is true, or at least likely true; this author is just suggesting that true things are those which yield a p-value under 0.05 twice!

What's needed, in my opinion at least, is a more existential, practically- grounded view of science, in which we are more agnostic about the "truth" of our models with a closer eye to what we should actually _do_ given the data. Instead of worrying about whether or not a particular model is "true" or "false," and thus whether we should "accept" or "reject" an experiment, focus on the predictions that can be made from the total data given, and the way we should actually live based on the datapoints collected. Instead, we have situations like the terrible state of debate on global warming, because any decent scientist knows they shouldn't say they're absolutely sure it's happening, or a replication crisis caused by experiments focused on propping up a larger model, instead of standing on their own.

</comment></comment><comment><USER22> I agree in principle. There are a few concerns:

1\. How should we receive costly research that took special equipment and lots of time to develop and cultivate? I.e., CERN?

2\. A lot of research is published, ignored, and then rediscovered. In this case, we may want to accept the research until it cannot be repeated (i.e., in another journal publication).

3\. Reviewers of academic publications probably are not qualified or have the time to recreate all scientific research.

4\. Isn't the academic system at its core kinda... broken?

<comment><USER23> "Isn't the academic system at its core kinda... broken?"

can your elaborate on what you mean?

<comment><USER24> I gather its things like misaligned incentives. Like funding sources dictating the "desirable" result, reluctance to publish negative results, pursue of (vanity) metrics that leads to quota system by universities, etc.

</comment></comment></comment><comment><USER25> Maybe conferences should have a "reproducibility" track for that purpose? Also, I don't know about other fields, but I'm pretty sure that in CS, if you just took a paper and tried to reproduce the results, you'll get rejected on the ground that you offer no original contribution; no original contribution => no publication => no funding.

<comment><USER26> For CS, reproducing should be easy given the code and input data right? Other sciences' input isn't so easily shared

<comment><USER27> Ideally you should re-implement the algorithm based on the description in the paper to verify that the description of the algorithm is correct. You should also test with your own data to make sure that the algorithm works on all reasonable data and not only on some provided cherry picked data. If you can't get the expected results with your own implementation and your own data then the results aren't reproduced.

<comment><USER28> Yes. So being able to rerun with the same code and same inputs to get the same outputs is a lower bar. Many papers don't meet even that bar.

(Mostly because they don't publish code nor data; and academic code is often a horrible mess, and the code was mucked around with between different stages of running.)

</comment></comment></comment></comment><comment><USER29> Many people have mentioned that replicating an experiment can be expensive, but I don't think anybody has really brought up just how expensive this can be.

Not all science is done in a lab. Replicating an experiment is obviously feasible for a short term psychology experiment, but in earth sciences (oceanography for instance.) it is far less often possible to reproduce an experiment for the following reasons. N.B. This is all from my personal experience of one field of science.

1.) Cost. If you got funding to take an ice-breaker to Antarctica to "do science" it required several million dollars to fund. It is difficult enough to secure funding for anything these days, none the less prohibitively expensive attempts to reproduce results. (honestly any serious research vessel will run costs into the millions, regardless of destination.)

2.) Time. Say you are on a research vessel taking measurements of the Amazon river basin. This is a trip that takes months to years to plan and execute. If you return to duplicate your experiment 2 years later, the ecology of the area you were taking measurements of may have changed completely.

3.) Politics. Earth sciences often require cooperation from foreign entities, many of which are not particularly stable, or whom may be engaging in political machinations that run counter to your nationality's presence in the country, or both. Iran and China are two good examples. Both are home to some excellent oceanographers, and both of which can be very difficult to Science in when your team includes non Iranian/Chinese nationalities.

</comment><comment><USER30> The big issue right now is funding of replicated research, who wants to fund a research to prove someone else was right? Most of these funds are granted based on potential outcome of the new discovery like: potential business, patents, licenses, etc... not being the first one would probably wipe most of these benefits, cutting down to a small probably getting funded...

Now, straight to the point, who's going to pay for the repeated research to prove the first one?

<comment><USER31> On a low level, I think it should be mandatory for masters students to do a pre-thesis project, which is replicating findings in a published paper.

It would do something about low hanging fruit in terms of testing reproduceability and since there is a published paper, the student has access to guidelines for setting up and reporting on a large project, which will help them learn how to do their own, original thesis.

<comment><USER32> I had my Masters students do this as part of my wireless networking class this year. It was very instructive for me and the students seemed to enjoy it, so I'll definitely keep it in the syllabus.

</comment></comment></comment><comment><USER33> Totally agree. I'd go even further and make free licenses on scientific source and datasets mandatory. Research that is funded by public money should lead to public code and data.

<comment><USER34> What about defense research?

<comment><USER35> Well, there's the GPL-styled approach: anyone with access to the results must also have access to the associated data. This doesn't mean it is mandatory to make it public, though you'd have to restrict the redistribution freedom.

<comment><USER31> I recently used a large dataset of tweets in a research project. As far as I know, I do not have the rights to distribute these.

I also used a dataset consisting of newspaper articles. It cost me $1.000 to get access to, and I definitely do not have the rights to redistribute it.

<comment><USER27> As long as you provide a detailed enough description of the source of your dataset that I can reproduce it myself then that is fine. So in your first case tell me what criteria you used to select your tweets and in the second tell me where to send my $1000 and what to ask for.

<comment><USER32> Unfortunately not everyone reports this information. Here is a study that we did of over 500 papers using online social network data: [http://tnhh.org/research/pubs/tetc2015.pdf](http://tnhh.org/research/pubs/tetc2015.pdf) While most authors would report high-level characteristics (e.g., which social network they measured), fewer authors reported how they sampled the network or collected data, and very few people reported on how they handled ethics, privacy and so forth.

</comment></comment></comment></comment></comment></comment><comment><USER36> Lots of scientific results are repeated but not published. If it doesn't work then people just move on. The problem is journals. There is no way to publish your attempts to repeat an experiment, unless you put it into another paper.

The other issue, especially in the life sciences, is inaquedate statistical input. If someone performs an underpowered, confounded experiment and gets a positive result, then someone else performs the same underpowered confounded experiment and gets a negative result, what have we learned except that the experiment is underpowered?

</comment><comment><USER37> With science, the profession and the product are distinctly different, and we are failing to hold the profession to the standards of the product. Science, the profession, is political, incentive driven, and circumstantial. Scientists need to get paid. Science, the product, is apolitical, existential, and universal. So those who love and believe in the products of science may wish upon themselves to be these things also. I know I do. Except, sometimes it just ins't practical, or even possible.

But repeatability actually matters more professionally. Scientifically speaking, if the science is bad it just won't work when others try to make use of it. All bad science will be identified or corrected as we try and make use of it and convert it into new technology. Technology mandates repeatability. So those scientists who fail to produce repeatable science, regardless of how professionally successful they may be, will inevitably fail to produce any new technology or medicine, and vice versa.

</comment><comment><USER38> Obviously I agree that scientific results must be reproducible. But I also realize that it's simply infeasible to repeat the entirety of every study, and much less to also go to the effort to write and peer-review those repeated results.

What I think is overlooked in this discussion as that a lot of confirmation work already happens. Most (all?) scientific results are incremental progress built on a heap of previous work. In the course of normal research, you reproduce existing results as necessary before altering conditions for your own study. If you can't confirm the results, well then perhaps you have a paper (though it can be politically challenging to get it published, and that's a separate problem). But if you do, then you don't waste time publishing that, you get on with the new stuff.

Ultimately, I don't think scientists do accept results _in their field_ that they have not repeated.

</comment><comment><USER39> Cue all the people justifying their pseudoscientific behavior. If it is too expensive to fund twice, it shouldn't be funded once. If that means the LHC and LIGO wouldn't get done, then we should have only funded one of them. We need to remain skeptical of those results until replicated by a new team. Even one replication is pretty weak...

Independent replications of experiment (and the corresponding independent reports of observations) are a crucial part of the scientific method, no matter how much you wish it wasn't. Nature doesn't care if it is inconvenient for you to discover her secrets, or that it is more difficult for you to hype up your findings to the unsuspecting public.

<comment><USER40> You do realize that scientists who work on the LHC have the highest repeatability standards of any science profession, right?

<comment><USER41> The LHC experiment is not the issue here.

There is a lot of transparency there, a lot of well meaning people with a lot of oversight.

I suggest most would admit 'there could be a problem' there, but it's out in the open if there is.

The problem of lack of repeatability I think has to do with subconscious bias on the part of the experimenters which will be less pronounced when there are 5000 people working on it.

</comment></comment></comment><comment><USER42> Having wasted time trying to replicate someone else's results who 'lost' their code, I agree! Maybe repeating the experiment should be part of the peer review.

<comment><USER43> So frustrating. Lost = "I didn't think anyone would hold me accountable for it."

</comment></comment><comment><USER44> Looking good is, sadly, better rewarded than doing good in many areas of life. It's doubly sad that this affects our body of scientific knowledge. Even claims that are reproduced can suffer from funding bias and confirmation bias. The truth hopefully comes out in the end, but I'm sad for the harm that's caused in the interim.

</comment><comment><USER45> I don't get why this is not top on the agenda for the scientific community and the government. Huge amounts of research money is lost in repeating stuff that doesn't work. Huge amounts of money is lost chasing broken science.

I blame this on the neo-liberal ideology. This intense focus on getting money's worth, on tying grants to specific goals, counting publications etc. Driving research exclusively on a very narrowly defined money incentive has driven us further into this sort of mess. The money grabbing journals which has prevented any significant innovation in how science is shared.

I think what science needs is a model closer to that of open source. With open projects anybody can contribute to but where verification happens through personal forged relationships. The Linux kernel code quality is verified by a hierarchy of people trusting each other and knowing something about each others quality of work. Work should be shared like Linux source code in a transparent fashion and not behind some antiquated paywall.

I don't think the grant system can entirely away, but perhaps it should be deemphasized and instead pay a higher minimum amount of money to scientists for doing what they want. Fundamental science breakthrough doesn't happen because people had a clear money incentive. Neither Einstein, Nils Bohr, Isaac Newton or Darwin pursued their scientific breakthroughs with an aim of getting rich. Few people become scientists to get rich. Why not try to tap into people's natural desire to discover?

</comment><comment><USER46> This problem, like many in modern day science, can in large part be traced back to unstable funding. On the Maslow's-style hierarchy of research lab needs, the need for funding is a lot lower on the scale than the aspiration for scientific purity, just as a human's need for food is lower on the scale than their desire for self-actualization.

If competition for research dollars ceases to be so cutthroat, it will go a long way towards solving this and many other seemingly entrenched cultural problems.

</comment><comment><USER47> A big distinction here is that different fields have different levels of dependence on prior results. In fields like psychology etc, you don't need the previous results to work in order to run your own experiment. In other words, if you cite a well-known paper saying "people seem to work faster near the color red" and your paper runs an experiment to see if they work faster near the color yellow, if the red paper is later unreplicable, it doesn't change the outcome of your experiment in any way.

In contrast, if you are in machine learning and you are extending an existing architecture you are very directly dependent on that original technique being useful. If it doesn't "replicate" the effectiveness of the original paper, you're going to find out quickly. Same for algorithms research. Some other comments here have mentioned life sciences being the same.

So I think there's a qualitative difference between sciences where we understand things in a mostly statistical way (sociology, psychology, medical studies) where the mechanism is unknown (because it's very very complicated), but we use the process of science mechanistically to convince ourselves of effectiveness. e.g. I don't know why this color makes people work faster/ this drug increases rat longevity / complex human interactions adhere to this simple equation, but the p value is right, so we think it's true. Versus sciences where we have a good grasp of the underlying model and that model is backed up by many papers with evidence behind it, and we can make very specific predictions from that model and be confident of correctness.

</comment><comment><USER48> In the world of chemistry, biochemistry and microbiology a huge step forward would be for journals to require a complete list of products used. The publication should also include the certification of analysis for each item as they vary over time.

For example, here are two product specifications for a dye called Sirius Red, the first by Sigma-Aldrich[1] and the second by Chem-Impex[2]. The Sigma- Aldrich product contains 25% dye while the Chem-Impex contains equal or greater than 21%. These two dyes could be quickly assessed with a spectrophotometer in order to determine an equivalency, however you need both dyes on hand which doesn't seems like a good use of funding. Also this touches on another problem in replication which is, what is in the other 75%+ of the bottle?

[1] [http://www.sigmaaldrich.com/Graphics/COfAInfo/SigmaSAPQM/SPE...](http://www.sigmaaldrich.com/Graphics/COfAInfo/SigmaSAPQM/SPEC/36/365548/365548-BULK_______SIAL_____.pdf) [2] [http://www.chemimpex.com/MSDSDoc/22913.pdf](http://www.chemimpex.com/MSDSDoc/22913.pdf)

</comment><comment><USER49> Look at research done on many political hot button topics. They love results that have not been repeated. I see all sorts of posts even on HN that reference such "science" as well. The root problem, people who are pushing an agenda.

</comment><comment><USER50> > The inconvenient truth is that scientists can achieve fame and advance their > careers through accomplishments that do not prioritize the quality of their > work

An even more inconvenient truth is that scientists cannot even keep their jobs if they prioritize the quality of their work. The pressure to publish novel results is too strong and it is almost impossible to get any support for confirming previous ones.

</comment><comment><USER51> I agree with the main point of this article but in terms of its analysis and prescriptions I think it gets two things backwards. (1) Most scientists seek fame as a means to the end of getting tenure and funding, not the other way around; if you gave them tenure (and the ability to move their tenure to somewhere else if they wanted to move) and perpetual funding and told them they could choose to be anonymous, I think many would choose that option. (2) Replication is not done/published enough because the incentive to do so (measured in: increase in probability of getting tenure per hour spent) is not high enough, not because people are overly willing to accept unreplicated work.

In order for a lot more replication to get published, what would be needed would be for people who spent their careers replicating others' results (at the expense of not producing any important novel results of their own) to get tenure at top institutions (outcompeting others who had important novel results but not enough published replications).

</comment><comment><USER52> "repeated" in this context is not incorrect, but i think "replicated" is perhaps a better choice.

That aside, i think _repeatability_ is a much more useful goal (rather than "has been repeated"). For one thing, meaningful replication must be done by someone else; for another, it's difficult and time consuming; the original investigator has no control over whether and when another in the community chooses to attempt replication of his result. What is within their control is an explanation of the methodology they relied on to produce their scientific result in sufficient detail to enable efficient repetition by the relevant community. To me that satisfies the competence threshold; good science isn't infallible science, and attempts to replicate it might fail, but some baseline frequency for ought to be acceptable.

</comment><comment><USER53> This is wrong-headed in the extreme.

What we should demand is scientific results that have FAILED.

When we see a p=0.05, but we don't know that this SAME EXACT EXPERIMENT has been run 20 times before, we're really screwing ourselves over.

Relevant: [https://xkcd.com/882/](https://xkcd.com/882/)

</comment><comment><USER54> Replication isn't enough. It's also necessary to know how many non- replications have occurred but got swept under the rug. It's not the existence of replications that matter---it's the rate of replication relative to number of replication attempts.

So I agree with the title "We Should Not Accept Scientific Results That Have Not Been Repeated". But I would add to it "We Should Not Accept Scientific Results from Studies That Weren't Preregistered". Registration of studies forces negative results to be made public, allowing for the positive result rate / replication rate to be calculated.

Otherwise the existence of a "positive" result is more a function of the trendiness of a research area than it is of the properties of the underlying system being studied.

</comment><comment><USER55> More pragmatically, we should not accept scientific _publications_ and _conferences_ which do not publish negative results and disconfirmations.

</comment><comment><USER56> I disagree.

One part of science is observation. Including observations which cannot be, or at least have not been, repeated. For example, consider a rare event in astronomy which has only been detected once. Is that science? I say it is. But it's surely not repeatable. (Even if something like it is detected in the future, is it really a "repeat"?)

Some experiments are immoral to repeat. For example, in a drug trial you may find that 95% survive with a given treatment, while only 5% survive with the placebo. (Think to the first uses of penicillin as as real-world example.)

Who among you is going to argue that someone else needs to repeat that experiment before we regard it as a proper scientific result?

<comment><USER57> > One part of science is observation. Including observations which cannot be, > or at least have not been, repeated. For example, consider a rare event in > astronomy which has only been detected once. Is that science? I say it is. > But it's surely not repeatable.

First off, you can accept the observation at face value as an observation, but conclusions drawn from the claims which have no other support or means of verification should not be accepted and would not be accepted. Fortunately, most of the time even if something is initially sparked by a very rare occurrence, it will have some kind of implications that are verifiable by some other means other than just waiting for something to happen in space.

But even something that is rare and relies on observation, like gravitational waves, we have already been able to identify more than one occurrence.

> Some experiments are immoral to repeat. For example, in a drug trial you may > find that 95% survive with a given treatment, while only 5% survive with the > placebo.

What's more immoral, releasing a drug that's only had one test, even a striking one, on the public as a miracle cure that you have not truly verified or performing another test to actually be sure of your claims before you release it?

> Who among you is going to argue that someone else needs to repeat that > experiment before we regard it as a proper scientific result?

That's how science works. If something is not independently repeatable and verifiable then science breaks down. Look at the recent EM drive. Most scientists in the field were skeptical of it, and once it was finally attempted to be independently verified the problems were found.

Independent verification is the cornerstone of science and what makes it different from bogus claims by charlatans.

<comment><USER56> > conclusions drawn from the claims which have no other support or means of > verification should not be accepted and would not be accepted

I disagree. In _all cases_ , even with repeated experiments, the claims are only tentatively accepted. The confirmation by others of Blondlot's N-rays didn't mean they were real, only that stronger evidence would be needed to disprove the conclusions of the earlier observations.

Astronomy papers make conclusions based on rare or even singular observations. Take SN1987a as an example, where observations from a neutrino detector were used to put an upper limit on the neutrino mass, and establish other results.

> "or performing another test"

This question is all about _repeating_ an experiment. Repeating the experiment would be immoral.

There are certainly other tests which can confirm the effectiveness, without repeating the original experiment and without being immoral. For the signal strength I gave, we can compare the treated population to the untreated population using epidemiological studies.

But under current medical practices, if a drug trial saw this sort of effectiveness, the trial would be stopped and _everyone_ in the trial offered the treatment. To do otherwise is immoral. As would repeating the same trial.

<comment><USER58> > But under current medical practices, if a drug trial saw this sort of > effectiveness, the trial would be stopped and everyone in the trial offered > the treatment. To do otherwise is immoral. As would repeating the same > trial.

Then perhaps current medical practices should change. The benefits to those who were previously given the placebo should be balanced against the probability that the observed outcomes may not occur in other circumstances.

<comment><USER56> Are you for real? You would sacrifice people upon the alter of reproducibility?

Down that path lies atrocities. The system was put into place to prevent repeats of horrors like the "Tuskegee Study of Untreated Syphilis in the Negro Male".

<comment><USER58> I'd rather not sacrifice people on the altar of a single study, no matter how significant the results. Down that path lies atrocities, too, albeit of a quieter sort.

<comment><USER56> As I said earlier, there are alternatives which are both moral and can verify effectiveness without having to repeat the original experiment.

You chose to not verify, and insist upon repeating, thus likely consigning people to unneeded pain and even death.

I'll give a real-world example to be more clear cut about modern ethics and science. Ever hear of TGN1412? [https://en.wikipedia.org/wiki/TGN1412](https://en.wikipedia.org/wiki/TGN1412)

It went into early human trials, and very quickly caused a reaction. "After very first infusion of a dose 500 times smaller than that found safe in animal studies, all six human volunteers faced life-threatening conditions involving multiorgan failure for which they were moved to intensive care unit." ([http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2964774/](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2964774/) )

Here's a publication of the effects: [http://www.nejm.org/doi/full/10.1056/NEJMoa063842](http://www.nejm.org/doi/full/10.1056/NEJMoa063842) .

Is it moral to reproduce that experiment? I say it is not moral, and must not be repeated even though it is possible to do so.

Can a publication about the effects still be good science even though medical ethics prevent us from repeating the experiment? Absolutely.

What say you?

</comment></comment></comment></comment></comment></comment></comment><comment><USER59> Didn't John Oliver say the same thing few months ago in his episode on scientific studies, [https://www.youtube.com/watch?v=0Rnq1NpHdmw](https://www.youtube.com/watch?v=0Rnq1NpHdmw)

</comment><comment><USER60> It's not just money that prevents people from repeating experiments, it's recognition.

The general idea for research to be accepted is that it makes some novel, albeit small, impact on the field, acceptable for publication in a peer reviewed journal or proceeding. Repeating someone else's experiments wont get you that, so in general it wont help you graduate or move you toward a higher position at a university or in your profession, meaning there is very little motivation for researchers to pursue such endeavors.

So instead of just throwing money at the problem, we may need to entirely revamp how we recognize the pursuits of researchers.

</comment><comment><USER61> We learned the importance of this in high school science and it baffles me that it's not already the case.

<comment><USER62> We have some kind of weird hero-worship of scientists where the general public just believes what they say, even if they never even attempt to replicate their results. They do an experiment (which may or may not be scientifically sound to start with) and then publish results, and the public eats it up.

And then people have the nerve to say, "Last week chocolate was bad for me, now it's good? Make up you mind!" No, _stop_ listening to un-replicated studies! Jeez.

<comment><USER44> Good point about the public involvement. The public and the news systems are part of the problem.

I've lost count of how many 'battery breakthrough' articles I've come across, but they seem to pass the newsworthy test.

<comment><USER28> Wasn't the problem with battery breakthroughs that they don't commercialise well, rather than that the science doesn't repeat?

</comment></comment></comment></comment><comment><USER63> I have an alternative proposal: do a study right the first time.

That means:

A) Pre-registering the study design, including the statistical analysis. Otherwise, attaching a big label "Exploratory! Additional confirmation needed!"

B) Properly powering the study. That means gathering a sample large enough that the chances of a false negative aren't just a coin flip.

C) Making the data and analysis (scripts, etc.) publicly available where possible. It's truly astounding that this is not a best practice _everywhere_.

D) Making the analysis reproducible without black magic. That includes C) as well as a more complete methods section and more automation of the analysis (one can call it automation but I see it more as reproducibility).

Replication of the entire study is great, but it's also inefficient in the case of a perfect replication (the goal). Two identical and independent experiments will have both a higher false negative and false positive rate than a single experiment with twice the sample size. Additionally, it's unclear how to evaluate them in the case of conflicting results (unless one does a proper meta-analysis--but then why not just have a bigger single experiment?).

<comment><USER64> Your proposal is comparable to saying that checks and balances are not needed in a democracy, politicians just need to govern "right". This is about incentivising scientists to do the right thing instead of merely demanding it, like you do.

<comment><USER63> How is advocating for a new set of best practices any more "demanding" or wishful than a regime of obligatory replication? And how is this categorically different from current practices such as peer review, disclosing conflicts of interest, an IRB, etc.?

</comment></comment></comment><comment><USER65> I think with the increased visibility of scientific research to the general public, it's less that science needs to stop accepting unrepeated results, but instead the paper process needs to be updated to reflect the new level of availability, and journal databases need better relationship views between papers and repeated tests.

As an outsider looking in on the Scientific process, I am not really sure how applicable my opinions are, but I see these as useful changes.

Basically, in reverse order, my suggestions for science to adopt are as follows:

Papers in databases need to have fields related to reproduction studies, and it needs to start becoming a prideful part of the scientific process; just as there is a lot of pride and money, researchers should start to thump their chest based on the reproducibility of their work, actively seeking out contemporaries and requesting a reproduction study as part of the pubilshing process, and subsequently updating.

The papers published themselves should take a moment (perhaps no more than a paragraph) to include a "for media" section that outlines the "do's and don't's" on reporting on the research. For example, cancer research should clearly state examples of acceptable understandings in lay person terms as a sort of catch for sloppy reporting. Something like "Do not write "cure for cancer found" or "Effective treatment", instead write "progress made, etc". Basically put a sucker punch to outlandish headlines and reporting right in the paper itself, and let journalists who want to be sensationalist embarrass themselves.

This seems like two very simple acts that could raise the bar for science a bit.

<comment><USER57> Those are both good but the key here is the media needs to understand that scientific papers that have not been independently verified are in a "maybe" state.

Of course, they probably do know this and just choose to ignore it because "Unverified Study that MIGHT Point to M&M's Being Good For You" won't get as many clicks as "M&M's Are Good For You Says New Study!"

<comment><USER65> This is sort of why I think having it stated explicitly within the paper, not just an aside but part of the actual process. It's to pit less scrupulous journalists against one another, in an "honor among thieves" sort of way I guess. If someone wants to go ahead and write clickbait, they can, but it leaves them open to someone else looking to discredit them going "well, did you even read the paper? they told you not to write that."

it's not so much checking for the public purpose, it's for others.

</comment></comment></comment><comment><USER66> Most disciplines where correctness is important seem to end up having some adversarial component. It is explicitly how the justice system in the US works [1]. Many software companies have separate QA departments that are deliberately kept at a remove from the engineers to encourage some rivalry between them. Security issues are almost always managed in an adversarial way (though here, you could argue that's because it reflects how the system itself is [mis-]used). Markets are intended to allow fair competition between producers to find an optimal price and product for consumers.

Peer review is supposed to do this, but the fact that peer reviewers are often colleagues leads to collusion, whether intended or not.

Maybe we need a separate body of scientists whose sole job—and whose entire prestige—derives from taking down and retracting bad science.

[1]: [https://en.wikipedia.org/wiki/Adversarial_system](https://en.wikipedia.org/wiki/Adversarial_system)

</comment><comment><USER67> It's unfortunate that the suggestions at the end don't seem to offer a realistic attack vector.

> First, scientists would need to be incentivized to perform replication > studies, through recognition and career advancement. Second, a database of > replication studies would need to be curated by the scientific community. > Third, mathematical derivations of replication-based metrics would need to > be developed and tested. Fourth, the new metrics would need to be integrated > into the scientific process without disrupting its flow.

Yes, absolutely those things need to happen, but the problem is how to get this funded, how to get people to not see reproducing results as career suicide, right? Items 2-4 will fall out as soon as item #1 happens.

How do we make item #1 happen? What things could be done to make reproducing results actually an attractive activity to scientists?

<comment><USER68> The problem is that, if you put mere reproduction as a goal, many scientists would see that as low hanging fruit to beef up the resume, so we'd get countless unnecessary "experiments".

I'd say the goal that gets credited should not be merely reproducing the results, but finding errors in the previous research. That would count as novel, and is something that is presently recognized as contribution. The only problem is that journals or conferences treat it as unattractive, so good luck publishing something of the kind...

<comment><USER67> > The problem is that, if you put mere reproduction as a goal, many scientists > would see that as low hanging fruit to beef up the resume, so we'd get > countless unnecessary "experiments".

Only if you assume the incentives for the 2nd, 3rd, 4th, etc. reproduction experiments remain the same, right? I wouldn't assume that, both because the first reproduction is the most valuable, and for the reasons Ahmed discussed in the article - that scientists are motivated by their perceived ability to do something novel. So first reproduction might be novel, but the fifth would certainly be less valuable, so I wouldn't personally assume we'd get a flood of useless experiments.

> I'd say the goal that gets credited should not be merely reproducing the > results, but finding errors in the previous research

Reproducing an experiment is meant to, without prejudice, either confirm or deny the previous research. It's not meant to confirm the previous results, it is meant to ask whether there could be errors in the research, but without assuming there are errors.

It _is_ novel to validate a result the first time, whether it's positive or negative, and for this incentive system to work, it has to appeal to people who might not find something dramatic or contradictory. It _must_ be appealing to do the work, regardless of the outcome, or it's not an incentive at all.

</comment></comment></comment><comment><USER69> I thought this was in the definition of "scientific"

<comment><USER17> Peer review is how most science is defined as science and peer review does not require reproduction of the work.

<comment><USER70> Much of what we peer review is not real science, at least in its definition of applying the scientific method.

For example, much of computer "science" is not. Math maybe, engineering probably, design sometimes, but "science" is rarely done. BUT the science envy is there, especially post 1990s, and it is as confusing as heck when multiple definitions of "science" collide in a conference culture.

Yes I'm a researcher, no I'm not a scientist.

</comment></comment></comment><comment><USER71> Define reproduced? Do we mean "conduct the same experiment multiple times so we can assess the variance on the outcome"? Or do we mean "conduct the same experiment multiple times to figure out if the first result is a screw-up"?

Those two aren't the same, and I think far too many think that the point is the latter when, imho, it's actually the former. Pure screwups will likely get found out, just like glaring bugs are usually found. It's when your result actually has a huge variance but you're looking at only one (or a few) samples and draw conclusions from it that's insidious, like the fact that it's the bugs that just change the output by a tiny bit that are the hardest to notice.

</comment><comment><USER72> I've always been amazed by how widely the Stanford Prison Experiment results are accepted when a) the experiment has not been repeated and b) the experiment didn't even get completed. It was stopped when the researchers had made up their minds about the results.

</comment><comment><USER73> So, we should have a structured way to represent that one study reproduces another? (e.g. that, with similar controls, the relation between the independent and dependent variables was sufficiently similar)

\- RDF is the best way to do this. RDF can be represented as RDFa (RDF in HTML) and as JSON-LD (JSON LinkedData).

... " #LinkedReproducibility "

[https://twitter.com/search?q=%23LinkedReproducibility](https://twitter.com/search?q=%23LinkedReproducibility)

It isn't/wouldn't be sufficient to, with one triple, say (example.org/studyX, 'reproduces', example.org/studyY); there is a reified relation (an EdgeClass) containing metadata like _who_ asserts that studyX reproduces studyY, _when_ they assert that, and _why_ (similar controls, similar outcome).

Today, we have to compare PDFs of studies and dig through them for links to the actual datasets from which the summary statistics were derived; so specifying _who_ is asserting that studyX reproduces studyY is very relevant.

Ideally, it should be possible to publish a study with structured premises which lead to a conclusion (probably with formats like RDFa and JSON-LD, and a comprehensive schema for logical argumentation which does not yet exist). ("#StructuredPremises")

Most simply, we should be able to say "the study control type URIs match", "the tabular column URIs match", "the samples were representative", and the identified relations were sufficiently within tolerances to say that studyX reproduces studyY.

Doing so in prosaic, parenthetical two-column PDFs is wasteful and shortsighted.

An individual researcher then, builds a set of beliefs about relations between factors in the world from a graph of studies ("#StudyGraph") with various quantitative and qualitative metadata attributes.

As fields, we would then expect our aggregate #StudyGraphs to indicate which relations between dependent and independent variables are relevant to prediction and actionable decision making (e.g. policy, research funding).

</comment><comment><USER74> According to old-school philosophy of science truth could be discovered only by removing all the nonsense, as a remainder, not by pilling up nonsense on top of nonsense out of math and probabilities.

Probabilities, for example, are not applicable to partially observed, guessed and modeled phenomena. It should be a type-error.

As for math - existence of a concept as a mathematical abstraction does not imply its existence outside the realms of so-called collective consciousness. Projecting mathematical concepts onto physical phenomena which could not be observed is a way to create chimeras and to get lost in them.

Read some Hegel to see how it works.)

</comment><comment><USER75> Ironically, one of the reasons Semmelweis's colleagues rejected his "hand- washing" hypothesis was that it did not have a good enough empirical/statistical basis.

[http://www.methodquarterly.com/2014/11/handwashing/](http://www.methodquarterly.com/2014/11/handwashing/) [https://en.wikipedia.org/wiki/Contemporary_reaction_to_Ignaz...](https://en.wikipedia.org/wiki/Contemporary_reaction_to_Ignaz_Semmelweis)

</comment><comment><USER76> Or at least, the media shouldn't report on results until they have been repeated. This would cut down on the daily "X causes cancer / X doesn't cause cancer" media spam.

</comment><comment><USER77> The solution is easy and it applies to most sciences: all research articles should include a pointer to download the dataset that was used and an annex with the details on how it was collected.

</comment><comment><USER78> Agreed, which means 50% of social science at least is disqualified and should not be making into future publications or become part of curriculum.

</comment><comment><USER79> Like climate science, right? Let's set up a statistical meaningful set of equivalent earths, and start doing some serious peer review.

</comment><comment><USER80> this increasingly includes code that needs to run in the future, and citations within code, see this group working in that field [https://www.force11.org/sites/default/files/shared- documents...](https://www.force11.org/sites/default/files/shared- documents/software-citation-principles.pdf)

</comment><comment><USER81> Just to play devils advocate, won't there be a self correcting mechanism?

If results are genuinely useful, then people will want to build upon that work, and will have to repeat the science. On the other hand if it can't be repeated, then it will not get further work done and fade into obscurity. Curious what other peoples opinion on this are?

</comment><comment><USER82> I think a better way of thinking about what we want than "repetition" is "independent corroboration".

</comment><comment><USER83> Sometimes in cs if your research is embedded in a huge ecosystem, it can become quite expensive to reproduce results. I mean proper reproduction, not just rerunning the Benchmarks. If you are dealing with complicated stuff, the reproducer might also just not be able to do the same thing technically.

<comment><USER84> Maybe, maybe not. Do you have something specific in mind here?

I hope researchers and scientists don't considers others not capable enough, and therefore withhold info on how to reproduce.

Even if the experiment is crazy expensive and complex right now it might be considered much more tractable in 10 years, or someone builds upon your work and invents a simpler method to show the same thing.

<comment><USER83> I am thinking of huge endeavors like building an asic or huge complex systems like virtual machines. Not always a comparable system for Repetition is available and must be built from scratch. Affording such rebuilds require huge sums.

Of course nobody does consider others not capable enough. Its just that there are not so many people experienced enough to build certain systems in a decent amount of time.

</comment></comment></comment><comment><USER85> In for instance, a bioscience lab, I don't believe that results should even be accepted unless they're repeated with similar reagents. Some reagents are so specific they only prove something.... for that one single thing, which could be unique on this planet.

</comment><comment><USER86> In that case, macro-economics is simply disqualified from being scientific. It's almost impossible to repeat large-scale events, controlling for all variables. Have to say I'm not particularly impressed with the quality of Nautilus's analysis.

<comment> T-A > In that case, macro-economics is simply disqualified from being scientific.

Well, duh? (In other words: of course it's not a science.)

</comment></comment><comment><USER87> One problem is that there is no incentive to replicate. From the PhD onwards, academia creates incentives for original research. Replications, particularly those that confirmed existing research, would not benefit the researcher much.

</comment><comment><USER88> From the authors of "Why Science Needs Metaphysics" this rings a little hollow.

Nautilus is just a slightly less vitriolic version of the Aeon-class anti- science postmodernist blog. Like Aeon, it's garbage.

</comment><comment><USER89> Independently verified and repeated I would add.

After all any scientific test that fails when somebody else repeats it is not science but the domain of magic and religion, so clearly not science.

</comment><comment><USER90> I searched for the word "tenure" in the article, but didn't find it.

The drive to get tenure is a big reason that scientists publish so much, funny that was not mentioned.

</comment><comment><USER91> What we need is accountable statistics - something that cannot be manipulated.

One idea is to enforce storing or indemnifying a time-stamped data base of the raw data on the block chain

</comment><comment><USER92> How do you that in the medical field? Studies are often based on a small number of patients affected by a particular condition.

</comment><comment><USER93> wait! we should use the scientific method for science?

its a radical suggestion. sad that it is, but true... ;)

seriously though... you can't have falsifiable results if you don't constantly try to falsify them. then it just becomes a result, which means the conclusion you can draw is close to nothing.... not quite nothing, but exceptionally close. :)

</comment><comment><USER94> There is no science without repetition.

</comment><comment><USER95> We also should not accept historical claims that have not been repeated :)

</comment><comment><USER3> We now have the tools to do it , and we should be doing it. The fate of scientific findings is not to publish papers, they belong to open and continuous scrutiny. And someone should build a github of scientific facts.

<comment><USER96> let's try it, if interested, send me a tweet at @isaacpei, seriously thinking about creating something for this

<comment><USER3> There is no lack of efforts to get scientists discussing (shameless related plug [http://sciboards.com](http://sciboards.com)), but unfortunately there is a disincentive for scientists to do so (politics).

</comment></comment></comment><comment><USER97> UGH. Scientists are not getting funds to repeat result.

</comment><comment><USER98> Lottery < Statistics < Science</comment>


Sweden Needs More Trash, Because It Has Turned All It's Got Into Energy - <AUTHOR> http://www.fastcoexist.com/1680763/sweden-needs-more-trash-because-it-has-turned-all-its-got-into-energy

<comment><USER2> The municipality in my city Linköping, Sweden has actually made a really impressive business model out of this. I pay them to pick up my garbage twice every month, and then I pay them to heat my house by burning that same garbage. And they have me put organic waste in special green bags that are separated from the other garbage using computer vision, making bio gas that I pay for to drive my car on.

<comment><USER3> It's like you are selling them an energy source for a negative amount of money.

</comment></comment><comment><USER4> In my home town of Trollhättan, the entire public transportation network was driven on biogas from the organic trash of the people. While sorting the garbage into organic/inorganic was a bit of a pain, the result was really impressive.

<comment><USER5> A bit OT, but I sure hope you trolls can resume building Saab cars soon. I understood they're going to be some sort of electric vehicles?

</comment></comment><comment><USER6> Does anyone have an idea what the impact of burning the trash has on the air pollution?

<comment><USER7> Burning trash releases (among other things) dioxins. They are thought to cause cancer, according to the EPA.

<http://cfpub.epa.gov/ncea/CFM/nceaQFind.cfm?keyword=Dioxin>

However, I am sure clever filtering could solve the problem.

(Credit where credit is due: A banned user posted a link to a UK guardian article about air pollution due to garbage incineration, giving me a good place to start my research.)

<comment><USER8> Dioxins are created by burning trash at low temperatures, e.g. in a personal burn barrel. At high temperatures dioxins are destroyed. Please don't scare the Swedes.

<comment><USER9> At high temperatures above 1000 degrees C, dioxin decomposes and the incinerators here are constructed for that.

Offical numbers: "The total emissions of dioxins to air in Sweden in 2008 was about 38 grams of TCDD equivalents."

Of that 5g is supposed to be from wasteburning. (That number is disputed by greenpeace who claims that to be 114g)

</comment></comment></comment></comment><comment><USER10> Wait, isn't it better to actually recycle this stuff, rather than just burn it? By burning it aren't we forever losing access to that physical matter by just turning it into heat radiation? What if we need it later?

EDIT: It sounds like the Swedes mainly burn the organic waste, and recycle what can be recycled. If that is the case, fair enough. Would have liked to see discussion about this in the article, seems fairly key.

<comment><USER11> The energy released is stored in the chemical bonds of the material being burned.

Matter is neither created nor destroyed in the process. Some of it "goes up in smoke", but we can use plants to capture those things back out of the air (either the green things or energy intensive industrial ones) .

<comment><USER10> Yeah, mainly correct but a small part of the matter is lost. That part which escapes from the system into space as IR radiation is practically speaking ever beyond our grasp. Its the Law of Conservation of matter/energy.

<comment><USER11> Human energy utilization is a fraction of a fraction of the solar energy that moves in and out of the earth system. Heat lost from burning is not something worth worrying about right now.

</comment></comment></comment></comment><comment><USER12> Same discussion from 10 days ago: <https://news.ycombinator.com/item?id=4706196></comment>


Wal-Mart Gives Consumers Number 1 Reason Why DRM is Not the Answer - <AUTHOR> http://www.readwriteweb.com/archives/wal-mart_gives_consumers_number_1_reason_why_drm_not_answer.php

<comment><USER2> How is this not illegal?</comment>