Why I've lost faith in p values - <AUTHOR> https://lucklab.ucdavis.edu/blog/2018/4/19/why-i-lost-faith-in-p-values <comment><USER3> Here's a more simple thought experiment that gets across the point of why p(null | significant effect) /= p(significant effect | null), and why p-values are flawed as stated in the post.

Imagine a society where scientists are really, really bad at hypothesis generation. In fact, they're so bad that they only test null hypothesis that are true. So in this hypothetical society, the null hypothesis in any scientific experiment ever done is true. But statistically using a p value of 0.05, we'll still reject the null in 5% of experiments. And those experiments will then end up being published in scientific literature. But then this society's scientific literature now only contains false results - <USER2> all published scientific results are false.

Of course, in real life, we hope that our scientists have better intuition for what is in fact true - that is, we hope that the "prior" probability in Bayes' theorem, p(null), is not 1.

<comment><USER4> > But statistically using a p value of 0.05, we'll still reject the null in 5% > of experiments. And those experiments will then end up being published in > scientific literature. But then this society's scientific literature now > only contains false results - literally all published scientific results are > false.

The problem with this picture is that it's showing publication as the end of the scientific story, and the acceptance of the finding as fact.

Publication should be the _start_ of a the story of a scientific finding. Then additional published experiments replicating the initial publication should comprise the next several chapters. A result shouldn't be accepted as anything other than partial evidence until it has been replicated multiple times by multiple different (and often competing) groups.

We need to start assigning WAY more importance, and way more credit, to replication. Instead of "publish or perish" we need "(publish | reproduce | disprove) or perish".

Edit: Maybe journals could issue "credits" for publishing replications of existing experiments, and require a researcher to "spend" a certain number of credits to publish an original paper?

<comment><USER5> That's a good idea: encourage researchers to focus on a mix of replication and new research. When writing grants, a part of that grant might be towards replicating interesting/unexpected results and the rest for new research. Moreover, given that the experiment has already been designed, replication could end up demanding much less effort from a PI and allow his students to gain some deliberate practice in experiment administration and publication. On the other hand, scholarly publication might have to be changed in order to allow for summary reporting of replication results to stave off a lot of repition.

<comment><USER6> My field has less of a "You publish first or you're not interesting" culture than many others, and part of what that is is recognizing that estimating an effect in a different population, with different underlying variables, is, itself, an interesting result all its own.

Tim Lash, the editor of Epidemiology, has some particularly cogent thoughts about replication, including some criticisms of what is rapidly becoming a "one size fits all" approach.

</comment></comment></comment></comment><comment><USER7> One of the best articles covering this issues is Meehl[1][2]. You can find discussion in various places like Gelman[3] and Reinhart[4].

[1] Meehl, Paul E (1990). Why summaries of research on psychological theories are often uninterpretable. Psychological Reports, 66(1), 195–244.

[2] [http://meehl.umn.edu/files/144whysummariespdf](http://meehl.umn.edu/files/144whysummariespdf)

[3] [http://andrewgelman.com/2015/03/23/paul-meehl-continues- boss...](http://andrewgelman.com/2015/03/23/paul-meehl-continues-boss/)

[4] [https://www.refsmmat.com/notebooks/meehl.html](https://www.refsmmat.com/notebooks/meehl.html)

</comment><comment><USER8> 'The fundamental problem is that p values don't mean what we "need" them to mean, that is p(null | significant effect).'

From Bayes' theorem, this more useful probability is given by p * x, where x = p(null) / p(significant effect). Maybe we could just lower the accepted threshold for statistical significance by several orders of magnitude so that, for statistically significant p, p * x is still small even for careful (i.e. big) estimates of x (e.g. maybe a Fermi approximation of the total number of experiments ever performed in the field in question). This doesn't necessarily imply impractically big sample sizes, although obviously this depends on the specifics (I believe the p value for a given value of the t-statistic decays exponentially with sample size).

<comment><USER9> I don't follow your argument. You've got two premises:

1) You are saying that people are committing the transposing the conditional fallacy: p(H0|data) != p(data|H0):

\- OK

2) You say to use Bayes theorem to get the value we want:

\- OK, but actually a better formulation is

      p(H_0|data) = p(H_0)*p(data|H_0)/[p(H_0)*p(data|H_0) + p(H_1)*p(data|H_1) + ... + p(H_n)*p(data|H_n)]

You probably don't need to add up all the way to hypothesis _n_ since the terms eventually become negligible and can be dropped from the denominator. The point is that you have to compare how likely the result would be under other hypotheses, not just H_0.

3) You propose lowering the threshold for "significance"

\- How does this follow from the premises? Lets say you get a very low value for p(H_0)p(data|H_0), this can still be much higher than p(H_1)p(data|H_1), etc so it is still the best choice. Ie, you can get a low p-value given H_0 but if there is no better model out there you should still keep H_0.

<comment><USER8> I'm assuming the question we are trying to answer is not "which H_n is most probable", but rather "how safe is it to conclude that H_0 is not true". For example, say we are concerned with whether the difference between two groups is less than a certain amount or greater than a certain amount.

<comment><USER9> >"how safe is it to conclude that H_0 is not true"

I would just take "very safe" as a principle, there is even the truism "all models are wrong".

>"the difference between two groups is less than a certain amount or greater than a certain amount"

You are ignoring a lot of the model being tested here (eg, normality, independence of the measurements, etc) and only considering one parameter.

<comment><USER8> "how safe is it to conclude that H_0 is not true"

What I meant here by H_0 was the hypothesis that the difference between groups is less than some particular threshold. I think if you made the threshold large enough then it would not be safe to conclude that H_0 is not true.

"You are ignoring a lot of the model being tested here (eg, normality, independence of the measurements, etc) and only considering one parameter. "

I said enough so that if you were arguing in good faith you could fill in the gaps yourself.

<comment><USER9> I am arguing in good faith. There may be some cases in physics where there is some theoretical distribution derived related specifically to the problem, and they believe the model to be actually 100% true. Otherwise, the model should be assumed to only be an approximation at best.

<comment><USER8> Yes, the t-test does assume normality and you can never be sure of perfect normality if that's what you are getting at (although I believe that simulation tests of the robustness of the t-test against deviations from normality generally show that this isn't too much of a practical concern). I wasn't trying to address every potential weakness with the t-test (or p-values in general); I was addressing the one stated in the article.

<comment><USER9> I'm saying that unless you have bothered to derive a statistical model from your theory, and you believe that theory may actually be correct, then you know that you will reject your model if enough time/money is spent on testing it.

<comment><USER8> Ok, I will assume that by "model" here you mean a probability distribution on the parameters relevant to your experiment. In that case I agree with what you just said: knowing exactly the correct model is impossible in a similar way that knowing someone's height to an infinite degree of precision is impossible. But I never said anything to the contrary. The H_0 I gave corresponds to an infinite set of models and not a single one (note that I said the difference is less than a certain threshold, not that the difference is 0 (although it still would be an infinite set in that case, but the probability would be 0)).

<comment><USER8> And in anticipation of the rebuttal that the probability is still 0 because it's never exactly normal, what I really meant originally but didn't write out explicitly for brevity and because I assumed it would be implicit: when I talk about giving an upper bound on p(null | data) from p(data | null), what I really mean is giving an upper bound on p(null | data, normal) from p(data | null, normal) where normal is the assumption that the distribution of whatever parameter we are looking at is normally distributed and null is the event that the difference in means between the two groups we are looking at it is less than some predetermined positive threshold. Or, for a 1-sample test, that the mean of a single group is within that threshold of some default value.

<comment><USER9> If you write out the actual calculation you will see normality (which was just one example of an assumption) is actually part of the null model being tested. It is not something different or outside of it.

<comment><USER8> That is just a trivial semantics issue, and yes I am familiar with the calculation.

Where do you think the flaw is specifically? Say we are doing a 1-sample test.

0\. (setup) Suppose we have a real number mu and a positive epsilon. Define the interval I as [mu – epsilon, mu + epsilon]. For each "candidate mean" within this interval, we have a corresponding t-statistic. Let the statistic t_0 be the inf of all these t-statistics. Let T be the event that t_0 is at least as big as the observed value.

1\. You can use Student's t-distribution to compute an upper bound for the probability of T under the assumptions that the observations are iid normal and the mean lies in I. I will call this probability p(T | null, normal, iid), where "null" is the event that the mean exists and is in I. It makes no difference that it is more typical to lump these assumptions together as "null" because in math you can define things however you want as long as you are consistent.

2\. We have that p(null | T, normal, iid) = p(T | null, normal, iid) * p(null | normal, iid) / p(T | normal, iid).

3\. Therefore, if we have an upper bound for x = p(null | normal, iid) / p(T | normal, iid) then we can get an upper bound for p(null | T, normal, iid). That is my main claim.

Which of the above statements do you object to?

<comment><USER9> >"You can use Student's t-distribution to compute an upper bound for the probability of T under the assumptions"

I'm not sure what you are arguing anymore. I am saying you will _never_ test a parameter value in isolation, it is always part of a model with other assumptions. There is simply no such thing as testing a parameter value alone. To define a likelihood you need more than simply a parameter...

You seemed to be disagreeing with that, but are now acknowledging the presence of the other assumptions.

<comment><USER8> "I'm not sure what you are arguing anymore."

It's the claim I make in 3, and then the secondary claim that making our upper bound on p(null | T, normal, iid) small for significant p-values (i.e. p(T | null, normal, id)) could be used as a criterion for whether our threshold for statistical significance is small enough.

"You seemed to be disagreeing with that"

I'm not sure what I said that gave that impression. I didn't mention anything about the normal / iid assumptions initially not because I thought we weren't making these assumptions but because I didn't think these details were essential to my point.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER10> Here is a better way to think about this.

The proper role of data is to update our existing beliefs about the world. It is not to specify what our beliefs should be.

The question that we really want to answer is, "What is the probability that X is true?" What p-values do is replace that with the seemingly similar but very different, "What is the probability that I'd have the evidence I have against X by chance alone were X true?" Bayesian factors try to capture the idea of how much belief should shift.

The conclusion at the end is that replication is better than either approach. I agree. We know that there are a lot of ways to hack p-values. Bayesian factors haven't caught on because they don't match how people want to think. However if we keep consistent research standards, and replicate routinely, the replication rate gives us a sense of how much confidence we should have in a new result that we hear about.

(Spoiler. A lot less confidence than most breathless science reporting would have you believe.)

<comment><USER11> This is like Functional programming , and people have a very hard time with it. Instead of passing around numbers "95% true" or whatever, we're passing around function "It's 2x as likely as you though it was, please insert your own prior and update", but even _worse_ , it's "please apply this complicated curve function at whatever value you chose for your prior". It's just too hard for people to manage. Computers can do it (but it's hard for them too, very computationally intensive), and you have to really trust your computer program to be working properly (and you have to put your ego in the incinerator!) to hand over your decision-making to the computer.

<comment><USER10> I question whether computers can do it at all in useful practice.

Take a look at the results quoted in [https://en.wikipedia.org/wiki/Bayesian_network#Inference_com...](https://en.wikipedia.org/wiki/Bayesian_network#Inference_complexity_and_approximation_algorithms) about how updating a Bayesian net is an NP hard problem, and even an approximation algorithm that gets the probability right to within 0.5 more than 0.5 of the time is NP-hard.

</comment></comment></comment><comment><USER12> My favorite probability theory problem is related to this article.

You have a test for a disease that is 99% accurate. This means that 99% of the time the test gives a correct result. You test positive for the disease and it is known that 1% of the population has the disease. What is the probability you have the disease?

The answer is not at all the one most people think at first when given this problem. This problem is why getting two tests is always a good thing to do when testing positive for a disease.

EDIT: I updated the statement of the problem to be one that can be answered!

<comment><USER13> I'm not sure you phrased the problem correctly. If we follow your explanation, then the probability of having the disease is indeed 99%.

If you want to show the implication of Bayes' Theorem then you need to be more precise : Say you have a 1% of false positive and false negative rates (99% reliability) and 1% of the population is sick. If you are tested positive, then the probability of being sick is much less than 99%.

<comment><USER14> > If we follow your explanation, then the probability of having the disease is > indeed 99%.

This is not correct; the probability of having the disease is unknown. He didn't say what he meant by the test being "99% accurate", but that doesn't mean you can just make your own assumption.

Note that in your more precisely specified scenario, when the test has 99% reliability, it is perfectly true that "99% of the time the test gives a correct result", which immediately disproves the claim that, if we follow that definition, the probability of having the disease given a positive test result is 99%.

<comment><USER13> The problem is that "99% the time gives a correct result" is imprecise.

It can be understood as both:

\- p(sick|positive) = 0.99 \- p(positive|sick) = 0.99

We get totally different results, the first one is obvious (99% change of being sick), and the second one needs Bayes' Theorem (and is the one we want to use).

<comment><USER14> I would only interpret "the test gives a correct result 99% of the time" to mean that out of every 100 test results, 99 are correct and one is wrong. Neither of your interpretations matches that. You need all kinds of additional information to say anything more specific. "99% of results are correct" can easily be true while p(sick | positive) and p(positive | sick) each vary anywhere between 0 and 1.

</comment></comment></comment></comment></comment><comment><USER15> The core issue is that p-values are cheaper to get than replicating the study, but replicating the study is the only reliable way to see if it's true or not. Sometimes the expensive/time-consuming way, is the only good way.

<comment><USER16> Replication by itself is not enough. You need pre-registration too. Otherwise you can p-hack the replications.

<comment><USER17> Pre-registration?

<comment><USER16> [http://www.apa.org/science/about/psa/2015/08/pre- registratio...](http://www.apa.org/science/about/psa/2015/08/pre- registration.aspx)

</comment></comment></comment></comment><comment><USER18> I'm not trying to be facetious, but isn't this something you learn in junior- level stats? I had this drilled in in both undergrad math courses and grad machine learning courses; I'm confused to see it warrant an article.

<comment><USER19> It's well known what p-values show. But they are, in practice, used as a gatekeeping mechanism in academic journals in many fields (including mine). Worse, getting p<0.05 is informally seen as a measure of practical significance, rather than simply as one statistical test amongst many passed.

So yes, it is something you learn in introductory quantitative methods classes. But I don't think most researchers understand just how much it matters.

Also, a key R package for producing regression tables of coefficients for journal articles is called 'stargazer'. Given the unwarranted focus of many readers on those indicia of 'significant' results, I think it's well named.

I currently have the opposite problem. Given that I work with very large online datasets (N=1M or so) _everything_ , including the random noise, is statistically significant to p<0.05. It really is effect sizes or busy at that point.

<comment><USER20> Real measures of practical significance are OR (odds ratio), effect size and dose response curve. Response histogram for statistical effects. (Or the 2D component analysis island histogram.)

</comment></comment></comment><comment><USER21> Andrew Gelman's blog provides regular insightful commentary on this issue, I highly recommend it:

[http://andrewgelman.com/](http://andrewgelman.com/)

The post that turned me on to all of this is at:

[http://andrewgelman.com/2016/09/21/what-has-happened-down- he...](http://andrewgelman.com/2016/09/21/what-has-happened-down-here-is-the- winds-have-changed/)

</comment><comment><USER22> The article says:

> Note: this has nothing to do with p-hacking (which is a huge but separate > issue).

I disagree. p-hacking is when one experimenter checks many statistical tests to find one that is significant. The effect the author is discussing is that many experimenters do many experiments and the significant ones get published. One is more unethical (or maybe just incompetent) than the other, but they're essentially the same phenomenon.

<comment><USER23> They are the same in that they both create a situation in which the p-value cannot be trusted. However, in one case this is deliberate. In the other it's a problem with the whole enterprise.

Also, running multiple tests without correcting for multiple testing (usually by reducing the threshold for significance) is just one form of p-hacking. The more insidious version is when one runs the test after every few participants until random chance makes it "slip over the edge of significance". In that case there might not even be enough variables for multiple testing to have occurred, and it becomes very difficult to detect.

</comment></comment><comment><USER24> I'm honestly more tired of essays about p-values than p-values.

It's true that like all metrics if it becomes a target then it maybe abused (Goodharts Law).

However if you abolished p-values people would start hacking or misunderstanding priors or confidence limits or OR.

It's an easy dumb stat that most anyone can do in excel and most everyone recognises. The emphasis should be that it remains a quick shorthand for casual use but that more complex studies have more sophisticated models and probabilistic reasoning.

But the emphasis on the p-values is bizarre. As best illustrated by JT Leek the pipeline of data research has multiple points of failure that may lead to false findings or irreproducible research. But we talk very little about them whilst essays about p-values come out every week...

[https://www.nature.com/news/statistics-p-values-are-just- the...](https://www.nature.com/news/statistics-p-values-are-just-the-tip-of- the-iceberg-1.17412#/pipe)

</comment><comment><USER25> This was a really interesting article. I've worked with researchers who try to defend a small but statistically significant finding that just doesn't seem likely to be real, and this provides a statistical explanation for my skepticism. The p-value mentality is deeply entrained in a lot of researchers, though

The challenge for journal editors seems very real. There's another group that deals with this challenge of interpreting the validity of significant findings for a living, though: biotech VCs. A lot of times trying to reproduce the work is their best way of addressing this, and often the first work done by startups is to try to replicate the academic work. For some other heuristics VCs use to assess "reproducibility risk", see here;

[https://lifescivc.com/2012/09/scientific-reproducibility- beg...](https://lifescivc.com/2012/09/scientific-reproducibility-begleys-six- rules/)

</comment><comment><USER26> 2 solutions: a) stop doing experiments that just look for correlation without any attempt to get at mechanism. Of course sometimes you can't avoid this and then 2) use lower p values. Don't waste thousands or millions (more) dollars following up 5% results.

</comment><comment><USER27> > Many researchers are now arguing that we should, more generally, move away > from using statistics to make all-or-none decisions and instead use them for > "estimation". In other words, instead of asking whether an effect is null or > not, we should ask how big the effect is likely to be given the data.

I couldn't agree more with this statement, and even moreso in a business setting than in research. It's just so easy to get caught up in statistical significance and lose perspective on practical significance. I've found confidence intervals most informative and easy to understand.

</comment><comment><USER28> When I was first taught statistics, I was told that the researcher had to justify a plausible hypothesis first - and then do a hypothesis test/p-value to prove their theory.

If this combination of the scientist's intuitive understanding and the p-value test result align, then this is a credible result.

On the other hand, the trend now is to conduct every possible test whether or not there is any justification for doing so (corrected for multiple testing, no p-hacking, yes, sure)

For example, in tech, we might test every shade of blue. Some of those blues are gonna come up as p-value hits - but since we had no good reason to do this test, this was probably just random noise.

Similarly, in genetics, we're gonna test every single gene against everything - just to see what happens (yes, yes, do a Bonferroni correction on each set of tests). Hmm, recent results in genetics don't seem to be very robust or repeatable, for some reason.

The likelihood of a truthful link in these tests is incredibly low. When have no particular reason to believe there is a truthful link, and are just blind testing, the false positive rate is very high (as described in the article), and probably even higher than the article speculates with - almost all hits are gonna be false positives.

Maybe p-values just don't work well with modern day data. Or, maybe, Big Data just doesn't contain information about mysterious, unexplored, and innovative correlations that we hope it does.

<comment><USER29> "On the other hand, the trend now is to conduct every possible test whether or not there is any justification for doing so (corrected for multiple testing, no p-hacking, yes, sure)"

You are literally describing p-hacking.

<comment><USER6> He's describing a multiple comparisons problem, not p-hacking, enabled by, essentially, ease of statistical computation. An honest researcher can trigger his problem without ever p-hacking.

See: Genome Wide Association Studies.

</comment></comment></comment><comment><USER30> > instead of asking whether an effect is null or not, we should ask how big > the effect is likely to be given the data. However, at the end of the day, > editors need to make an all-or-none decision about whether to publish a > paper

Yet another way in which the traditional publishing structure actively harms science.

<comment><USER31> Do you have an alternative way of publishing in mind?

<comment><USER30> My criticism is not necessarily constructive. ;) But it's not too hard to imagine something along the lines of arXiv combined with a rating/commenting system not unlike HN itself, combined with a Facebook-ish algorithm for surfacing articles relevant to each reader. The devil is in the details, of course, but it shouldn't be hard to do better than the expensive, artificially constrained, and arbitrary system we're saddled with now. The real trick will be convincing entrenched academics to switch -- I'm still not convinced this is actually possible.

</comment></comment></comment><comment><USER32> Here's a follow-up to the original blog post: [https://lucklab.ucdavis.edu/blog/2018/4/28/why-ive-lost- fait...](https://lucklab.ucdavis.edu/blog/2018/4/28/why-ive-lost-faith-in-p- values-part-2)

</comment><comment> alan-crowe I remember reading [http://andrewgelman.com/2016/11/13/more-on-my-paper-with- joh...](http://andrewgelman.com/2016/11/13/more-on-my-paper-with-john-carlin- on-type-m-and-type-s-errors/)

with its graph "This is what power = 0.06 looks like". So I got the point that you have to have sufficient statistical power. A useful rule of thumb is that you need a power of at least 0.8. You need to have some idea how big the effect is likely to. Perhaps from previous exploratory research, from claims of other researchers, from reasoning "well, if this is happening the way we think it is, there has to be an effect greater than x waiting to be discovered.". Then you work out how big a sample size you need to use. Then you roll up your sleeves and get down to work.

But the reason for using p values rather than Bayesian inference is that it gets you out of the tricky problem of coming up with a prior. You only need to think about the null hypothesis and ask yourself whether the probability of the data, given the null hypothesis, is less than 0.05.

So there is a bit of contradiction. p values don't really work unless you ensure that you have sufficient power. To do that you need a plausible effect size, to feed into your power calculation. And that is implicitly an rough approximate prior, 50:50 either null or that effect. You could just do a Bayesian update, stating how much you shifted from 50:50.

Basically, if you don't already know enough to have an arguable prior to get a Bayesian approach started, you don't know enough to do a power calculation, so you shouldn't be using p-values either.

I went looking on andrewgelman.com for a reference for wanting power = 0.8 and found a more recent post

[http://andrewgelman.com/2017/12/04/80-power- lie/](http://andrewgelman.com/2017/12/04/80-power-lie/)

Oh shit! The situation is much worse than I realised :-(

<comment><USER33> > But the reason for using p values rather than Bayesian inference is that it > gets you out of the tricky problem of coming up with a prior.

It technically doesn't even do this. Using a frequentist approach is equivalent to a Bayesian approach with an uninformative prior, which is itself an assumption being baked into the analysis, only one that is almost unquestionably incorrect. Its essentially saying you have literally no idea how a data is being generated, which is certainly not true.

</comment></comment><comment><USER34> James Abdey wrote his Ph.D. on this subject several year ago and proposed an alternative method for making decisions based on statistical evidence: [http://etheses.lse.ac.uk/31/](http://etheses.lse.ac.uk/31/)

</comment><comment><USER35> This is an old thread already and I don't know if I'm getting my voice heard. But at any rate: hypothesis testing (slightly different philosophically from p-values, but anyway) is bogus because conjectures-and-refutations falsificationism is bogus. That's not how good science has ever happened, only how bogus research programs have dressed themselves in science.

The core of science is "the unity of science". Signal-to-noise measurements tell you very little outside a general coherentist/holistic verificationist framework.

<comment><USER36> About unity of science: [https://en.wikipedia.org/wiki/Unity_of_science](https://en.wikipedia.org/wiki/Unity_of_science)

</comment></comment><comment><USER37> This is especially troubling when combined with confirmation bias. The whole point of data is that it anchors us to reality. Data should be the check that prevents us from believing something simply because we want it to be true. But if we only test theories we already suspect are true, we are already biasing the kinds of false positives we will get.

</comment><comment><USER38> pvals are a lot like the weather - everyone complains about it, nobody does anything about it. Specifically, what tends to be missing from these conversations is a good alternative - the author seems to be asking for false discovery rates/q values. Or maybe effect sizes? The reality is one size doesn't fit all, and the most useful statistic depends on the context. Oh, and the target: good luck submitting your work to a biological journal without pvals. I'm sure the editor will briefly marvel at your courage in taking a stand as she rejects without review.

While we're on the subject: there's a tendency to appeal to larger sample sizes, as the author also mentions. Worth remembering that for some of us data isn't a thing you download from the interweb, it's something you generate - and it costs money and time to do so. (And for human subjects research, the stakes are even higher...)

</comment><comment><USER39> I don't think there's any cause to abandon p-values and NHST if you're running experiments with high power and intelligent, deliberate priors.

With power = 0.8 and p(h1) = 0.6, p(h0 | p < 0.05) = 0.04. Even if power = 0.8 and p(h1) = 0.2 then p(h0 | p < 0.05) = 0.2.

</comment><comment><USER40> Does anyone have that article about how a pro/anti parapsychology both designed a study, analyzed the data, and got conflicting results? (There was a joke about how it was the only paper published that explained a discrepancy by saying the other side cheated)

</comment><comment><USER41> Is there a book (written in plain language) that goes into the history of academic journals and the details the current state of the "replication crisis", "data dredging", etc?

</comment><comment><USER42> Despite using statistics daily, I still feel utterly uncomfortable about its philosophical grounding. Are there any resources HN can suggest to soothe the heart of a sceptic?

<comment><USER18> Skeptic * quite different from sceptic :)

<comment><USER43> I think it's just an alternate spelling that is preferred in some locales: [https://en.wiktionary.org/wiki/sceptic](https://en.wiktionary.org/wiki/sceptic)

</comment></comment></comment><comment><USER44> p-values that are not in the physics ranges are ridiculous.

It's a shame everyone started copying physics but decided for higher acceptance/rejection values.

I was a little bit disappointed when I realized that a bunch of valid modern science is just proper experiment design and number crunching. If it's not physics, there's no models of why things work, there's just a p-value on the correlation or some other comparison function.

Medicine has turned into a field where you can't know a thing.

[http://www.cochrane.org/CD005427/BACK_combined- chiropractic-...](http://www.cochrane.org/CD005427/BACK_combined-chiropractic- interventions-for-low-back-pain)

I love reading reports like the above:

> There is currently no evidence that supports or refutes that these > interventions(chiropractic intervention) provide a clinically meaningful > difference for pain or disability in people with [lower back pain] when > compared to other interventions.

p-values really do not help that much.

<comment><USER45> > It's a shame everyone started copying physics but decided for higher > acceptance/rejection values.

P-values were developed outside of physics, it's not like people took them from physics relaxing the significance thresholds.

"If your experiment needs statistics, you ought to have done a better experiment." Lord Ernest Rutherford (maybe)

</comment></comment><comment><USER46> Sometimes dumbing down a concept can totally screw up a person's learning curve. During early days a lot of Java tutorials (in Indian engineering books) mentioned that the reason Java has Interfaces is because otherwise it is not possible to inherit from multiple classes. While it is true that you can implement multiple interfaces, the whole point of interface is to define "interface" without forcing an implementation. It has nothing to do with the "limitation" of single inheritance.

Coming back to p values. A simple google search will find you many articles that say

> A small p-value (typically ≤ 0.05) indicates strong evidence against the > null hypothesis, so you reject the null hypothesis.

The whole idea of p-values is to warn a scientist that they should look for statistical significance. The behaviour of hypothesis over infinite trials is what that matters and hence more data => better reliance. But "more", "better" etc. are subjective ideas and in many cases when everything else considered normal <0.05 might be good but not always. There are far too many factors such as wrong sampling method, things you can not measure vs things you can measure etc. that affect this number truly.

I think author nails it when he writes "Replication is the best statistic."

Always think of these tests in biological evolution perspective. Do you think this hypothetis would survive the test of time where it has to frequently face the real world ?

<comment><USER47> It seems to me that a number of things are not clear at all.

1) p-values is a metric on a whole library of procedures, not a single check (although one of them, the Normal distribution is the one usually meant). There's 5 basic ones everyone should know, and you could fill a decent bookshelf with details on all of them.

This means a true p-value should be accompanied by

a) what the source data was

b) how it's distributed, and why (better yet, proof). This should include assurances that there are no attempts to game the measurement (or otherwise any change in the source data directly related to this measurement), as that of course invalidates it.

c) a sanity check (like a normality test, or redoing the procedure on generated data and verifying the expected result)

d) what the exact claim is (e.g. this is normally distributed with a mean > X)

e) what the procedure was to verify this claim (e.g. normality test + mean > X ... you need to COMBINE your p values, because if the data isn't normal your P test is invalid and of course your hypothesis should be rejected even if the numbers in the t-test say it shouldn't)

f) a HUGE disclaimer that this measurement and it's actual numerical result only apply to past values, and if it's used to change something it is no longer valid, even if it's numerical value changes as a result, and you can still calculate it.

2) it's not a given that it's possible at all. The truth is that there are many things that don't follow the central limit theorem and therefore cannot correctly be used for statistical approximation.

Essentially you need to be utterly convinced that every measurement is the result of somehow combinding a large number of effects that keep reappearing.

For instance, planetary orbits do NOT satisfy this criterion. Sure they're the result of large numbers of influences, but almost none of those influences ever repeat (e.g. comets passing in close orbits is a big modifier of orbits, and in 4-5 billion years that'll happen once with any given asteroid. Almost every perturbation is unique and doesn't repeat, so it can't be predicted and won't follow statistical laws. That is the sort of effect that no form of statistics will ever find, and that invalidates your results. Because these are rare it works in the short term, but it doesn't work in the long term. E.g. if you repeat the experiment to determine the speed of light with Jupiter's moons, that won't work with the original data because their orbits have shifted too much)

<comment><USER46> You are far more articulate than I am.

</comment></comment></comment><comment><USER48> I never understood why people took p-values seriously. They never seemed to mean anything of use.

Whenever I brought it up around other academics, no one seemed to want to comment on it. Maybe they were afraid to admit they didn't understand a topic that's apparently important to publishing? Anyone can follow the formula to make a p-value, but there's no requirement to understand its meaning.

I'd love to find their use, but I still haven't found it.

<comment><USER49> > there's no requirement to understand its meaning.

This is a feature, not a bug. The fact that the p-value has nothing to do with whether or not the hypothesis is correct is intended to force people to think for themselves rather than using math as a substitute for logic.

<comment><USER50> If a feature is working in reverse it's probably a bug though- it might be _intended_ to force independent thinking, but in reality all it does is serve as a gameable goalpost, no thought required.

</comment></comment></comment><comment><USER9> tldr: The author got a PhD in 1993[1] and is just now figuring out that p-values are not false positive rates

[1][http://mindbrain.ucdavis.edu/people/sjluck](http://mindbrain.ucdavis.edu/people/sjluck)

I was lucky and figured it out before getting a degree. Its got to be hard for people in this position to look back on their previous work where the most fundamental aspect of interpreting the results was incorrect.

He gets it right that statistics are good for estimation, but there is a part two. You need to come up with a theory that makes a prediction to compare to these estimates, and then test _that_. Ie, your prediction about the distribution of the results is the "null hypothesis". I think p-values are probably ok for that.

<comment><USER51> I've met Steve Luck multiple times over the last decade. He is a very rigorous and insightful scientist, with expertise in a wide range of methods (especially eeg) and psychological phenomena. His wife Prof. Lisa Oaks may be even more impressive in these regards, but that is an aside. The point I would like to make though is that there are lots of aspects to research and stats is just one part to master of many. It should not be up to a neuroscientist to make posts like this. The stats community should be ___actively_ __pushing the scientific community to alternatives that match interpretative intuition with reality of the statistical metric.

<comment><USER9> I went through the same thing. I was taught the same half-assed BS statistics with all the wrong interpretations and was surrounded by people just following along with that.

Still, it was a point of duty/honor for me to figure out what these statistical results meant. How is interpreting results not a key part of a scientists job? I guess if you don't want to bother learning to interpret results then you can do science by either being a lab tech or theorist.

</comment></comment></comment><comment><USER14> Sounds like someone who never understood statistics, still doesn't, and doesn't want to.

A particularly glaring issue is this offhand comment:

> this is a statement about what happens when the null hypothesis is actually > true. In real research, we don't know whether the null hypothesis is > actually true. If we knew that, we wouldn't need any statistics! In real > research, we have a p value, and we want to know whether we should accept or > reject the null hypothesis.

That isn't a question that _any_ statistical approach will help you with. There's a reason we talk in terms of "rejecting" or "failing to reject" a hypothesis. We don't do statistical tests to accept hypotheses, only to reject them.

The concept of accepting one hypothesis based on a comparison between it and one other hypothesis is ludicrous on its face, suffering exactly the problems associated with Pascal's wager.

<comment><USER52> He clearly meant "fail to reject" where he said "accept", you're quibbling about semantics. The whole post is about how p-values (and indeed, known statistical techniques) don't actually help you decide whether or not you should reject the null.

<comment><USER14> The post is about a confusion between the question that p-values address ("how likely is this data to have come from the null hypothesis?") and another very different question ("given several findings that were unlikely to have come from various different hypotheses, how many are likely to be spurious?").</comment></comment></comment>


Ask HN: Safe to release source code after DMCA? - <AUTHOR>

This past spring, I created a popular Tetris clone that I released for free on the Android Market. After I had finished and released it, I started to hear about how protective Tetris Holding is of their trademark. After a few months, as expected, along came a DCMA take down from Tetris Holding and Google removed my game from the market. End of that story.
My question is, would it be "safe" for me to put the source code on GitHub for others to learn from and perhaps even learn something myself from others who make suggestions to improve it?
The reason I ask is if someone takes my code and builds and releases it on the Android Market, I would rather not find myself in a situation where I am in any way liable. (Risk averse...)
Basically, I am wondering if anyone knows of any red flags I should be aware of regarding this, or how I could do it safer (or if I don't even need to worry). Thanks!
Also, the clone is called Titres. (I hope some of you got your hands on it before it got taken down!) <comment><USER2> It might be better to call it "falling blocks", or another generic name, or else you might get another DMCA request.</comment>


How to Crunch Your Resume for Optimum Appeal to Automatic Tracking Systems - <AUTHOR> http://blog.smartbear.com/software-quality/bid/246678/How-to-Crunch-Your-Resume-for-Optimum-Appeal-to-Automatic-Tracking-Systems

<comment><USER2> Quotes from the article:

"The problem with PDFs, of course, is that they represent text as images, which glitches when going through rendering."

"Lesson learned: Use Word or another open format, not PDF."

The conclusions (don't rely on online applications) are correct but man.</comment>


Ask HN: How to manage burnout / jadedness and job hunt? - <AUTHOR> I am currently employed at a small web dev agency and it's killing me.
There is no focus on quality of work just an obsession with older technologies and a lack of any desire to better understand the tools the developers are forced to work with.
The owner of the organization is more interested with being a sales person and has no concern for the technical ramifications of their sloppy "yes man" dealmaking approach. It has financially cost the company money, driven away all our senior developers / architects. Management responded by taking a magnifying glass to all the time spent by developers... while conveniently ignoring their own role in the failures.
My problem is that the day to day grind keeps me in a negative mental state. It's impossible to get the stink of desperation off and causes me to interview very poorly. I am not doing technical work up to the level I am capable of and feel rusty / out of practice with whatever opportunity I am reaching for.
Depression causes things I used to enjoy to feel like a chore... I literally cannot will myself to do things that I was intensely driven to do months ago. Being forced to return daily to this negative mental space prevents me from effectively being the creative, positive person I know I can be and that I need to be to be offered a new job.
I am currently in therapy and have been for the past 6 months or so. It is helping but there is no medication, therapy or treatment for a bad job... you just have to get up and move on.
How do you stay motivated to work on portfolio pieces when your job makes you hate what you used to enjoy?
How do you talk about the work you have done for the past 1+yrs when all that comes to mind is how inept and terrible the people you work for have been?
I have been abused and neglected for so long that I'm unable to let go of my victim complex. Please help.
If you need a remote web developer with 10+yrs of PHP / Node experience please DM me <comment><USER3> "It's impossible to get the stink of desperation off and causes me to interview very poorly."

followed by

"I have been abused and neglected for so long that I'm unable to let go of my victim complex. Please help.

If you need a remote web developer with 10+yrs of PHP / Node experience please DM me"

is kinda rough...

It seems like you may be your own worse enemy here. You usually don't need portfolio pieces to find a new job. You have 10 years of experience. I can't speak to your mental health, and I wish you the best on that. However, appearing desperate in a job interview is in general a red flag (and you know this).

Just keep on applying and interviewing. Stop brooding over the things your coworkers are doing wrong (guess what: they probably can say the same about you). You don't need to give specifics about where you work so much, you can "make things up" (just don't lie about your abilities). Also, if you don't have savings for extended job breaks, that sounds like something you should start on because a job is not worth ruining your life over.

Good luck :(

<comment><USER4> I agree with this. You must embellish in interviews. The person hiring you is going to talk-up their company or team and leave out the bad - you need to do the same.

</comment></comment><comment><USER4> I'm sorry to hear that. I hope you find a new job.

I am in a similar situation. I have been repeatedly passed over and screwed over by my company for the past 5 years. They also don't follow their own policies and show poor leadership as you mentioned.

I handle this situation by trying to ignore the facts and tell myself I have a good job (the company's propaganda helps). I was once told by a manager that a high rating is awarded, not earned. This mindset helps. I was also told by a manager that not everyone can be more than an intermediate developer, so this helps me accept that I won't be promoted again.

It's a little disappointing to live like this, but it will be ok. I notice a degradation in performance and joy, but I still have a job.

I think your position is similar to many others', like mine. Just live life and pretend your job doesn't matter.

</comment><comment><USER5> Step back and realize your situation is simple.

Goal: Get a new job.

Actions: 1\. Live healthy, sleep well. Run. 2\. Spruce up resume 3\. Apply for jobs 4\. Cruise at current job

I find that when you think you have the weight of the world on your shoulders, super straightforward tasks seem insurmountable.

Think of it like a dumb person would: "I don't like this job. I'm gonna get a new one"

<comment><AUTHOR> Everything except this in your plan is doable:

>4\. Cruise at current job

It's such a chaotic shit show... more people are leaving in the next few weeks and it's going to very negatively affect me.

>I find that when you think you have the weight of the world on your shoulders, super straightforward tasks seem insurmountable.

It feels unsurmountable because I have no viable path to success or resolution right now. Whenever a new job lead or something else comes up it is an instant uplift of spirits. If only we could schedule success.

I really appreciate your simplifying of things and reminder of the things unrelated to programming that are essential to success.

<comment><USER6> The situation sounds impossible. I hope you are able to manage when more people lave.

That being said, what's going on right now? Literally right in this moment.

What is the next, smallest, most immediate action? Do that.

</comment></comment></comment><comment><USER7> Have you tried automating some of your work? It helps if the works is _automatable_ and mundane.

</comment><comment><USER8> I think you just need to get a new job. Start applying today.

</comment><comment><USER9> I started as a dev in a small software consultancy and worked there for 9+ years. I too felt the owner was too concerned to saying yes to every customer and contract and gave short shrift to the technical side of the business. I also felt similarly oppressed by management.

It's not always clear cut though - since running my own development company I've come to appreciate the difficulty of making payroll, of keeping the contracts coming in so that there's never a month where we're short and have nothing for our team to do.

It's tough, certainly for you, but also for management too. Try and see both sides, I guarantee there are few bosses who are intentionally malignant, just unintentionally so!

Regarding your skills stagnating, yes, that's partly on your employer for having a boring and crufty tech stack but it's also partly your responsibility.

I spent close to a decade focusing exclusively on my employer's tech stack, it was pure JSP/Java 6 built around an old school asynchronous messaging engine. Whilst I am a proficient back end architect I ignored the world of HTTP, REST, Websockets and web development so by the time I left my employer and entered the job market I didn't even know what a HTTP verb was!

It took 6 months of intensive crash courses and small side projects around the training to solidify the current web paradigm. That de-skilling was on me.

It would've been great if my employer actively sent me on mentoring and training programs, etc. but even when I've contracted at super large companies they often pay short shrift to training in hot new skills and focus more on health/safety training and other corporate compliance.

Your skills and qualifications are on you.

As far as motivation and burnout goes, this one is a bummer. Towards the end of it I was literally having difficulty breathing whilst working for my previous employer. I found it impossible to get up before 10am and struggled to stay motivated. My health started to inexplicably fail and I was getting aches all over. I eventually found out I had symptoms of depression and to get out of it took a long time.

I started by chipping away at simple things. Walk after dinner round the block, every day, non-negotiable. That morphed into daily exercise, which morphed into eating better, which gave me the energy to start weekend coding and training on the skills I was missing.

It's still tough, I still miss workouts all-the-time and I still lapse into a funk - but it's good knowing the funk is temporary, that the mind plays tricks on itself and catastrophises or kids itself that the present situation will stretch into eternity. Everything is temporary, which is good because bad situations can turn into good situations.

It takes time, don't be hard on yourself, or blame yourself. Recognizing that just as I had a hand to play in putting myself into a predicament so too I had a part to play in getting myself out of it. It takes baby steps and time.

Good luck!

</comment><comment><USER10> to start, i would see if you can switch to 30 hours/week.

so, you stay full time, keep the bennies, take the 25% pay cut, done.

if you can actually get there, that alone would be a huge win.

if you're not too worried about losing your job (you should be), then just ask your boss today - right now - stop reading this, slack your boss, "Hey boss - can we have a quick phone chat? Short story is -- I wanted to see if I could switch to 30 hours/week, stay 'full time' (i.e. keep my benefits), take 25% pay cut. That's the gist of it."

Why?

Ugh. I don't know, honestly. Make something up that sounds reasonable. You want to learn how to paint watercolor.

You're not burned out, you're not unhappy, you love pandemics and life and you're so gd thankful for everything all the time and happy and god bless america, but for some reason you only want to do 30 hours/week. Ideally in 4 days a week instead of 5, but even 5 is doable -- 6 hour days instead of 8.

see what the answer is.

worst that can happen is you get shitcanned on the spot, and now you're in deep shit.

so, tread carefully.

but let us know if you decide to do this, and if you got shitcanned or not.

outside of that, i'd suggest as many of these as you can do, even if sporadically, etc.:

       - meditate 5+ minutes/day (headspace)
       - fast 16:8 (zero)
       - exercise/walk 12+ miles/day (pedometer step counter)
       - eat veggies (even if you still eat a bunch of shit)

my personal fave of those is the exercise. even walking - at least 90min - <USER2> make a _big_ diff in your mental and physical. and you get to listen to some great stories/pods/books.

i feel like i've been where you are -- about 20 or so times -- really -- so nothing will _actually_ help, except changing jobs -- but the goal is to stay semi-sane and ideally employed while you sort shit out.

yeah, crushing yourself with real endurance exercise, is a great way to go.

also take a week+ vacation if you can. again, try not to get shitcanned.

also, small web dev shop seems like an awesome place to be. but might depend what you're looking for. i've always wanted to be closer to 'the business' \-- i.e. the 'sales/dealmaking' \-- but your tone regarding the owner makes you sound like...me and every other nerd who has at least at one point in time despised the sales folks for '[insert reason here]'. point of this graf is...put yourself in the sales folks' shoes. my current take on the 'business' vs. 'tech' debate hasn't shifted much in the last 20 years -- once I figured out how important sales was to business, and tried to do it myself, i started getting a _lot_ less cranky about shady/shitty business/sales folks. and the point of the empathy is it might _greatly_ reduce your jadedness, and make you happier/healthier/smarter/better.

shoot - one thing i never tried but think would have been cool -- try to help the company get some sales. but i would not go out there and try to get sales myself - i would ask the owner.

"hey bro, i still love nerding out, but i'd like to learn more about sales and getting customers and return work and all the things that make this company go. i have no idea how i could possibly help, but...i do follow tech and business news, and...".

ok, that's hella awkward, but the point is, imagine you were for a day put in charge, by the supreme being of this dystopia called Earth2020, of helping your company grow - make more money/revenue, get more customers, sell more shit -- what would you do to achieve this goal?

start reading more about Apple's VR play, and how small businesses might be able to benefit? look to sell solutions that just happen to coincide with your dev interests? you ask bossman, "What do you think about when you call a previous customer? Do you just call and say hi, or do you have a specific idea, or shoot the breeze, or talk about VR, or...?"

i do like the idea that...if you're about to try to do something that is relatively or extremely uncomfortable for you to do, it's relatively or extremely the right thing for you to do, respectively. act upon the universe. impose your will. all that type of shit.</comment>


New Weather Service supercomputer faces chaos - <AUTHOR> http://www.washingtonpost.com/blogs/capital-weather-gang/wp/2013/06/25/new-weather-service-supercomputer-faces-chaos/ <comment><USER2> Tracking down numerical errors is brutal. I worked at a company re- implementing some MATLAB research code into optimized C++ (with parallelization where possible), where output had to match exactly bug-for- bug[1]. The algorithms were obviously very simple compared to weather forecasting on supercomputers, but it was still incredibly tedious dumping & comparing MATLAB debug values at dozens of points across multiple scenarios, each of which took hours to run. We even calculated one of the datasets they gave us would take 86 years(!!!) to process in the original MATLAB code. Sympathy for the team working on this. Fond memories of getting enormous arrays of NaNs after a fix.

[1] Bug-for-bug _means_ bug-for-bug. There was a clear attempt to sort an array in the research code, which was implemented as a single pass through the data just deleting elements that were smaller than their direct predecessor. Research code is weird. It's one of the few places in software where "hack it till it works" actually produces valuable results.

</comment><comment><USER3> This post is years old, and those systems went into production in July of 2013.

</comment><comment><USER4> As explained well by Nate Silver's "Signal and the Noise", when the weather forecast model says a 50% chance of rain, that doesn't mean it "might" rain, that means that in 50.001293% of millions of simulations precipitation above some threshold developed in the spot in question.

A news station can't get away with saying 50% though. The public assumes that's a cop-out answer. As such they will say 60% even if the real number is 50%. They will never say 40%. Better to seem over-prepared then rain on you when it wasn't supposed to.

Even if we had the perfect computer, at the end of the day it's got to get to your average person through a local news weather reporter.

<comment><USER5> > Even if we had the perfect computer, at the end of the day it's got to get > to your average person through a local news weather reporter.

An average person maybe, but not us if you use Dark Sky for iOS or its web equivalent [http://forecast.io/](http://forecast.io/)

I'm simplifying but it essentially plugs your GPS coordinates into a weather model and gives you the output.

<comment><USER6> I have to say I was using a feature like this on the NOAA site well before there was an App Store or .io domains.

</comment></comment></comment><comment><USER7> I do not understand why this is news. All CA (cellular automata) simulations tend to diverge in various ways as they run forward. In fact, the unusual situation is where they have some sort of stability. With the degree to which CA is being used, why hasn't this become common knowledge? All these folks going on and on about "mathematical models" of this or that, and they don't even understand the nature of the math? Huh?

More counter-intuitiveness follows. The more complicated you make the model, oddly enough, the less likely it is to be stable. (I'm looking at you, weather model)

 _I suspect that differences are random so that on average they are performing statistically about the same._

Oddly enough, nope! Nice try, though. The rate of divergence is unpredictable and not guaranteed to average out to anything. In fact, that's the definition of chaotic. Unless you've discovered an attractor, you just have unpredictable noise. Add that with CA and you have systems that diverge in various ways at various rates. Sure if you're sample size was _huge_ , it'd probably all work out. But 2-3 or even a few dozen? Not so much.

<comment><USER8> Discretizing your domain into a grid doesn't necessarily imply you're solving the problem in question using cellular automata. I'm not familiar with the specifics of GFS, but it's common to model physical phenomena using the finite difference method.

It's newsworthy and interesting because they're running the same exact model on two different computers and it's yielding wildly divergent results. It'd be as if you took the same CA rules with the same initial state and it ran differently on two different computers.

It's tricky because these discrepancies are probably due to subtle differences like the way the new CPUs are handling the floating point calculations.

</comment></comment><comment><USER9> So assuming the two programs running are in fact the exact same, it seems like there's some error in one of the systems' floating point implementations.

Either the old computer (and systems that came before it) were producing the wrong results, and the new one is more accurate.

Or the old system (and systems that came before it) produced the right results, and the new system has a bug.

These things do happen, but it's very expensive to fix hardware.

<comment><USER10> Or both systems are wrong, but just wrong in different ways.

<comment><USER11> Or both are right, and the software is hitting some undefined behaviour in the compiler. The article does in fact mention that a different compiler version was used.

With C is quite easy to happen, don't know about fortran though.

<comment><USER12> I would call an error placed by the compiler due to failure to exactly follow standards a problem in the system.

We are talking totality of systems here, the compiler that produces the programs _is part of the system_ that uses the program.

<comment><USER11> I wasn't referring to failures of the compiler to follow the standards, but rather to failures of the _software_ to follow the standard.

"So assuming the two programs running are in fact the exact same,"

The source code for the programs might as well be the same, but the actual (binary) program is not necessarily the same.

Or to put it in other words, both systems might be perfectly correct, but there is a class of software bugs that get revealed only when incarnated after being compiled in one of the several perfectly legit ways. The dimension of this set of possible alternative optimisations can be quite high if your language offers many areas for undefined behaviour.

Does anybody know if fortran is plagued by it as much as C ?

</comment></comment></comment></comment></comment><comment><USER10> M&S is one of the areas I work in, and there are quite a few things to examine. Hardware floating point implementations are one area. Compiler (internals and/or flags) and assembly language differences (as author alluded to) are others. Heck, even a thread race condition might be showing up now for whatever reason.

</comment><comment><USER13> I think this merely points out that both prediction systems were basically B.S. to begin with. Ian Malcolm would agree.

<comment><USER12> I don't see how you got there. Numerical analysis is a whole branch of computer science which regularly deals with slight differences in computation. The usage of a constant (like pi) that is one bit different could cascade into massive differences in the results between the two programs.

Neither are completely wrong (as their results are similar) they just aren't exactly the same, especially when dealing with 7 day forecasts, this isn't to be entirely unexpected. We are trying to approximate massive systems we don't fully understand. A physics simulator is decent at getting a broad sense of things, but is not precise enough to get it exactly right, even though we use them to fly airplanes, same thing here.

<comment><USER14> "The usage of a constant (like pi) that is one bit different..."

That brings back memories of debugging a Fast Fourier Transform code that had a slightly miscoded version of pi. Someone typed it in from memory, to a dozen or so places (single precision), and got it slightly wrong.

Talk about weird and hard to track down. Like being transported to the parallel universe where the cars ride on hexagons.

<comment><USER15> We could do that here... More like a universe where you can tile a floor with pentagons.

</comment></comment></comment></comment><comment><USER16> Mathematician has proven by the Lorenz equations that you can do weather predictions up to 3 days adding more computing power on it won't help since its a chaotic system. Did you know that the weather forecast for the next day is abort 45% accurate?

<comment><USER17> It is more-less globally true; locally it depends on the weather itself -- sometimes you can make up to few weeks and be confident about it from ensemble self-consistency (think desert in dry period), sometimes it blows up few hours into future. Also there is no way of defining single "weather prediction accuracy"; this one you cite is probably some meaningless mixture of various factors (;</comment></comment>


Beyond Gravity: the complex quest to take out our orbital trash - <AUTHOR> http://arstechnica.com/science/2014/05/beyond-gravity-the-complex-quest-to-take-out-our-orbital-trash <comment><USER2> Have they considered some kind of super heavy gravitational mop. Something big enough to attract enough small particles and just 'mop' them up and send them into the sun or towards Earth?

<comment><USER3> I'm sure the idea has been considered - and then ignored as completely ridiculous when you do the math.

The biggest rocket ever built (the Saturn V) could take 140,000kg to low earth orbit. That's nowhere near big enough to have any kind of significant gravitational attraction to "mop" objects away from their orbits. Some kind of magnet would be more feasible, but still not practical because operating a huge magnet in the Earth's magnetic field would affect the spacecraft itself.

Finally, once you would have "mopped" these objects into neat bunches, it would take 4 km/s additional velocity to escape the earth and a whopping 30 km/s to send things towards the sun (without any gravitational slingshot). Now you can calculate the amount of propellant (using the Rocket Equation) that it would take to send any meaningful payload that distance.

</comment></comment><comment><USER4> Planetes: a hard science manga with space debris as a core story element.

[http://en.wikipedia.org/wiki/Planetes](http://en.wikipedia.org/wiki/Planetes)

</comment><comment> guard-of-terra Why can't we launch something like blobs of clay. Let objects stuck in it, and then it would deorbit on its own.

<comment><USER5> They're too fast I guess. Try catching a rifle bullet with a blob of clay. And space debries can be way faster, i.e. have more kinetic energy.

</comment></comment><comment><USER6> Why can't we gradually push the largest objects away from earth into a graveyard orbit and make them spiral out and away from earth over time?

<comment><USER3> It requires energy to move objects in space, you'd need to add velocity (in the order of kilometers per second) to move objects out to a graveyard orbit. And once they are at a graveyard orbit, nothing would make them "spiral out and away" from earth. The lunar gravity is not strong enough at practical altitudes to do this (this is known as a Lissajous orbit [0], used sometimes to put objects to Earth-Moon L2 point).

Geostationary satellites are put into a graveyard orbit a little bit above the geostationary orbit (but they stay there).

Moving objects from lower altitudes up is not feasible due to the propellant requirements and the fact that the old satellites/debris are not made for that.

And you can't just chuck stuff away from Earth either (that needs 11km/s of velocity w.r.t the Earth in the first place), they will remain in a heliocentric orbit near the Earth and perturbations from the Earth, the Moon, and the planets may bring those objects back, travelling at interplanetary velocities. For example, the S-IVB stages of Apollo missions were sent to heliocentric orbit (with a gravity assist from the Moon), but the third stage of Apollo 12 eventually came back to Earth orbit in 2002 (and may come again in 2040) [1]

[0] [http://en.wikipedia.org/wiki/Lissajous_orbit](http://en.wikipedia.org/wiki/Lissajous_orbit) [1] [http://en.wikipedia.org/wiki/J002E3](http://en.wikipedia.org/wiki/J002E3)

</comment></comment><comment><USER7> Let's clean up our oceans first? That plastic island out there has irritated me for years. Oh yea--and maybe stop eating so much fish? That includes Sushi. I had a girlfriend who called herself a strict vegetarian, but fish don't matter. Yea, we had numerous arguments about the power of denial.

</comment><comment><USER8> This article is junk. It describes in very alarming way space junk, but it never even mentions atmosphere drag at LEO and natural orbital decay. ISS has to periodically increase its orbit and it already brought down Skylab and to some extend Mir.

> and amped the volume up to 11 for dramatic purposes. The filmmakers kept > only as much science as they felt like keeping

> We've already reached the point where the growth of debris in low Earth > orbit (LEO) has become self-sustaining.

<comment><USER9> The problem will not solve itself if it's ignored. All the current calculations of future positions of course include the effects of the changes of the orbits.

<comment><USER8> It would in a few decades.

<comment><USER10> Only if we stopped putting more crap up there for those few decades (where a "few" is probably more on the order 10 to 100 ... yes, _decades_ ). Human exploration and exploitation of space resources is far too important to the short and medium term benefit of mankind — leaving entirely aside its implications for our _long term survival_ — to wait it out.

<comment><USER11> Or put it another way: the GPS system is used to synchronize power phases, guide aircraft, cars, people, ships. It's built into emergency transponders, military drones and bombs.

Without that system, we're back to compasses, line of sight and dead reckoning. We lose the current insane advantage of realtime positioning anywhere on the planet.

That's 1 18 satellite system - and we absolutely can't run a modern civilization without it.

<comment><USER12> Actually, nowadays a system of ground stations and high altitudes atmospheric floating balloons can be made to serve the very same purpose. The ground stations would serve for triangulation of balloon's position and those at their turn could serve the rest of the service-consumer base. As a bonus, the balloons would be obviously more manageable compared to satellites, the cost of taking them into operation would be of course much lower (no more atmospheric pollution), and be disposed cleanly without causing the problem discussed in the article.</comment></comment></comment></comment></comment></comment>


Ask HN: How do you find developers with same interests, for side projects? - <AUTHOR> I have a lot of side projects on which I try to work a bit everytime I have the opportunity, but honestly sometimes it feels like I will never finish all the work I'm giving to myself.
Contributing on GitHub and other platforms is nice, but it's often one way contributions: when you contribute to a project, there are few chances the author will contribute back to one of yours. Of course I'm not asking for the authors of projects I contributed to to contribute back, as it's not something I expect from people or myself. But I would love to know some developers with projects at the same level of popularity as mine, who want/need help and are willing to contribute back, and with who I could discuss, share ideas, plan sprints, etc.
In short: I would like to create a friendly group of developers who share the same interests. For example, for me it would be around CLI tools, but more specifically, CLI tools that help managing repositories, shell libraries (Bash/Zsh), and tools that help improving your comfort/productivity in the terminal.
The question is: how to find them? How do YOU find them, if you do? I don't know any platform serving this purpose. Of course I can just go on GitHub and contact people with projects similar to mine, but it seems a bit random and I think I would spend a lot of time sending emails for small results. <comment><USER2> You are asking the wrong question: How do they Find you?! Make your projects easy to join and contribute. And do Some marketing. Check e.g. the great book The Art of Community, cc licensed so you can check the pdf on [https://www.jonobacon.com/books/artofcommunity/](https://www.jonobacon.com/books/artofcommunity/)

</comment><comment><USER3> It's really a tough problem. What I do when I need a programmer-buddy for a side project — just pitching people, whom I personally know. Of course I need to target exactly those guys who have corresponding skills. But I don't see any other approach to find reliable partners.</comment>


Oragono – Go IRC server implementing bleeding-edge IRCv3 - <AUTHOR> https://oragono.io/ <comment><AUTHOR> To forestall the inevitable "Why mention Go?" questions: when I see something marked as written in Go, I give it an extra look because I work with Go every day--I know I'll have a much easier time hacking on it / submitting PRs if needed.</comment>