Paper result validation and the scientific method in earth science The context of this question is computational earth science specifically meteorology/climatology but I am sure it applies to all the scientific disciplines where numerical methods are used.

Recently I had asked the author of a peer reviewed paper for the software that was used in the publication of his paper. He not only  immediately provided the software but also gave me test data to verify the results of his paper. The same was done by a climate change institute affiliated to NOAA. They gave me their software and asked me to download the relevant data from the NOAA site to validate their conclusions and perhaps reuse their software for solving my problems. In both these cases the only requirement was that the software had to be cited in any paper that I publish in the future.

These two instances are not the problem. In another couple of cases when I asked the authors of the peer reviewed paper for their software to validate their work they refused claiming that is "private". My questions are therefore:


Which one of these two  occurrences is more common in meteorology/climatology?
As somebody who is originally from a CS background (where code reviews are extremely common) I'm curious what is the current standard of reproducibility in meteorology/climatology?
Let's start with the trivially true: programming is not earth science; earth science is not programming. Programming is one tool of many in earth science. In programming, the end result is the program itself. In earth science, that's never the case: but one end result is algorithms developed that inform us about the real world, and implemented in a progam.

What if there is a bug in a peer-reviewed paper? Well, of course there is, just as there are bugs in pretty much every program that does anything useful in the real world. So what?

Validating source code is  the domain of programming. Validating algorithms and observations is the domain of earth science. Re-running someone else's code with the same input data is barely any kind of validation at all, and rarely the interesting kind. Taking someone else's algorithms and implementing them yourself, that's the sort of validation I see across the physical, life, social sciences.

So I don't see someone else's source code? No big deal. Indeed, that's often a good thing: it means I can genuinely clean-room their algorithm.