I've been looking through the other answers, and I haven't found a good answer yet.  I'll describe the simulation I am running, and then the problems.  The program simulates particles undergoing brownian dynamic motion (thermal motion) alternating with stochastic processes (kinetic monte carlo).  When I move to OpenMP code, I gain performance (finally), but I lose the determinism in the system.  Therefore, I have two problems.I can only repeat the same 'experiment' twice using a single threaded version.  Otherwise, the order in which the threads operate on the particles destroys any determinism.Profiling how much faster the OpenMP code runs as a function of threads, or versus the serial version, requires running 10x sims at every point since I test for a particular ending configuration, and the speed of the program is very sensitive to the configuration of the particles.  Basically, the closer more particles are, the longer it takes to run, so different ending configurations can take drastically different amounts of time.I know that this is a normal problem found in MD, but I am seeing if there are any ways around it rather than running the single threaded version.  If that is the case, then so be it, I was just hoping for something else mostly so that I could optimize the OpenMP code and environment variables.
Removing non-determinism from molecular dynamics code
parallel computing;simulation;hpc;molecular dynamics;biophysics
Both the standard cluster and custom supercomputer (Anton) versions of molecular dynamics at D. E. Shaw Research are both deterministic and parallel invariant.  That is, a test run on a single core generates the same bits as a massively parallel run.  The techniques includeInteger summation: Although each force term is computed in floating point, the total force on each atom is summed in fixed point.  As a result, the sums are both commutative and associative, and therefore trivial to parallelize without losing determinism.  All state (positions and momenta) are maintained in fixed point as well.Any required random numbers are generated using counter mode techniques, as described in http://www.thesalmons.org/john/random123/papers/random123sc11.pdf.  A counter mode generator amounts to encrypting the integer sequence 0, 1, 2, $\ldots$ using a sufficiently strong cypher.  See the paper for details.