PyPy displaying inaccurate benchmark results?
Question: I was working on [Project Euler](http://projecteuler.net/problem=204) and
wondered if I could speed up my solution using PyPy. However, I found results
quite disappointing, as it took more time to compute.

    
    
    d:\projeuler>pypy problem204.py
    3462.08630405 mseconds
    
    d:\projeuler>python problem204.py
    1823.91602542 mseconds
    

Since mseconds output were calculated using python's `time` modules, so I ran
it again using builtin benchmark commands.

    
    
    d:\projeuler>pypy -mtimeit -s "import problem204" "problem204._main()"
    10 loops, best of 3: 465 msec per loop
    
    d:\projeuler>python -mtimeit -s "import problem204" "problem204._main()"
    10 loops, best of 3: 1.87 sec per loop
    

PyPy reports that it took about half second to finish running. However, I
tried running pypy problem204 several times and outputs were never even close
to benchmarked .5 seconds. unlike pypy, python's mtimeit results are
consistent with outputs. Is pypy giving me inaccurate benchmarks, or is there
some magic I don't understand?


Answer: Note that timeit

  1. runs the statement several times (10 in your case), and
  2. does that several times (3 by default) and gives the minimum of that, for reasons [outlined in the documentation](http://docs.python.org/3/library/timeit.html#timeit.Timer.repeat).

It depends on your code, but it's entirely possible that the JIT compiler is
to blame for this confusing result. The JIT warmup overhead is incurred every
time you launched a new pypy process, but only once during the timeit
benchmark (because that one runs `_main` several times in the same process).
Moreover, if some part of your code is run so often that it's not compiled
when `_main` runs once, but only when it runs, say, three times, subsequent
runs will also be faster, which further removes the best result from the first
one (i.e. the one for running `pypy problem204.py` once).

The `timeit` result is correct in that it (roughly) matches how fast the code
will be in the best case - warmed-up JIT compiler, rarely losing the CPU to
other programs, etc. Your problem is that you want to know something different
- the time including JIT warmup.

