Trying to use timeit.timeit
Question: I'd like to measure the time of running for two codes, I tried looking up the
python documentation for timeit, but I didn't really understand. Could someone
explain in a more beginner-level vocabulary?


Answer: _Note:_ Copied to [How to use timeit
module](http://stackoverflow.com/questions/8220801/how-to-use-timeit-module/).

I'll let you in on a secret: the best way to use `timeit` is on the command
line.

On the command line, `timeit` does proper statistical analysis: it tells you
how long the shortest run took. This is good because _all_ error in timing is
positive. So the shortest time has the least error in it. There's no way to
get negative error because a computer can't ever compute faster than it can
compute!

So, the command-line interface:

    
    
    %~> python -m timeit "1 + 2"
    10000000 loops, best of 3: 0.0468 usec per loop
    

That's quite simple, eh?

You can set stuff up:

    
    
    %~> python -m timeit -s "x = range(10000)" "sum(x)"
    1000 loops, best of 3: 543 usec per loop
    

which is useful, too!

If you want multiple lines, you can either use the shell's automatic
continuation or use separate arguments:

    
    
    %~> python -m timeit -s "x = range(10000)" -s "y = range(100)" "sum(x)" "min(y)"
    1000 loops, best of 3: 554 usec per loop
    

That gives a setup of

    
    
    x = range(1000)
    y = range(100)
    

and times

    
    
    sum(x)
    min(y)
    

* * *

If you want to have longer scripts you might be tempted to move to `timeit`
inside a Python script. I suggest avoiding that because the analysis and
timing is simply better on the command line. Instead, I tend to make shell
scripts:

    
    
     SETUP="
    
     ... # lots of stuff
    
     "
    
     echo Minmod arr1
     python -m timeit -s "$SETUP" "Minmod(arr1)"
    
     echo pure_minmod arr1
     python -m timeit -s "$SETUP" "pure_minmod(arr1)"
    
     echo better_minmod arr1
     python -m timeit -s "$SETUP" "better_minmod(arr1)"
    
     ... etc
    

This can take a bit longer due to the multiple initialisations, but normally
that's not a big deal.

* * *

But what if you _want_ to use `timeit` inside your module?

Well, the simple way is to do:

    
    
    def function(...):
        ...
    
    timeit.Timer(function).timeit(number=NUMBER)
    

and that gives you cumulative (_not_ minimum!) time to run that number of
times.

To get a good analysis, use `.repeat` and take this min:

    
    
    min(timeit.Timer(function).repeat(repeat=REPEATS, number=NUMBER))
    

You should normally combine this with `functools.partial` instead of `lambda:
...` to lower overhead. Thus you could have something like:

    
    
    from functools import partial
    
    def to_time(items):
        ...
    
    test_items = [1, 2, 3] * 100
    times = timeit.Timer(partial(to_time, test_items)).repeat(3, 1000)
    
    # Divide by the number of repeats
    time_taken = min(times) / 1000
    

* * *

You can also do:

    
    
    timeit.timeit("...", setup="from __main__ import ...", number=NUMBER)
    

which would give you something closer to the _interface_ from the command-
line, but in a much less cool manner. The `"from __main__ import ..."` lets
you use code from your main module inside the artificial environment created
by `timeit`.

It's worth noting that this is a convenience wrapper for
`Timer(...).timeit(...)` and so isn't particularly good at timing. I
personally far prefer using `Timer` as I've shown above.

* * *

### Warnings

There are a few caveats with `timeit` that hold everywhere.

  * Overhead is not accounted for. Say you want to time `x += 1`, to find out how long addition takes:
    
        >>> python -m timeit -s "x = 0" "x += 1"
    10000000 loops, best of 3: 0.0476 usec per loop
    

Well, it's _not_ 0.0476 µs. You only know that it's _less_ than that. All
error is positive.

So try and find _pure_ overhead:

    
        >>> python -m timeit -s "x = 0" ""      
    100000000 loops, best of 3: 0.014 usec per loop
    

That's a good **30%** overhead just from timing! This can massively skew
relative timings. But you only really cared about the _adding_ timings; the
look-up timings for `x` also need to be included in overhead:

    
        >>> python -m timeit -s "x = 0" "x"
    100000000 loops, best of 3: 0.0166 usec per loop
    

The difference isn't much larger, but it's there.

  * Mutating methods are dangerous.
    
        python -m timeit -s "x = [0]*100000" "while x: x.pop()"
    10000000 loops, best of 3: 0.0436 usec per loop
    

But that's _completely wrong!_ `x` is the empty list after the first
iteration. You'll need to reinitialize:

    
        >>> python -m timeit "x = [0]*100000" "while x: x.pop()"
    100 loops, best of 3: 9.79 msec per loop
    

But then you have lots of overhead. Account for that separately.

    
        >>> python -m timeit "x = [0]*100000"                   
    1000 loops, best of 3: 261 usec per loop
    

Note that subtracting the overhead is reasonable here _only because_ the
overhead is much a small-ish fraction of the time.

