performance issue : big numpy array and system call
Question: There is a performance issue when using a system call after pre-allocating big
amount of memory (e.g. numpy array). The issue grows with the amount of
memory.

test.py :

    
    
    import os
    import sys
    import time
    import numpy
    
    start = time.clock()
    test = int(sys.argv[1])
    a = numpy.zeros((test,500,500))
    for i in range(test) :
        os.system("echo true > /dev/null")
    elapsed = (time.clock() - start)
    print(elapsed)
    

The per-iteration time increases dramatically :

    
    
    edouard@thorin:~/now3/code$ python test.py 100
    0.64
    edouard@thorin:~/now3/code$ python test.py 200
    2.09
    edouard@thorin:~/now3/code$ python test.py 400
    14.26
    

This should not be related to virtual memory. Is it a known issue?


Answer: You seem to have narrowed the problem down to `os.system()` taking longer
after you've allocated a large NumPy array.

Under the covers, `system()` uses
[`fork()`](http://en.wikipedia.org/wiki/Fork_%28operating_system%29). Even
though `fork()` is supposed to be very cheap (due to its use of [copy-on-
write](http://en.wikipedia.org/wiki/Copy-on-write)), it turns out that things
are not quite as simple.

In particular, there are known issues with Linux's `fork()` taking longer for
larger processes. See, for example:

  * [Faster forking of large processes on Linux?](http://stackoverflow.com/questions/2731531/faster-forking-of-large-processes-on-linux)
  * [Linux Fork Performance Redux: Large Pages](http://kirkwylie.blogspot.co.uk/2008/11/linux-fork-performance-redux-large.html).

Both documents are fairly old, so I am not sure what the state of the art is.
However, the evidence suggests that you've encountered an issue of this sort.

If you can't get rid of those `system()` calls, I would suggest two avenues of
research:

  * Look into enabling huge pages.
  * Consider the possibility of spawning an auxiliary process on startup, whose job would be to invoke the necessary `system()` commands.

