Why is pool.map slower than normal map?
Question: I'm trying the following code:

    
    
    import multiprocessing
    import time
    import random
    
    def square(x):
        return x**2
    
    pool = multiprocessing.Pool(4)
    
    l = [random.random() for i in xrange(10**8)]
    
    now = time.time()
    pool.map(square, l)
    print time.time() - now
    
    now = time.time()
    map(square, l)
    print time.time() - now
    

and the `pool.map` version consistently runs several seconds more slowly than
the normal `map` version (19 seconds vs 14 seconds).

I've looked at the questions: [Why is multiprocessing.Pool.map slower than
builtin map?](http://stackoverflow.com/questions/9169538/why-is-
multiprocessing-pool-map-slower-than-builtin-map) and [multiprocessing.Pool()
slower than just using ordinary
functions](http://stackoverflow.com/questions/20727375/multiprocessing-pool-
slower-than-just-using-ordinary-functions) and they seem to chalk it up to to
either IPC overhead or disk saturation, but I feel like in my example those
aren't obviously the issue; I'm not writing/reading anything to/from disk, and
the computation is long enough that it seems like IPC overhead should be small
compared to the total time saved by the multiprocessing (I'm estimating that,
since I'm doing work on 4 cores instead of 1, I should cut the computation
time down from 14 seconds to about 3.5 seconds). I'm not saturating my cpu I
don't think; checking `cat /proc/cpuinfo` shows that I have 4 cores, but even
when I multiprocess to only 2 processes it's still slower than just the normal
map function (and even slower than 4 processes). What else could be slowing
down the multiprocessed version? Am I misunderstanding how IPC overhead
scales?

If it's relevant, this code is written in Python 2.7, and my OS is Linux Mint
17.2


Answer: `pool.map` splits a list into N jobs (where N is the size of the list) and
dispatches those to the processes.

The work a single process is doing is shown in your code:

    
    
    def square(x):
        return x**2
    

This operation takes very little time on modern CPUs, no matter how big the
number is.

In your example you're creating a huge list and performing an irrelevant
operation on every single element. Of course the IPC overhead will be greater
compared to the regular `map` function which is optimized for fast looping.

In order to see your example working as you expect, just add a
`time.sleep(0.1)` call to the square function. This simulates a long running
task. Of course you might want to reduce the size of the list or it will take
forever to complete.

