Use cases of parallel programming using multiprocessing module in python
Question: Im novice even in python and Im trying to write fast code with the
multiprocessing module of python. Actually my question is very general: I'd
like to know different ways of using multiprocessing and Im very confused
because Im not sure how exactly this code works in order to do correct
generalizations

    
    
    import numpy as np
    from multiprocessing import Process, Pool
    
    def sqd(x):
      return x*x.T
    
    A = np.random.random((10000, 10000))
    
    if __name__ == '__main__':
       pool = Pool(processes = 4)
       result = pool.apply_async(sqd, [A])
       print result.get(timeout = 1)
       print len(pool.map(sqd, A))
    

However, when I performed the following generalization in order to accelerate
the random matrix generation, things are not so good

    
    
    import numpy as np
    from multiprocessing import Pool
    
    def sqd(d):
      x = np.random.random((d, d))
      return x*x.T
    
    D=100
    
    if __name__ == '__main__':
       pool = Pool(processes = 4)
       result = pool.apply_async(sqd, [D])
       print result.get(timeout = 1)
       print pool.map(sqd, D)
    

So the output is:

    
    
    $ python prueba2.py
    [[ 0.50770071  0.36508745  0.02447127 ...,  0.12122494  0.72641019
    0.68209404]
    [ 0.19470934  0.89260293  0.58143287 ...,  0.25042778  0.05046485
    0.50856362]
    [ 0.67367326  0.76929582  0.4232229  ...,  0.72910757  0.56047056
    0.11873254]
    ..., 
    [ 0.91234565  0.20216969  0.2961842  ...,  0.57539533  0.99836323
    0.79875158]
    [ 0.85407066  0.99905665  0.12948157 ...,  0.58411818  0.06688349
    0.71026483]
    [ 0.0599241   0.82759421  0.9532148  ...,  0.22463593  0.0859876
    0.41072156]]
    Traceback (most recent call last):
    File "prueba2.py", line 14, in <module>
    print pool.map(sqd, D)
    File "/home/nacho/anaconda/lib/python2.7/multiprocessing/pool.py", line 251, in map
    return self.map_async(func, iterable, chunksize).get()
    File "/home/nacho/anaconda/lib/python2.7/multiprocessing/pool.py", line 304, in   map_async
    iterable = list(iterable)
    TypeError: 'int' object is not iterable
    

In this case, I know that Im passing incorrect arguments to "something" but Im
not sure what is the reason for that, what I can and what I can't do for this
specific cases and others different to pass lists or ranges to the
multiprocessing module, also I'd like to know how to free memory after this
given that I permitted once executing without memory error...

I'd like to add some details, regardless to I'd like to know different use
cases using multiprocessing, the motivation underlying this question is
because I took a picture of my processors just at working and there is an
isolated process working at single processor which I suppose is due to
random() so I'd like to parallelize the complete task

I hope not being so ambiguous. Thank you in advance...


Answer: You can not define the argument of the function that you want to compute with
`multiprocessing` inside the function. What `pool.map` does, chop your `A`
array and map it between different processors and the job is contributed
between the number of processors you have submitted until it finishes. However
in your code, you just gave the dimension of your input array as an argument
to `pool.map`, therefore it just computed it for once and raises error, while
`map` needs your function plus an _iterable_ argument.

