parallellization in python multiprocessing
Question: I need to do parallelization in Python for my existing code and I am blocking
in one for loop. I used `multiprocessing.process` but it's freezing the
computer.

    
    
    import multiprocessing
    
    def func(pos,r,h,grid):
        for i in arayb:
    
            l= L(r,g, p[i,:],h) #process need to be parallelized (L is function in another file)
            p = multiprocessing.Process(target=func)
            p.start()
            p.join()
            print('l',l)
    if __name__ == '__main__':
        lock = Lock()
    

When I am using `if __name__ == '__main__':` above like:

    
    
    import multiprocessing
    from source.RUN import* # imported main file 
    
    def func(pos,r,h,grid):
        for i in arayb:
    
         l= L(r,g, p[i,:],h) #process need to be parallelized (L is function in another file)
    if __name__ == '__main__':
        lock = Lock()
    
            p = multiprocessing.Process(target=func)
            p.start()
            p.join()
            print('l',l)
    

Then it will not go to `multiprocess.Process`.

So please give me appropriate way to do `l = L(r,g, p[i,:],h)` in parallel for
`for i in arrayb`.

I have 13 to 14 processes like `l = L(r,g, p[i,:],h)` that need to be
parallelized, so give me guidance about one, then I can make the others
parallel.


Answer: Your first example is freezing the computer because you're recursively calling
`func` in new processes. That's going to quickly use up all your CPU/memory.
Your second example is just creating one process that iterates over `arayb`
and calls `L` without any concurrency, so you're not getting any performance
boost.

What you want to do is execute `L` concurrently across multiple processes, but
without spawning so many processes that you bog down your system. You can use
[`multiprocessing.Pool`](https://docs.python.org/2.7/library/multiprocessing.html#multiprocessing.pool.multiprocessing.Pool)
for that:

    
    
    import multiprocessing
    # L must is imported or defined somewhere up here...
    
    if __name__ == '__main__':
        p = multiprocessing.Pool()  # Creates a pool with as many workers as you have CPU cores
        results = []
        for i in arayb:
            results.append(pool.apply_async(L, (r, g, p[i,:], h)))
        p.close()
        p.join()
        for result in results:
            print('l', result.get())
    

