Single process code performs faster than Multiprocessing - MCVE
Question: My attempt to speed up one of my applications using Multiprocessing resulted
in lower performance. I am sure it is a design flaw, but that is the point of
discussion- How to better approach this problem in order to take advantage of
multiprocessing.

My current results on a 1.4ghz atom:

  1. SP Version = 19 seconds
  2. MP Version = 24 seconds

Both versions of code can be copied and pasted for you to review. The dataset
is at the bottom and can be pasted also. (I decided against using xrange to
illustrate the problem)

First the SP version:

    
    
    *PASTE DATA HERE*    
    
    def calc():
        for i, valD1 in enumerate(D1):
            for i, valD2 in enumerate(D2):
                for i, valD3 in enumerate(D3):  
                    for i, valD4 in enumerate(D4):
                        for i, valD5 in enumerate(D5):
                            for i, valD6 in enumerate(D6):
                                for i, valD7 in enumerate(D7):
                                    sol1=float(valD1[1]+valD2[1]+valD3[1]+valD4[1]+valD5[1]+valD6[1]+valD7[1])
                                    sol2=float(valD1[2]+valD2[2]+valD3[2]+valD4[2]+valD5[2]+valD6[2]+valD7[2])
        return None
    
    print(calc())
    

Now the MP version:

    
    
    import multiprocessing
    import itertools
    
    *PASTE DATA HERE*
    
    def calculate(vals):
        sol1=float(valD1[0]+valD2[0]+valD3[0]+valD4[0]+valD5[0]+valD6[0]+valD7[0])
        sol2=float(valD1[1]+valD2[1]+valD3[1]+valD4[1]+valD5[1]+valD6[1]+valD7[1])
        return none
    
    def process():
        pool = multiprocessing.Pool(processes=4)
        prod = itertools.product(([x[1],x[2]] for x in D1), ([x[1],x[2]] for x in D2), ([x[1],x[2]] for x in D3), ([x[1],x[2]] for x in D4), ([x[1],x[2]] for x in D5), ([x[1],x[2]] for x in D6), ([x[1],x[2]] for x in D7))
        result = pool.imap(calculate, prod, chunksize=2500)
        pool.close()
        pool.join()
        return result
    
    if __name__ == "__main__":    
        print(process())
    

And the data for both:

    
    
    D1 = [['A',7,4],['B',3,7],['C',6,1],['D',12,6],['E',4,8],['F',8,7],['G',11,3],['AX',11,7],['AX',11,2],['AX',11,4],['AX',11,4]]
    D2 = [['A',7,4],['B',3,7],['C',6,1],['D',12,6],['E',4,8],['F',8,7],['G',11,3],['AX',11,7],['AX',11,2],['AX',11,4],['AX',11,4]]
    D3 = [['A',7,4],['B',3,7],['C',6,1],['D',12,6],['E',4,8],['F',8,7],['G',11,3],['AX',11,7],['AX',11,2],['AX',11,4],['AX',11,4]]
    D4 = [['A',7,4],['B',3,7],['C',6,1],['D',12,6],['E',4,8],['F',8,7],['G',11,3],['AX',11,7],['AX',11,2],['AX',11,4],['AX',11,4]]
    D5 = [['A',7,4],['B',3,7],['C',6,1],['D',12,6],['E',4,8],['F',8,7],['G',11,3],['AX',11,7],['AX',11,2],['AX',11,4],['AX',11,4]]
    D6 = [['A',7,4],['B',3,7],['C',6,1],['D',12,6],['E',4,8],['F',8,7],['G',11,3],['AX',11,7],['AX',11,2],['AX',11,4],['AX',11,4]]
    D7 = [['A',7,4],['B',3,7],['C',6,1],['D',12,6],['E',4,8],['F',8,7],['G',11,3],['AX',11,7],['AX',11,2],['AX',11,4],['AX',11,4]]
    

And now the theory:

Since there is little actual work (just summing 7 ints) there is too much CPU
bound data and Interprocess Communication creates too much overhead to make
Multiprocessing effective. This seems like a situation where I really need the
ability to multithread. So at this point I am looking for suggestions before I
try this on a different language because of the GIL.

********Debugging

    
    
    File "calc.py", line 309, in <module>
        smart_calc()
      File "calc.py", line 290, in smart_calc
        results = pool.map(func, chunk_list)
      File "/usr/local/lib/python2.7/multiprocessing/pool.py", line 250, in map
        return self.map_async(func, iterable, chunksize).get()
      File "/usr/local/lib/python2.7/multiprocessing/pool.py", line 554, in get
        raise self._value
    TypeError: sequence index must be integer, not 'slice'
    

In this case, totallen = 108 and CHUNKS is set to 2. When CHUNKS is reduced to
1, it works.


Answer: Ok, I think I've figured out actually get a speed boost from multiprocessing.
Since your actual source lists aren't very long, it's reasonable to pass them
in their entirety to the worker processes. So, if each worker process has
copies of the same source lists, then ideally we'd want all of them iterate
over different pieces of the lists in parallel, and just sum up that unique
slice. Because we know the size of the input lists, we can accurately
determine how long `itertools.product(D1, D2, ...)` will be, which means we
can also accurately determine how big each chunk should be to evenly
distribute the work. So, we can provide each worker with a specific range of
the `itertools.product` iterator that they should iterate over and sum:

    
    
    import math
    import itertools
    import multiprocessing
    import functools
    
    def smart_calc(valD1, valD2, valD3, valD4, valD5, valD6, valD7, slices):
        # Build an iterator over the entire data set
        prod = itertools.product(([x[1],x[2]] for x in valD1), 
                                 ([x[1],x[2]] for x in valD2), 
                                 ([x[1],x[2]] for x in valD3), 
                                 ([x[1],x[2]] for x in valD4), 
                                 ([x[1],x[2]] for x in valD5), 
                                 ([x[1],x[2]] for x in valD6), 
                                 ([x[1],x[2]] for x in valD7))
    
        # But only iterate over our unique slice
        for subD1, subD2, subD3, subD4, subD5, subD6, subD7 in itertools.islice(prod, slices[0], slices[1]):
            sol1=float(subD1[0]+subD2[0]+subD3[0]+subD4[0]+subD5[0]+subD6[0]+subD7[0])
            sol2=float(subD1[1]+subD2[1]+subD3[1]+subD4[1]+subD5[1]+subD6[1]+subD7[1])
        return None
    
    def smart_process():
        CHUNKS = multiprocessing.cpu_count()  # Number of pieces to break the list into.
        total_len = len(D1) ** 7  # The total length of itertools.product()
        # Figure out how big each chunk should be. Got this from 
        # multiprocessing.map()
        chunksize, extra = divmod(total_len, CHUNKS)
        if extra:
            chunksize += 1
    
        # Build a list that has the low index and high index for each
        # slice of the list. Each process will iterate over a unique
        # slice
        low = 0 
        high = chunksize
        chunk_list = []
        for _ in range(CHUNKS):
            chunk_list.append((low, high))
            low += chunksize
            high += chunksize
    
        pool = multiprocessing.Pool(processes=CHUNKS)
        # Use partial so we can pass all the lists to each worker
        # while using map (which only allows one arg to be passed)
        func = functools.partial(smart_calc, D1, D2, D3, D4, D5, D6, D7) 
        result = pool.map(func, chunk_list)
        pool.close()
        pool.join()
        return result
    

Results:

    
    
    sequential: 13.9547419548
    mp: 4.0270690918
    

Success! Now, you do have to actually combine the results after you have them,
which will add additional overhead to your real program. It might end up
making this approach slower than sequential again, but it really depends on
what you actually want to do with the data.

