Pool Multiprocessing Python
Question: Basically the issue is as follows: I have a bunch of workers that have a
function prescribed to each (the function is worker(alist) ) and am trying to
process 35 workers at the same time. Each worker reads their line from the
file (the modulo part) and should process the line using the "worker"
function. I've pen-tested and found that the raw manipulation and deletion of
the useless indices is working 100% as intended.

The args part of the "pool.apply_async" function isn't passing the list "raw"
into it and starting the process. Raw is completely correct and functions
normally, worker by itself functions normally, the pool.apply_async function
is the only place that there seems to be an issue and I have no idea how to
fix it. Any help please?

The relevant code is here:

    
    
    NUM_WORKERS=35
    f=open("test.csv")
    pool=multiprocessing.Pool()
    open("final.csv",'w')
    for workernumber in range(1, NUM_WORKERS):
        for i,line in enumerate(f):
            if i==0:
                print "Skipping first line" #dont do anything
            elif i%workernumber==0:
                raw = line.split(',')[0][1:-1].split()
                uselessindices=[-2,-3,-4,-5,-6]
                counter=0
                for ui in uselessindices:
                    del raw[ui+counter]
                    counter+=1
                print raw
                pool.apply_async(worker, args=(raw,))
    pool.close()
    pool.join()
    


Answer: 
    import multiprocessing
    
    def worker(arg):
        print 'doing work "%s"' % arg
        return
    
    NUM_WORKERS=35
    with open('test.csv', 'w') as test:
        for i in xrange(100):
            if i % 10 == 0:
                test.write('\n')
            test.write('"%s 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23",' % i)
    
    f=open("test.csv")
    pool=multiprocessing.Pool(processes=NUM_WORKERS)
    open("final.csv",'w')
    
    for i, line in enumerate(f):
        if i == 0:
            continue
        raw = line.split(',')[0][1:-1].split()
        uselessindices=[-2,-3,-4,-5,-6]
        counter=0
        for ui in uselessindices:
            del raw[ui+counter]
            counter+=1
        pool.apply_async(worker, args=(raw,))
    pool.close()
    pool.join()
    print 'last raw len: %s' % len(raw)
    print 'last raw value: %s' % raw
    

Output:

    
    
    doing work "['0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '23']"
    doing work "['10', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '23']"
    doing work "['20', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '23']"
    doing work "['30', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '23']"
    doing work "['40', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '23']"
    doing work "['50', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '23']"
    doing work "['60', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '23']"
    doing work "['70', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '23']"
    doing work "['80', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '23']"
    doing work "['90', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '23']"
    last raw len: 19
    last raw value: ['90', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '23']
    

