Python multiprocessing map_async ready() and _number_left not working as expected
Question: This is a little complicated, and I am sure this is a newbie error, but I
cannot for the life for me figure out where to even look.

Trying to do a multiprocessing map_async to process a large number of files.
Essentially, the code gets a list of files, looks in each file for a match on
a MAC address and writes out a line to another file if it matches.

nodb is my library....I have not included everything here yet (it's kind of
convoluted). I am hoping that someone can point me where to even look for
debugging this.

Here is the problem: the code works perfectly on anything under 60,000 files.
However, when I point it at a directory with 595,200 files, the little "while
true" loop that checks to see if it is done (using _number_left) stops
working....processing appears to continue but the _number_left does not
decrease and the ready() function returns TRUE...which it isn't.

And it stops after processing 62111 or 62112 files every single time I run it.
I added the little "dump" function thinking my queue was filling up.

Not sure what else to tell you...am I missing something? (probably) Please let
me know what more I can tell you to figure this out. I really have no idea
what is relevant....

Code is:

    
    
    import nodb_v09d as nodb
    
    import netaddr
    import sys
    import collections
    from multiprocessing import Pool, Queue
    import itertools
    import time
    
    # Handle CLI arguments
    #
    nmultip = 1
    args = sys.argv[1:]
    nmultip = nodb.parseArgs(args)
    
    # this function just gets the file list in the directory
    todoPif = nodb.getFileList('/data/david/data/2012/05')
    
    filterfields = { 10:set([int(netaddr.EUI('00-00-0a-0e-c9-be')),\
                             int(netaddr.EUI('00:15:ce:de:78:f3')),\
                             int(netaddr.EUI('3c-75-4a-ea-15-01')),\
                             int(netaddr.EUI('00-24-d1-1e-e9-be'))])
                     }
    ff=collections.OrderedDict(sorted(filterfields.items()))
    
    resultfields = [28,29,30,33]
    rf=resultfields.sort
    
    cutoff = 40
    
    todocnt = len(todoPif)
    
    outQ = Queue()
    hdrQ = Queue()
    
    # output file
    gpif = '/media/sf_samplePifs/test2.gpif'
    
    append = 0
    
    if __name__ == '__main__':
    
        # this is a little trick I picked up off the internet for passing multiple queues.  works ok
        todopool = Pool(None,nodb.poolQueueInit,[outQ,hdrQ])
    
        # itertools used to create an arg that contains a constant (ff) for all calls)
        r = todopool.map_async(nodb.deRefCall,itertools.izip(todoPif, itertools.repeat(ff)),1)
    
        while (True):
            nodb.logging.info('number left: ' + str(r._number_left) + '\nready? ' + str(r.ready()))
            nodb.logging.info('queue size: ' + str(outQ.qsize()))
    
            if (r._number_left == 0): break
            if (outQ.qsize() >= cutoff):
                nodb.dumpQueueToGpif(gpif, hdrQ, outQ, append, cutoff)
                if (append == 0): 
                    append = 1
    
            sys.stderr.write('\rPIF Files DONE: ' + str(todocnt-r._number_left) + '/' + str(todocnt))
            print '\n'
            time.sleep(0.2)
        r.wait()  
    
    sys.stderr.write('\rPIF Files DONE: ' + str(todocnt) + '/' + str(todocnt) + '\n')
    
    # dump remainder to file
    nodb.dumpQueueToGpif(gpif, hdrQ, outQ, append,outQ.qsize())
    

**MAJOR ADDITION:**

At the request of another user, I simplified the code. No queues, no external
private libraries, etc:

    
    
    import sys
    import os
    import time
    from multiprocessing import Pool
    
    def doPifFile(pifFile):
    
        #readPif = call(['ls','-l',' > /tmp/out'])
        cmd = 'ipdr_dump ' + pifFile + ' | grep "," | wc -l > /tmp/dump'
        readPif = os.system(cmd)
        return readPif
    
    def getFileList(directory):
        flist = list()
        for root, dirs, files in os.walk(directory):
            for piffile in files:
                if piffile.endswith('.pif'):
                    flist.append(os.path.abspath(os.path.join(root,piffile)))
        return flist
    
    todoPif = getFileList('/data/david/data/2012/05')
    
    todocnt = len(todoPif)
    print '# of files to process: ' + str(todocnt)
    
    if __name__ == '__main__':
    
        todopool = Pool()
        r = todopool.map_async(doPifFile,todoPif,1)
    
        while (True):
            print 'number left: ' + str(r._number_left) + '\nready? ' + str(r.ready())
    
            #if (r.ready()): break
            if (r._number_left == 0): break
            sys.stderr.write('\rPIF Files DONE: ' + str(todocnt-r._number_left) + '/' + str(todocnt))
            print '\n'
            time.sleep(0.2)
    
    sys.stderr.write('\rPIF Files DONE: ' + str(todocnt) + '/' + str(todocnt) + '\n')
    

When I ran it, I got something VERY interesting that did NOT show up on the
runs with the more complex code but that happened at exactly the same spot,
although it claims it happened in my dump program:

    
    
    number left: 533100
    ready? False
    PIF Files DONE: 62100/595200
    
    number left: 533090
    ready? False
    PIF Files DONE: 62110/595200
    
    *** glibc detected *** ipdr_dump: corrupted double-linked list: 0x0000000001a58370 ***
    ======= Backtrace: =========
    /lib64/libc.so.6[0x36f5c76126]
    /lib64/libc.so.6[0x36f5c78eb4]
    /lib64/libc.so.6(fclose+0x14d)[0x36f5c6678d]
    /lib64/libz.so.1[0x36f6803021]
    ipdr_dump[0x405c0b]
    ipdr_dump[0x40546e]
    ipdr_dump[0x401c2a]
    /lib64/libc.so.6(__libc_start_main+0xfd)[0x36f5c1ecdd]
    ipdr_dump[0x4016b9]
    ======= Memory map: ========
    00400000-0040e000 r-xp 00000000 08:02 2364135                            /home/david/ipdr_dump
    0060d000-0060e000 rw-p 0000d000 08:02 2364135                            /home/david/ipdr_dump
    01a54000-01a75000 rw-p 00000000 00:00 0                                  [heap]
    32e0600000-32e0604000 r-xp 00000000 08:02 3932181                        /lib64/libuuid.so.1.3.0
    32e0604000-32e0803000 ---p 00004000 08:02 3932181                        /lib64/libuuid.so.1.3.0
    32e0803000-32e0804000 rw-p 00003000 08:02 3932181                        /lib64/libuuid.so.1.3.0
    36f5800000-36f5820000 r-xp 00000000 08:02 3932309                        /lib64/ld-2.12.so
    36f5a1f000-36f5a20000 r--p 0001f000 08:02 3932309                        /lib64/ld-2.12.so
    36f5a20000-36f5a21000 rw-p 00020000 08:02 3932309                        /lib64/ld-2.12.so
    36f5a21000-36f5a22000 rw-p 00000000 00:00 0 
    36f5c00000-36f5d8a000 r-xp 00000000 08:02 3932315                        /lib64/libc-2.12.so
    36f5d8a000-36f5f89000 ---p 0018a000 08:02 3932315                        /lib64/libc-2.12.so
    36f5f89000-36f5f8d000 r--p 00189000 08:02 3932315                        /lib64/libc-2.12.so
    36f5f8d000-36f5f8e000 rw-p 0018d000 08:02 3932315                        /lib64/libc-2.12.so
    36f5f8e000-36f5f93000 rw-p 00000000 00:00 0 
    36f6000000-36f6002000 r-xp 00000000 08:02 3932566                        /lib64/libdl-2.12.so
    36f6002000-36f6202000 ---p 00002000 08:02 3932566                        /lib64/libdl-2.12.so
    36f6202000-36f6203000 r--p 00002000 08:02 3932566                        /lib64/libdl-2.12.so
    36f6203000-36f6204000 rw-p 00003000 08:02 3932566                        /lib64/libdl-2.12.so
    36f6400000-36f6417000 r-xp 00000000 08:02 3932564                        /lib64/libpthread-2.12.so
    36f6417000-36f6617000 ---p 00017000 08:02 3932564                        /lib64/libpthread-2.12.so
    36f6617000-36f6618000 r--p 00017000 08:02 3932564                        /lib64/libpthread-2.12.so
    36f6618000-36f6619000 rw-p 00018000 08:02 3932564                        /lib64/libpthread-2.12.so
    36f6619000-36f661d000 rw-p 00000000 00:00 0 
    36f6800000-36f6815000 r-xp 00000000 08:02 3932563                        /lib64/libz.so.1.2.3
    36f6815000-36f6a14000 ---p 00015000 08:02 3932563                        /lib64/libz.so.1.2.3
    36f6a14000-36f6a15000 r--p 00014000 08:02 3932563                        /lib64/libz.so.1.2.3
    36f6a15000-36f6a16000 rw-p 00015000 08:02 3932563                        /lib64/libz.so.1.2.3
    36f6c00000-36f6c83000 r-xp 00000000 08:02 3932493                        /lib64/libm-2.12.so
    36f6c83000-36f6e82000 ---p 00083000 08:02 3932493                        /lib64/libm-2.12.so
    36f6e82000-36f6e83000 r--p 00082000 08:02 3932493                        /lib64/libm-2.12.so
    36f6e83000-36f6e84000 rw-p 00083000 08:02 3932493                        /lib64/libm-2.12.so
    36f7000000-36f7007000 r-xp 00000000 08:02 3935994                        /lib64/librt-2.12.so
    36f7007000-36f7206000 ---p 00007000 08:02 3935994                        /lib64/librt-2.12.so
    36f7206000-36f7207000 r--p 00006000 08:02 3935994                        /lib64/librt-2.12.so
    36f7207000-36f7208000 rw-p 00007000 08:02 3935994                        /lib64/librt-2.12.so
    36f7800000-36f781d000 r-xp 00000000 08:02 3932588                        /lib64/libselinux.so.1
    36f781d000-36f7a1c000 ---p 0001d000 08:02 3932588                        /lib64/libselinux.so.1
    36f7a1c000-36f7a1d000 r--p 0001c000 08:02 3932588                        /lib64/libselinux.so.1
    36f7a1d000-36f7a1e000 rw-p 0001d000 08:02 3932588                        /lib64/libselinux.so.1
    36f7a1e000-36f7a1f000 rw-p 00000000 00:00 0 
    36f7c00000-36f7c16000 r-xp 00000000 08:02 3932572                        /lib64/libresolv-2.12.so
    36f7c16000-36f7e16000 ---p 00016000 08:02 3932572                        /lib64/libresolv-2.12.so
    36f7e16000-36f7e17000 r--p 00016000 08:02 3932572                        /lib64/libresolv-2.12.so
    36f7e17000-36f7e18000 rw-p 00017000 08:02 3932572                        /lib64/libresolv-2.12.so
    36f7e18000-36f7e1a000 rw-p 00000000 00:00 0 
    36f8000000-36f800e000 r-xp 00000000 08:02 3935998                        /lib64/liblber-2.4.so.2.5.6
    36f800e000-36f820d000 ---p 0000e000 08:02 3935998                        /lib64/liblber-2.4.so.2.5.6
    36f820d000-36f820e000 r--p 0000d000 08:02 3935998                        /lib64/liblber-2.4.so.2.5.6
    36f820e000-36f820f000 rw-p 0000e000 08:02 3935998                        /lib64/liblber-2.4.so.2.5.6
    36f8800000-36f8849000 r-xp 00000000 08:02 3932243                        /lib64/libldap-2.4.so.2.5.6
    36f8849000-36f8a49000 ---p 00049000 08:02 3932243                        /lib64/libldap-2.4.so.2.5.6
    36f8a49000-36f8a4b000 r--p 00049000 08:02 3932243                        /lib64/libldap-2.4.so.2.5.6
    36f8a4b000-36f8a4d000 rw-p 0004b000 08:02 3932243                        /lib64/libldap-2.4.so.2.5.6
    36f8c00000-36f8c16000 r-xp 00000000 08:02 3936000                        /lib64/libgcc_s-4.4.7-20120601.so.1
    36f8c16000-36f8e15000 ---p 00016000 08:02 3936000                        /lib64/libgcc_s-4.4.7-20120601.so.1
    36f8e15000-36f8e16000 rw-p 00015000 08:02 3936000                        /lib64/libgcc_s-4.4.7-20120601.so.1
    36f9400000-36f9535000 r-xp 00000000 08:02 4206136                        /usr/lib64/libnss3.so
    36f9535000-36f9734000 ---p 00135000 08:02 4206136                        /usr/lib64/libnss3.so
    36f9734000-36f9739000 r--p 00134000 08:02 4206136                        /usr/lib64/libnss3.so
    36f9739000-36f973b000 rw-p 00139000 08:02 4206136                        /usr/lib64/libnss3.so
    36f973b000-36f973d000 rw-p 00000000 00:00 0 
    36f9800000-36f9825000 r-xp 00000000 08:02 4206135                        /usr/lib64/libnssutil3.so
    36f9825000-36f9a24000 ---p 00025000 08:02 4206135                        /usr/lib64/libnssutil3.so
    36f9a24000-36f9a2a000 r--p 00024000 08:02 4206135                        /usr/lib64/libnssutil3.sonumber left: 533078
    ready? False
    PIF Files DONE: 62122/595200
    
    number left: 533068
    ready? False
    PIF Files DONE: 62132/595200
    
    number left: 533056
    ready? False
    PIF Files DONE: 62144/595200
    

What is odd is that it continued on, whereas the previous runs caused a
failure of the _number_left and misfire on "ready()" (although the processes
still ran in the background).

I have run the dump program manually on the 16 processor box I have, and they
run in parallel fine, never seen that glibc error before. I have to assume
that it is associated with the python setup....I just don't know where.

This may be too complex for a forum diagnosis. Any further ideas on where I
might look or how I might be able to seem what happened are welcome.

One more tidbit...printed out pool._success. It changes to FALSE at the magic
moment with the _number_left stops moving.

    
    
    INFO  number left: 533167  
    ready? False  
    successful? True  
    INFO  queue size: 424  
    PIF Files DONE: 62033/595200  
    
    INFO  number left: 533117
    ready? False
    successful? True
    INFO  queue size: 424
    PIF Files DONE: 62083/595200
    
    INFO  number left: 533087
    ready? True
    successful? False
    INFO  queue size: 424
    PIF Files DONE: 62113/595200
    
    INFO  number left: 533087
    ready? True
    successful? False
    INFO  queue size: 424
    PIF Files DONE: 62113/595200
    


Answer: You could try sys.stdout.flush().

