python multicore queue randomly hangs for no reason, despite queue size being tiny
Question: in python here is my multiprocessing setup. I subclassed the Process method
and gave it a queue and some other fields for pickling/data purposes.

This strategy works about 95% of the time, the other 5% for an unknown reason
the queue just hangs and it never finishes (it's common that 3 of the 4 cores
finish their jobs and the last one takes forever so I have to just kill the
job).

I am aware that queue's have a fixed size in python, or they will hang. My
queue only stores one character strings... the id of the processor, so it
can't be that.

Here is the exact line where my code halts:

    
    
    res = self._recv() 
    

Does anyone have ideas? The formal code is below. Thank you.

    
    
    from multiprocessing import Process, Queue
    from multiprocessing import cpu_count as num_cores
    import codecs, cPickle
    
    class Processor(Process):
    
        def __init__(self, queue, elements, process_num):
            super(Processor, self).__init__()
            self.queue = queue
            self.elements = elements
            self.id = process_num
    
        def job(self):
    
            ddd = []
    
            for l in self.elements:
    
                obj = ... heavy computation ...
    
                dd = {}
                dd['data'] = obj.data
                dd['meta'] = obj.meta
            ddd.append(dd)
    
            cPickle.dump(ddd, codecs.open(
                urljoin(TOPDIR, self.id+'.txt'), 'w'))
    
            return self.id
    
        def run(self):
            self.queue.put(self.job())
    
    
    
     if __name__=='__main__':
    
            processes = []
    
            for i in range(0, num_cores()):
    
                q = Queue()
                p = Processor(q, divided_work(), process_num=str(i))
                processes.append((p, q))
                p.start()
    
            for val in processes:
    
                val[0].join()
                key = val[1].get() 
    
                storage = urljoin(TOPDIR, key+'.txt')
    
                ddd = cPickle.load(codecs.open(storage , 'r'))
    
                .. unpack ddd process data ...
    


Answer: Do a `time.sleep(0.001)` at the beginning of your `run()` method.

