Pool.map - Why doesn't the worker process crash earlier?
Question: Say I do this:

    
    
    import multiprocessing as mp
    
    def f(x):
        raise OverflowError  # raised BEFORE the print
        print x
    
    if __name__ == '__main__':
        pool = mp.Pool(processes=1)
        for _ in pool.imap_unordered(f, range(10)):
            pass
        pool.close()
        pool.join()
    

Out:

    
    
    Traceback (most recent call last):
      File "test0.py", line 9, in <module>
        for _ in pool.imap_unordered(f, range(10)):
      File "/Users/usualme/anaconda/lib/python2.7/multiprocessing/pool.py", line 659, in next
        raise value
    OverflowError
    

Ok the output makes sense. The exception is raised before the `print`
statement, so there is no output. Now almost the same code, but I switched 2
lines:

    
    
    import multiprocessing as mp
    
    def f(x):
        print x
        raise OverflowError  # raised AFTER the print
    
    if __name__ == '__main__':
        pool = mp.Pool(processes=1)
        for _ in pool.imap_unordered(f, range(10)):
            pass
        pool.close()
        pool.join()
    

Out:

    
    
    0
    1
    2
    3
    4
    5
    6
    7
    8
    9
    Traceback (most recent call last):
      File "test0.py", line 9, in <module>
        for _ in pool.imap_unordered(f, range(10)):
      File "/Users/usualme/anaconda/lib/python2.7/multiprocessing/pool.py", line 659, in next
        raise value
    OverflowError
    

I don't understand the output. I was expecting either the number 0 followed by
the stack trace, or all 10 numbers and 10 stack traces. Why is it printing all
numbers and only one stack trace? Why does the worker process wait the very
end to crash?


Answer: It's just a timing thing - The `worker` process doesn't care that an exception
is raised in the function it's running, it just returns the exception to the
parent and keeps chugging along with the next task. Here's the loop it's
running (slightly simplified):

    
    
    while maxtasks is None or (maxtasks and completed < maxtasks):
        try:
            task = get()  # Get task from parent
        except (EOFError, OSError):
            util.debug('worker got EOFError or OSError -- exiting')
            break
    
        if task is None:
            util.debug('worker got sentinel -- exiting')
            break
    
        job, i, func, args, kwds = task
        try:
            result = (True, func(*args, **kwds))  # Call the function pass from the parent
        except Exception as e:  # We end up in here if the worker raises an exception
            if wrap_exception:
                e = ExceptionWithTraceback(e, e.__traceback__)
            result = (False, e)  # The exception object is stored as the result
    
        put((job, i, result)) # Send result to parent process
    

So, even though the very first task raises an exception, it takes a little bit
of time for the result to travel between the two processes, and for the parent
process to actually pull the result out of the `Queue` and raise the
`Exception`. In that window of time, the worker is able to execute all the
remaining tasks. If you make the worker function slower, you'll see it
executes fewer tasks:

    
    
    import multiprocessing as mp
    import time
    
    def f(x):
        print x
        time.sleep(2)
        raise OverflowError 
    
    if __name__ == '__main__':
        pool = mp.Pool(processes=1)
        for _ in pool.imap_unordered(f, range(10)):
            pass
        pool.close()
        pool.join()
    

Output:

    
    
    0
    1
    Traceback (most recent call last):
      File "p.py", line 11, in <module>
        for _ in pool.imap_unordered(f, range(10)):
      File "/usr/lib/python2.7/multiprocessing/pool.py", line 626, in next
        raise value
    OverflowError
    

You'd also only see some percentage of the results printed if you passed a
larger iterable, since the worker wouldn't have enough time to get through all
of them before the parent died.

You only see one exception actually get raised because from the parent's
perspective, as soon as one task has failed, the entire `imap` call should be
aborted. The parent pulls the results from all its child processes from a
single `Queue` sequentially, so as soon as it sees the first exception, the
`imap` call ends, so the results from the rest of the tasks get thrown away.

