Python multiprocessing: where should join() be called, if process' children have grandchildren?
Question: Here is a toy problem I am working with:

    
    
    import multiprocessing as mp
    
    def task2():
        print "I am doing something important."
    
    def task1():
        listOfProcesses = []
        for i in range(5):
            process = mp.Process(target=task2)
            process.start()
            listOfProcesses.append(process)
    
    def task0():
        listOfProcesses = []
        for i in range(5):
            process = mp.Process(target=task1)
            process.start()
            listOfProcesses.append(process)
    
    if __name__ == '__main__':
        task0()
    

Now, I am having trouble understanding where join should be called in such a
scenario.

If I change `task0`'s definition like so:

    
    
    def task0():
        listOfProcesses = []
    
        for i in range(5):
            process = mp.Process(target=task2)
            process.start()
            listOfProcesses.append(process)
    
        for process in listOfProcesses:
            process.join()
    

Everything then seems to work correctly, but I don't understand what I am
really doing here. `task1` only starts its children, it does not join them. So
what does joining `task0` mean for `task1`?


Answer: `join` is fairly simple in concept - `x.join` says "the current thread (i.e.
process) of execution cannot proceed past _this_ point until `x` terminates."

So, in general you don't want your **main** thread to proceed past some point
until all your workers are done doing their work. Since you execute `task0` in
your main thread, doing a `join` there prevents your main thread from
proceeding past that point until all your workers (both `task1` _and_ `task2`)
are done.

# But wait, I didn't `join` in `task1`!

That's right. But `task1`'s process still won't terminate until all its
`task2`s are finished. This has to do with the POSIX concept of [_process
groups_](http://en.wikipedia.org/wiki/Process_group) \- a parent process will
not terminate until all its child processes are terminated. So, let's look at
the output of this reduced example:

    
    
    import multiprocessing as mp
    from time import sleep
    
    def task2():
        sleep(1)
        print "I am doing something important."
    
    def task1():
        for i in range(2):
            process = mp.Process(target=task2)
            process.start()
    
        print 'task1 done'
    
    def task0():
        process = mp.Process(target=task1)
        process.start()
        process.join()
    
    if __name__ == '__main__':
        task0()
        print 'all done'
    

output:

    
    
    task1 done
    I am doing something important.
    I am doing something important.
    all done
    

So as you can see, `task1` reached its end but did not terminate until its
child processes did - which meant that our `join` block in `task0` correctly
blocked our main thread from terminating until all the workers did.

For fun, here is the output of `ps jf` when running your original script with
no `join`s with the _only modification_ being `time.sleep` thrown into `task2`
so I could capture it running:

    
    
     PPID   PID  PGID   SID TTY      TPGID STAT   UID   TIME COMMAND
     6780  7385  7385  7385 pts/11    7677 Ss    1000   0:00 bash
     7385  7677  7677  7385 pts/11    7677 R+    1000   0:00  \_ ps jf
     6780  6866  6866  6866 pts/7     7646 Ss    1000   0:00 bash
     6866  7646  7646  6866 pts/7     7646 S+    1000   0:00  \_ python test
     7646  7647  7646  6866 pts/7     7646 S+    1000   0:00      \_ python test
     7647  7672  7646  6866 pts/7     7646 S+    1000   0:00      |   \_ python test
     7647  7673  7646  6866 pts/7     7646 S+    1000   0:00      |   \_ python test
     7647  7674  7646  6866 pts/7     7646 S+    1000   0:00      |   \_ python test
     7647  7675  7646  6866 pts/7     7646 S+    1000   0:00      |   \_ python test
     7647  7676  7646  6866 pts/7     7646 S+    1000   0:00      |   \_ python test
     7646  7648  7646  6866 pts/7     7646 S+    1000   0:00      \_ python test
     7648  7665  7646  6866 pts/7     7646 S+    1000   0:00      |   \_ python test
     7648  7666  7646  6866 pts/7     7646 S+    1000   0:00      |   \_ python test
     7648  7667  7646  6866 pts/7     7646 S+    1000   0:00      |   \_ python test
     7648  7668  7646  6866 pts/7     7646 S+    1000   0:00      |   \_ python test
     7648  7669  7646  6866 pts/7     7646 S+    1000   0:00      |   \_ python test
     7646  7649  7646  6866 pts/7     7646 S+    1000   0:00      \_ python test
     7649  7656  7646  6866 pts/7     7646 S+    1000   0:00      |   \_ python test
     7649  7657  7646  6866 pts/7     7646 S+    1000   0:00      |   \_ python test
     7649  7658  7646  6866 pts/7     7646 S+    1000   0:00      |   \_ python test
     7649  7659  7646  6866 pts/7     7646 S+    1000   0:00      |   \_ python test
     7649  7660  7646  6866 pts/7     7646 S+    1000   0:00      |   \_ python test
     7646  7650  7646  6866 pts/7     7646 S+    1000   0:00      \_ python test
     7650  7652  7646  6866 pts/7     7646 S+    1000   0:00      |   \_ python test
     7650  7653  7646  6866 pts/7     7646 S+    1000   0:00      |   \_ python test
     7650  7654  7646  6866 pts/7     7646 S+    1000   0:00      |   \_ python test
     7650  7655  7646  6866 pts/7     7646 S+    1000   0:00      |   \_ python test
     7650  7670  7646  6866 pts/7     7646 S+    1000   0:00      |   \_ python test
     7646  7651  7646  6866 pts/7     7646 S+    1000   0:00      \_ python test
     7651  7661  7646  6866 pts/7     7646 S+    1000   0:00          \_ python test
     7651  7662  7646  6866 pts/7     7646 S+    1000   0:00          \_ python test
     7651  7663  7646  6866 pts/7     7646 S+    1000   0:00          \_ python test
     7651  7664  7646  6866 pts/7     7646 S+    1000   0:00          \_ python test
     7651  7671  7646  6866 pts/7     7646 S+    1000   0:00          \_ python test
    

You can see that our main process (the one that did `task0`) and the "first
children" (the ones that did `task1`) are still alive, even though they
clearly were out of python code to execute. They are also all members of the
same process group (`TPGID`).

# Sum it up, man

All of that is a long-winded way to say: `join` in your main thread is usually
all you need, since you have the guarantee that any child processes will wait
for _their_ children to terminate before themselves terminating.

