Deadlock with logging multiprocess/multithread python script
Question: I am facing the problem with collecting logs from the following script. Once I
set up the `SLEEP_TIME` to too "small" value, the LoggingThread threads
somehow block the logging module. The script freeze on logging request in the
`action` function. If the `SLEEP_TIME` is about 0.1 the script collect all log
messages as I expect.

I tried to follow [this answer](http://stackoverflow.com/a/894284/533618) but
it does not solve my problem.

    
    
    import multiprocessing
    import threading
    import logging
    import time
    
    SLEEP_TIME = 0.000001
    
    logger = logging.getLogger()
    
    ch = logging.StreamHandler()
    ch.setFormatter(logging.Formatter('%(asctime)s %(levelname)s %(funcName)s(): %(message)s'))
    ch.setLevel(logging.DEBUG)
    
    logger.setLevel(logging.DEBUG)
    logger.addHandler(ch)
    
    
    class LoggingThread(threading.Thread):
    
        def __init__(self):
            threading.Thread.__init__(self)
    
        def run(self):
            while True:
                logger.debug('LoggingThread: {}'.format(self))
                time.sleep(SLEEP_TIME)
    
    
    def action(i):
        logger.debug('action: {}'.format(i))
    
    
    def do_parallel_job():
    
        processes = multiprocessing.cpu_count()
        pool = multiprocessing.Pool(processes=processes)
        for i in range(20):
            pool.apply_async(action, args=(i,))
        pool.close()
        pool.join()
    
    
    
    if __name__ == '__main__':
    
        logger.debug('START')
    
        #
        # multithread part
        #
        for _ in range(10):
            lt = LoggingThread()
            lt.setDaemon(True)
            lt.start()
    
        #
        # multiprocess part
        #
        do_parallel_job()
    
        logger.debug('FINISH')
    

How to use logging module in multiprocess and multithread scripts?


Answer: This is probably [bug 6721](http://bugs.python.org/issue6721).

The problem is common in any situation where you have locks, threads and
forks. If thread 1 had a lock while thread 2 calls fork, in the forked
process, there will only be thread 2 and the lock will be held forever. In
your case, that is `logging.StreamHandler.lock`.

A fix can be found [here](https://github.com/google/python-
atfork/blob/master/atfork/stdlib_fixer.py) for the `logging` module. Note that
you need to take care of any other locks, too.

