How should I log while using multiprocessing in Python?
Question: Right now I have a central module in a framework that spawns multiple
processes using the Python 2.6 [`multiprocessing`
module](http://docs.python.org/library/multiprocessing.html?#module-
multiprocessing). Because it uses `multiprocessing`, there is module-level
multiprocessing-aware log, `LOG = multiprocessing.get_logger()`. Per [the
docs](http://docs.python.org/library/multiprocessing.html#logging), this
logger has process-shared locks so that you don't garble things up in
`sys.stderr` (or whatever filehandle) by having multiple processes writing to
it simultaneously.

The issue I have now is that the other modules in the framework are not
multiprocessing-aware. The way I see it, I need to make all dependencies on
this central module use multiprocessing-aware logging. That's annoying
_within_ the framework, let alone for all clients of the framework. Are there
alternatives I'm not thinking of?


Answer: I just now wrote a log handler of my own that just feeds everything to the
parent process via a pipe. I've only been testing it for ten minutes but it
seems to work pretty well.

(**Note:** This is hardcoded to `RotatingFileHandler`, which is my own use
case.)

* * *

## Update: Implementation!

This now uses a queue for correct handling of concurrency, and also recovers
from errors correctly. I've now been using this in production for several
months, and the current version below works without issue.

    
    
    from logging.handlers import RotatingFileHandler
    import multiprocessing, threading, logging, sys, traceback
    
    class MultiProcessingLog(logging.Handler):
        def __init__(self, name, mode, maxsize, rotate):
            logging.Handler.__init__(self)
    
            self._handler = RotatingFileHandler(name, mode, maxsize, rotate)
            self.queue = multiprocessing.Queue(-1)
    
            t = threading.Thread(target=self.receive)
            t.daemon = True
            t.start()
    
        def setFormatter(self, fmt):
            logging.Handler.setFormatter(self, fmt)
            self._handler.setFormatter(fmt)
    
        def receive(self):
            while True:
                try:
                    record = self.queue.get()
                    self._handler.emit(record)
                except (KeyboardInterrupt, SystemExit):
                    raise
                except EOFError:
                    break
                except:
                    traceback.print_exc(file=sys.stderr)
    
        def send(self, s):
            self.queue.put_nowait(s)
    
        def _format_record(self, record):
            # ensure that exc_info and args
            # have been stringified.  Removes any chance of
            # unpickleable things inside and possibly reduces
            # message size sent over the pipe
            if record.args:
                record.msg = record.msg % record.args
                record.args = None
            if record.exc_info:
                dummy = self.format(record)
                record.exc_info = None
    
            return record
    
        def emit(self, record):
            try:
                s = self._format_record(record)
                self.send(s)
            except (KeyboardInterrupt, SystemExit):
                raise
            except:
                self.handleError(record)
    
        def close(self):
            self._handler.close()
            logging.Handler.close(self)
    

