Centralize logs from Python and NodeJS thanks to RabbitMQ
Question: I have a service using NodeJS, RabbitMQ and Python workers. The NodeJS brokers
use MongoDB and the Python workers have only a connection to the rabbitMQ
server.

I would like to be able to centralize all the logs from the different
languages in a db.

My idea was to push all the logs in a rabbitMQ queue and then push them in the
mongoDB used by NodeJS.

I would like to know if this is the best way to have centralized log and how
can I redirect the logging python module to a pika consumer?


Answer: This is custom python logging Handler for RabbitMQ which I had written. I had
inherited Handler class and had overridden emit and close methods. I used
Kombu as python library to access RabbiMQ

    
    
    import logging
    
    class RabbitMQHandler(logging.Handler):
        """
         A handler that acts as a RabbitMQ publisher
         Requires the kombu module.
    
         Example setup::
    
            handler = RabbitMQHandler('amqp://guest:guest@localhost//', queue='my_log')
        """
        def __init__(self, uri=None, queue='logging'):
            logging.Handler.__init__(self)
            try:
                import kombu
            except ImportError:
                raise RuntimeError('The Kombu library is required for the RabbitMQSubscriber.')
            if uri:
                connection = kombu.Connection(uri)
    
            self.queue = connection.SimpleQueue(queue)
    
        def emit(self, record):
            self.queue.put(record.msg)
    
        def close(self):
            self.queue.close()
    

Usage :

    
    
    # our own logger
    my_logger = logging.getLogger('teja_logger')
    my_logger.setLevel(logging.DEBUG)
    
    # rabbitmq handler
    rabbitmq_handler = RabbitMQHandler('amqp://guest:guest@localhost:5672//', queue = 'log_q')
    
    # adding rabbitmq handler
    my_logger.addHandler(rabbitmq_handler)
    my_logger.debug('hello')
    

