python: logging output from external API to logger module
Question: I am python beginner. My python script logs output to a file (say example.log)
using the basic python logging module. However, my python script also makes
some 3rd party API calls (for example, parse_the_file) over which I don't have
any control. I want to capture the output (usually on console) produced by the
API into my example.log. The following example code works partially but the
problem is that the contents get overwritten as soon as I start logging output
of the API into my log file.

    
    
    #!/usr/bin/env python
    
    import logging
    import sys
    import os
    from common_lib import *     # import additional modules
    
    logging.basicConfig(filename='example.log', filemode='w', level=logging.DEBUG, format='%(asctime)s - %(levelname)s - %(message)s')
    
    logging.debug('This is a log message.') # message goes to log file.
    
    sys.stdout = open('example.log','a')
    metadata_func=parse_the_file('/opt/metadata.txt') # output goes to log file but OVERWRITES the content
    sys.stdout = sys.__stdout__
    logging.debug('This is a second log message.') # message goes to log file.
    

I know there have been post suggesting to similar question on this site but I
haven't a workaround/solution for this problem that will work in this
scenario.


Answer: Try:

    
    
    log_file = open('example.log', 'a')
    logging.basicConfig(stream=log_file, level=logging.DEBUG)
    logging.debug("Test")
    sys.stdout = log_file
    sys.stderr = log_file
    stdout_fd = os.dup(1)
    stderr_fd = os.dup(2)
    os.dup2(log_file.fileno(), 1)
    os.dup2(log_file.fileno(), 2)
    
    os.system("echo foo")
    
    os.dup2(stdout_fd, 1)
    os.dup2(stderr_fd, 2)
    sys.stdout = sys.__stdout__
    sys.stderr = sys.__stderr__
    

However, this will not format it accordingly. If you want that, you can try
something like <http://plumberjack.blogspot.com/2009/09/how-to-treat-logger-
like-output-stream.html>

