Generator Reading Log File While Name Changes Python
Question: The goal is to read a log file in real time line by line (standard generator
stuff) but the catch is, the file name changes at various intervals. The name
change can't be helped (application dictated appended with a time string) and
the name is changed when the log file size reaches ~2MB (guesstimate).

My approach was to create a file getter function that got the file (or new
file) and then passed that to the generator. I thought that when the file
changed names I would get a 'File not found' error, but what my test showed,
is that the file name change is prevented entirely as 'another program is
using this file'. The name change must be allowed, and this reader code cannot
interfere with the application logging process at all.

    
    
    import os
    import time
    import fnmatch
    
    directory = '\\foo\\'
    
    def fileGenerator(logFile):
         """ Run a line generator """
         logFile.seek(0,2)
         while True:
             line = logFile.readline()
             if not line:
                 time.sleep(0.1)
                 continue
             yield line
    
    def fileGetter():
         """ Get the Logging File """
         matchedFiles = []
         for afile in os.listdir(directory):
            if fnmatch.fnmatch(afile,'amc_*.txt'):
                matchedFiles.append(afile)
         if len(matchedFiles)==1:
            #There was exactly one matching file found send it to the generator
            return os.path.join(directory,matchedFiles[0])
        else:
            #There either wasn't a file found or many matching
            #Error out and stop process... critical error
    
    if __name__ == '__main__':
        filePath = fileGetter()
        try:
            logFile = open(filePath,"r")
        except Exception as e:
            #Catch the file not found and go back to the file path getter
            #Send the file back to the generator
            print e
        if logFile:
            loglines = fileGenerator(logFile)
            for line in loglines:
                #handle the line
                print line,
    


Answer: If you can't hold the file open while waiting for new content to be written to
it, I suggest saving the file position you were last at and closing the file
before you sleep, and then reopening the file and seeking to that point
afterwards. You could also investigate filesystem notification systems if you
care about spotting file additions or renames immediately.

    
    
    def log_reader():
        filename = "does_not_exist"
        filepos = 0
    
        while True:
            try:
                file = open(filename)
            except FileNotFoundError:
                filename = fileGetter()
                # if renamed files start empty, set filepos to zero here!
                continue
    
            file.seek(filepos)
    
            while True:
                line = file.readline()
                if not line:
                    filepos = file.tell()
                    file.close()
                    sleep(0.1) # you may want to test different sleep lengths to avoid FS thrash
                    break
                yield line
    

The opening and closing of the file may stress out your filesystem if you do
it too much, so I'd suggest sleeping longer than your previous code did (but
you may want to test to see how well your OS handles it if you care about how
responsive your log reader is).

