Log monitoring with pythonic tail -f and process killing The purpose of this cluster of functions is to tail the log file and kill/restart processes from shell (the latter is not yet implemented).

import sys
import time
import datetime
import os
import re


def read_log(f):
    """
    Basically tail -f
    """
    logfile = open(f)
    logfile.seek(0, os.SEEK_END)
    while True:
        new_line = logfile.readline()
        if new_line:
            yield new_line
        else:
            time.sleep(0.1)


def find_error(line, base_time, line_time):
    """
    Search for ERROR keyword in log line. Return True if found, False if not.
    """
    match = re.search('.*ERROR.*', line)
    if match and line_time &lt; base_time:
        return True
    return False


def parse_time(line):
    """
    Parse string to datetime
    """
    date_regex = r'\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}'
    date_string = re.search(date_regex, line).group(0)
    return datetime.datetime.strptime(date_string, "%Y-%m-%d %H:%M:%S")


def increase_base_time(line_time):
    """
    Return a datetime that is line_time plus one minute
    """
    return line_time + datetime.timedelta(minutes=1)


def monitor(filename):
    """
    Read from a log file.
    For each line check if log line time is later than base comparison time.
    If so, update the base time and set error count to 0.
    Check for errors in line. Increment error count if any are found.
    Check the total error count. If it's greater than five, restart the logged 
    process. 
    Check if process has been restarted before. If so, kill the logged process.
    """
    count = 0
    base_time = datetime.datetime.min
    log = read_log(filename)
    restarted = False
    for line in log:
        line_time = parse_time(line)
        if line_time &gt; base_time:
            base_time = increase_base_time(line_time)
            count = 0
        if find_error(line, base_time, line_time):
            count += 1
        if count &gt;= 5:
            if restarted:
                print("Kill the process") # A placeholder, sorry
            else: 
                print("Restart the process") # Also a placeholder, sorry
                count = 0
                restarted = True


monitor(sys.argv[1])



There are a lot of if-statement for doing checks. I thought about making this class based, but it really doesn't make sense to do it (functions don't need to share states and it is a rather small program). 
Is the number of if's acceptable? How could I refactor the code to make it flow a bit better?

EDIT: As this was brought up, is it necessary that the tailing function would be non blocking? As I understand I wont be doing nothing until a line is yielded. What would be the benefits of going one way or the other?
Using regex for such a simple task as finding if a line contains a string is not worth it. The built-in str.__contains__, being called by the keyword in is faster in this case.

Here are some worst case (i.e. a string that does not contain the string "ERROR") timings for the two functions:

import re

def using_in(s):
    return "ERROR" in s

def using_re(s):
    return bool(re.search('.*ERROR.*', s))




Using re is always slower in this case. It does, of course, have its uses for more complex patterns. Note that the re module also compiles the regex automatically (and caches it), so that is not a way to make it faster in this case, since it is always the same regex being used.

Let's just hope your lines are less than 10,000 characters long.



In general, I think you still have to find a good balance between not having enough functions and having too many. IMO, instead of having a increase_base_time function, I would have a global constant and inline the call:

INCREMENT = datetime.timedelta(minutes=1)

...

for line in log:
    ...
    if line_time &gt; base_time:
        base_time = line_time + INCREMENT
        ...




If read_log is a tail -f equivalent, you could just call it tail_f. In any case, I would make the sleep time an optional parameter.

Note that your function does not ensure that the file is properly closed, which might cause issues. You should always use the with keyword to avoid this.

def read_log(f, sleep=0.1):
    """
    Basically tail -f with a configurable sleep
    """
    with open(f) as logfile:
        logfile.seek(0, os.SEEK_END)
        while True:
            new_line = logfile.readline()
            if new_line:
                yield new_line
            else:
                time.sleep(sleep)


This way the file is closed even if the user presses Ctrl+C.



You should only call your function under a if __name__ == "__main__": guard to allow importing from this script without it (trying and failing to) run:

if __name__ == "__main__":
    monitor(sys.argv[1])