Python script to alert on empty/missing logs
Question: I am working on a project to check a file directory and automatically add log
files as they are created. A file is being generated every five minutes, but
some of the files are being created with a "0" filesize and I would like to
alert when this happens.

So the sequence of steps I would like to have are essentially:

  * Get time (MM:DD:YY HH:MM:SS) *Not sure if I need to do this...
  * CD to Folder Directory /Netflow/YY/MM/DD
  * Search for filename "nfcapd.YYYYMMDDHHMM" where MM increments by 5.
  * If filesize is 0, then email Johnny, Sally and Jimmy 
  * Wait 6 minutes and repeat

This is what I have pieced together thus far. How can I get the desired
functionality?

    
    
    import os
    def is_non_zero_file(fpath): storage/Netflow/  
    return True if os.path.isfile(fpath) and os.path.getsize(fpath) > 0 else False
    
    # I need to check storage/Netflow for files named by time e.g 13_56_05.txt
    
    while True:
    time.sleep(360)
    


Answer: In addition to enumerating the files in a given path, and subsequently
filtering the files which are only zero-length, you probably want to maintain
some type of state to ensure you're aren't notified multiple times of the same
zero length file. That is, you probably don't want to get a notification that
the same file is zero-length indefinitely (although you can modify the example
below if you want said behavior).

You may optionally want to do things like verify that the file name strictly
meets your naming convention. You may also want to validate the the string
date-stamp included in the file name is a valid datetime.

The example below uses the glob module (itself leveraging `os.listdir()` and
`fnmatch.fnmatch()`) to build up a set of possible files for inclusion. [1]

The example is intentionally simple, and leverages a single class to store log
sample 'state'. `KEEP_SAMPLES` samples are maintained (instances of
`logState()` in the `log_states` list, achieved by using list slicing.

A single `alert(msg)` function is supplied as a stub to something that might
send mail, etc...

References:

[1] <https://docs.python.org/3.2/library/glob.html>

    
    
    #!/usr/bin/python3
    
    import os
    import glob
    import re
    from datetime import datetime, timezone
    import time
    from pprint import pprint
    
    class logState():
    
        def __init__(self, log_path, glob_patt, re_patt, dt_fmt):
    
            self.dt = datetime.now(timezone.utc)
            self.log_path = log_path
            self.glob_patt = glob_patt
            self.re_patt = re_patt
            self.dt_fmt = dt_fmt
            self.empty_logs = []
            self.nonempty_logs = []
    
            # Retrieve only files from glob
            self.files = [  f for f in
                    glob.glob(self.log_path + self.glob_patt)
                    if os.path.isfile(f) ]
    
            for f in self.files:
    
                unq_fname = f.split('/')[-1]
    
                if unq_fname == None:
                    continue
    
                # Tighter pattern matching
                if re.match(re_patt, unq_fname) == None:
                    continue
    
                # Get the datetime portion of the file name
                f_dtstamp = unq_fname.split('.')[-1]
    
                # Make sure the datetime stamp represents
                # a valid date
                if datetime.strptime(f_dtstamp, self.dt_fmt) == None:
                    continue
    
                # Check file size, add to the appropriate
                # list
                if os.path.getsize(f) <= 0:
                    self.empty_logs.append(f)
                else:
                    self.nonempty_logs.append(f)
    
    def alert(msg):
        print("ALERT!: {0}".format(msg))
    
    if __name__ == "__main__":
    
        # How long to sleep
        SLEEP_SECS = 5
    
        # How many samples to keep
        KEEP_SAMPLES = 5
    
        log_states = []
    
        # Definition for what logs states we'll look for
        log_path = './'
        glob_patt = 'nfcapd.[0-9]*'
        re_patt = 'nfcapd.([0-9]{12})'
        dt_fmt = "%Y%m%d%H%M"
    
        print("-- Setup --")
        print("Sample files in '{0}'".format(log_path))
        print("\t{0} samples kept:".format(KEEP_SAMPLES))
        print("\tglob pattern: '{0}'".format(glob_patt))
        print("\tregex pattern: '{0}'".format(re_patt))
        print("\tdatetime string: '{0}'".format(dt_fmt))
        print("")
    
        # Collect the initial state
        log_states.append(logState(log_path, 
                                   glob_patt, 
                                   re_patt, dt_fmt))
    
        while True:
    
            # Print state inventory and current state detail
            print( "-- Log States Stored --")
            for i, log_state in enumerate(log_states):
                print("Log state {0} @ {1}".format(i, log_state.dt))
    
            print(" -- Logs size > 0 --")
            pprint(log_states[-1].nonempty_logs)
            print(" -- Logs size <= 0 --")
            pprint(log_states[-1].empty_logs)
            print("")
    
            time.sleep(SLEEP_SECS)
            log_states = log_states[-KEEP_SAMPLES+1:]
    
            log_states.append(logState(log_path, 
                                       glob_patt, 
                                       re_patt, 
                                       dt_fmt))
    
            # p = previous sample, c = current
            p = set(log_states[-2].empty_logs)
            c = set(log_states[-1].empty_logs)
    
            # only report the items in the current sample
            # not in the last
            if len(c.difference(p)) > 0:
                alert("\nNew zero length logs: " + str(c.difference(p)) + "\n")
    

