Extract parts of log in Python to import to Excel
Question: I'm trying to get part of a log (txt file) using regex but I'm needing some
help. Basically the log comes like this:

    
    
    Tue Feb 24 17:51:10.835 SRV02    NOTICE  Event Loop - noop
    Tue Feb 24 17:51:10.835 SRV02    NOTICE     Exponential histogram:
    Tue Feb 24 17:51:10.835 SRV02    NOTICE     hist[ 0]: <      0.001: 728941854
    Tue Feb 24 17:51:10.835 SRV02    NOTICE  Event Loop - noop: samples: 728941854; avg: 0.00; min: 0.00; max: 0.00
    Tue Feb 24 17:51:10.835 SRV02    NOTICE  Data Quality Monitor Thread Processing Time
    Tue Feb 24 17:51:10.835 SRV02    NOTICE     Exponential histogram:
    Tue Feb 24 17:51:10.835 SRV02    NOTICE     hist[ 4]: <      0.016:         3
    Tue Feb 24 17:51:10.835 SRV02    NOTICE     hist[ 5]: <      0.032:        23
    Tue Feb 24 17:51:10.835 SRV02    NOTICE     hist[ 6]: <      0.064:        14
    Tue Feb 24 17:51:10.835 SRV02    NOTICE     hist[ 7]: <      0.128:         4
    Tue Feb 24 17:51:10.835 SRV02    NOTICE     hist[ 8]: <      0.256:         6
    Tue Feb 24 17:51:10.835 SRV02    NOTICE     hist[ 9]: <      0.512:         1
    Tue Feb 24 17:51:10.835 SRV02    NOTICE     hist[10]: <      1.024:         2
    Tue Feb 24 17:51:10.835 SRV02    NOTICE  Data Quality Monitor Thread Processing Time: samples: 53; avg: 0.08; min: 0.01; max: 0.67
    Tue Feb 24 17:51:10.835 SRV02    NOTICE  Client Hugepage Memory:   649/4096 MB 
    Tue Feb 24 17:51:10.836 SRV02    NOTICE  DQM: Num R: 0 RD: 0 ED: 0 W: 0 WH: 0 Q: 0 D: 0 DF: 0
    Tue Feb 24 17:51:10.836 SRV02    NOTICE  Num G: 0 M: 0 S: 0 D: 0 U: 0 R: 0 N: 0
    Tue Feb 24 17:51:10.836 SRV02    NOTICE  num_template_allocs                       =          4
    Tue Feb 24 17:51:10.836 SRV02    NOTICE  num_template_frees                        =          0
    Tue Feb 24 17:51:10.836 SRV02    NOTICE  num_internal_book_allocs                  =         24
    

and I need to get the information about the "Exponential histogram", so, in
this example I need to identify the string "Exponential histogram" and get all
the "hist[..." to import to a spreadsheet. Also I need this information:

    
    
    samples: XX; avg: X.XX; min: X.XX; max: X.XX
    

So, in the example above, I'll need to extract and rearrange the data like
this, where "Event Loop - noop" and "Data Quality Monitor Thread Processing
Time" need to be repeated in each line in order to identify the histogram:

    
    
    Event Loop - noop;hist[ 0];0.001;728941854
    Event Loop - noop;samples;728941854;avg;0.00;min;0.00;max;0.00
    Data Quality Monitor Thread Processing Time;hist[ 4];0.016;3
    Data Quality Monitor Thread Processing Time;hist[ 5];0.032;23
    Data Quality Monitor Thread Processing Time;hist[ 6];0.064;14
    (...)
    Data Quality Monitor Thread Processing Time;hist[ 10];1.024;2
    Data Quality Monitor Thread Processing Time;samples;53;avg;0.08;min;0.01;max;0.67
    

Somebody can help me how to do this? Thank you!


Answer: In your example output, you have data that does not exist in your sample
input. Specifically you have more `"Data Quality Monitor Thread Processing
Time"` strings then there are in your data. It seems you want to keep the
nearest indented header?

Anyway, I think it would be easier to just pull the data using a few different
regex statements instead of trying to make one all encompassing one:

    
    
    import re
    hists = re.findall(r'(hist\[\s\d+\]).*?(\d+\.\d+).*?(\d+)',input)
    sample_avg_etc = re.findall(r'(samples): (\d+); (avg): (\d+\.\d+); (min): (\d+\.\d+); (max): (\d+\.\d+)',input)
    

If you need to keep the local header as you show in your sample output. I
don't think you want to use regex. Instead just write a parser which will
extract your data.

You can start this by stripping each line of its `Tue Feb 24 17:51:10.835
SRV02 NOTICE` and then targeting the data line by line, keeping track of the
last header. See the comments, the below returns exactly what you listed
above:

    
    
    import re
    
    def parse(data):
        lines = data.split('\n')  # get the lines by splitting on the newline char
        lines = [line[len("Tue Feb 24 17:51:10.835 SRV02    NOTICE  "):]  for  line in lines]  # remove the number of characters equal to the logging info
        out = []
        header = ''
        for line in lines:
            if line.startswith('   '):
                if line.strip().startswith('hist'):
                    out.append(header + ";" + extract_hist_data(line))  # outsource the specific extracting to a function for ease of readability
            else:                      # header/samples line
                if all(i in line for i in ("samples", "avg", "min", "max")):  # if the line contains all these keywords
                    out.append(header + ";" + extract_stat_data(line))  # outsource the specific extracting to a function for ease of readability
                else:  # Treat as a header
                    header = line
        return '\n'.join(out)
    
    def extract_hist_data(line):
        data = re.findall(r'(hist\[\s*?\d+\]).*?(\d+\.\d+).*?(\d+)',line)
        if len(data) > 0:
            data = data[0]
        else:
            return ""
        return ';'.join(i for i in data)
    
    def extract_stat_data(line):
        data = re.findall(r'(samples).*?(\d+).*?(avg).*?(\d+\.\d+).*?(min).*?(\d+\.\d+).*?(max).*?(\d+\.\d+)',line)
        if len(data) > 0:
            data = data[0]
        else:
            return ""
        return ';'.join(i for i in data)
    
    def parse_log_file(log_file_path):
        with open(log_file_path,'r') as f:
            content = ''.join(i for i in f)
        return parse(content)
    
    print parse_log_file('test.log')
    

