How do I get only those lines that has highest value if they are inside a timewindow?
Question: I am new to the python and scripting in general, so I would really appreciate
some guidance in writing a python script. So, to the point:

I have a big number of files in a directory. Some files are empty, other
contain rows like that:

16 2009-09-30T20:07:59.659Z 0.05 0.27 13.559 6  
16 2009-09-30T20:08:49.409Z 0.22 0.312 15.691 7  
16 2009-09-30T20:12:17.409Z -0.09 0.235 11.826 4  
16 2009-09-30T20:12:51.159Z 0.15 0.249 12.513 6  
16 2009-09-30T20:15:57.209Z 0.16 0.234 11.776 4  
16 2009-09-30T20:21:17.109Z 0.38 0.303 15.201 6  
16 2009-09-30T20:23:47.959Z 0.07 0.259 13.008 5  
16 2009-09-30T20:32:10.109Z 0.0 0.283 14.195 5  
16 2009-09-30T20:32:10.309Z 0.0 0.239 12.009 5  
16 2009-09-30T20:37:48.609Z -0.02 0.256 12.861 4  
16 2009-09-30T20:44:19.359Z 0.14 0.251 12.597 4  
16 2009-09-30T20:48:39.759Z 0.03 0.284 14.244 5  
16 2009-09-30T20:49:36.159Z -0.07 0.278 13.98 4  
16 2009-09-30T20:57:54.609Z 0.01 0.304 15.294 4  
16 2009-09-30T20:59:47.759Z 0.27 0.265 13.333 4  
16 2009-09-30T21:02:56.209Z 0.28 0.272 13.645 6

and so on.

I want to get this lines out of the files into a new file. But there are some
conditionals! If two or more successive lines are inside a timewindow of 6
seconds, then only the line with highest treshold should be printed into the
new file.

So, something like that:

Original:  
16 2009-09-30T20:32:10.109Z 0.0 0.283 14.195 5  
16 2009-09-30T20:32:10.309Z 0.0 0.239 12.009 5

in output file:  
16 2009-09-30T20:32:10.109Z 0.0 0.283 14.195 5

Keep in mind, that lines from different files can have times inside 6s window
with lines from other files, so the line, that will be in output is the one
that has highest treshold from different files.

The code that explains what is what in the lines is here:

    
    
    import glob 
    from datetime import datetime
    
    path = './*.cat'   
    files=glob.glob(path)   
    for file in files:  
    
        in_file=open(file, 'r')  
        out_file = open("times_final", "w")
    
        for line in in_file.readlines():
            split_line = line.strip().split(' ')
            template_number = split_line[0]
            t = datetime.strptime(split_line[1], '%Y-%m-%dT%H:%M:%S.%fZ')
            mag = split_line[2]
            num = split_line[3]
            threshold = float(split_line[4])
            no_detections = split_line[5]
    
    in_file.close()
    out_file.close()
    

Thank you very much for hints, guidelines, ...


Answer: you said in the comments you know how to merge multiple files into 1 sorted by
`t` and that the 6 second windows start with the first row and are based on
actual data.

so, you need a way to remember the maximum threshold per window and write only
after you are sure you processed all rows in a window. sample implementation:

    
    
    from datetime import datetime, timedelta
    from csv import DictReader, DictWriter
    
    fieldnames=("template_number", "t", "mag","num", "threshold", "no_detections")
    with open('master_data') as f_in, open("times_final", "w") as f_out:
        reader = DictReader(f_in, delimiter=" ", fieldnames=fieldnames)
        writer = DictWriter(f_out, delimiter=" ", fieldnames=fieldnames,
                            lineterminator="\n")
        window_start = datetime(1900, 1, 1)
        window_timedelta = timedelta(seconds=6)
        window_max = 0
        window_row = None
        for row in reader:
            try:
                t = datetime.strptime(row["t"], "%Y-%m-%dT%H:%M:%S.%fZ")
                threshold = float(row["threshold"])
            except ValueError:
                # replace by actual error handling
                print("Problem with: {}".format(row))    
            # switch to new window after 6 seconds
            if t - window_start > window_timedelta:
                # write out previous window before switching
                if window_row:
                    writer.writerow(window_row)
                window_start = t
                window_max = threshold
                window_row = row
            # remember max threshold inside a single window
            elif threshold > window_max:
                window_max = threshold
                window_row = row
        # don't forget the last window
        if window_row:
            writer.writerow(window_row)
    

