Python Error Getting Timestamp from List Slice
Question: I'm in the process of writing a script that chunks a large CSV file down into
smaller chunked files. It cross references a log file that holds the last
timestamp that was chunked so that only the timestamps that are later than the
logged time are written/chunked.

The first column of the csv file has a timestamp that is in `%Y%m%d %H%M%S`
form. The CSV file also has four rows of header information that I don't
want/need in my script, which the `rows in ts_pre` clause removes.

The `log_lookup()` function just pulls the last timeseries from the log for a
particular station's CSV file that I'm viewing. Obviously, I'm working with
six different stations that all have different columns of information, except
they all share the same structure as I described in the second paragraph.

The partial script is:

    
    
    import csv, sys, datetime
    
    def log_lookup():  
        global STN_num
        global STN_date
        with open('/home/log.txt', 'rb') as open_log:    
            log_file = csv.reader(open_log)
                for row in log_file:
                    for item in row:
                         STN_date.append(item)
                if find == 'STN_1':
                    return STN_date[1]
                if find == 'STN_2':           
                    return STN_date[2]
                if find == 'STN_3':
                    return STN_date[3]
                if find == 'STN_4':
                    return STN_date[4]
                if find == 'STN_5':
                    return STN_date[5]
                if find == 'STN_6':
                    return STN_date[6]
    
    def get_ts(line):
        print line[0:19]
        return datetime.datetime.strptime(line, "%Y/%m/%d %H:%M:%S")
    
    def main():
        log = str(log_lookup()) #useful for knowing when to start chunking 
        log_datetime = datetime.datetime.strptime(log, "%Y/%m/%d %H:%M:%S")       
    
        with open(sys.argv[1], 'rb') as open_file: 
            ts_from_file = csv.reader(open_file)
            for genrows in ts_from_file:            
                ts_pre.append(genrows)  
                for rows in ts_pre:             
                    if rownum < 4:
                        ts_pre.pop()
                        rownum += 1
                    else:       
                        for line in rows:                            
                            if get_ts(line) > log_datetime:                 
                                timeseries.append(line)
    

The log file is simply:

    
    
    0
    2011/10/06 18:40:00
    2012/06/27 13:25:00
    1900/01/01 00:00:00
    2011/08/03 14:55:00
    2012/06/27 20:05:00
    2011/10/03 19:25:00
    

... with 0 as a placeholder. _(Is it obvious I'm not a programmer?)_

An example CSV file looks like:

    
    
    "2011/10/03 16:40:00",0,0
    "2011/10/03 16:45:00",1,0
    "2011/10/03 16:50:00",2,0
    "2011/10/03 16:55:00",3,0
    

The error I'm getting when the `ts_line(line)` function is that it's saying
`line[0:19]` is:

    
    
    2011/10/03 16:40:00
    0
    

and the function returns `0` and Python throws this error:

    
    
    ValueError: time data '0' does not match format '%Y/%m/%d %H:%M:%S'
    

I have verified that the `0` that's returned is the second item in the CSV
file, but I'm confused as to why Python returns it at all in my slice
selection. Can someone explain to me why it's returning that value and what I
need to do to get the timestamp to compare to the log timestamp?

For extra credit, any advice on coding/style is always appreciated and/or
advice for better ways accomplish what I'm doing. The CSV files I'm looking at
are quite large _(~8 MB)_ so the more efficient the better.


Answer: It looks like you are calling `get_ts` for each column in every line of the
csv.

Instead of

    
    
    for line in rows:                            
        if get_ts(line) > log_datetime:                 
            timeseries.append(line)
    

Try:

    
    
    if get_ts(line[0]) > log_datetime:                 
        timeseries.append(line)
    

