Python query: iterating through log file
Question: Please can someone help me solve the following query? I have a log file with
thousands of lines like the following:-

    
    
        jarid: 7e5ae720-9151-11e0-eff2-00238bce4216 recv: 1 timestamp: 00:00:02,217
        jarid: 7e5ae720-9151-11e0-eff2-00238bce4216 ack: 13 timestamp: 00:00:04,537
        jarid: 462c6d11-9151-11e0-a72c-00238bbdc9e7 recv: 1 timestamp: 00:00:08,018
        jarid: 462c6d11-9151-11e0-a72c-00238bbdc9e7 nack: 14 timestamp: 00:00:10,338
    

I would like to write a python script to iterate through this file and based
on the jarid (the second field in the log file) to get the timestamp from each
line where the jarid is found and print them on the same line. So for example,
for the following two lines:-

    
    
        jarid: 7e5ae720-9151-11e0-eff2-00238bce4216 recv: 1 timestamp: 00:00:02,217 
        jarid: 7e5ae720-9151-11e0-eff2-00238bce4216 ack: 13 timestamp: 00:00:04,537
    

I would get the following output:-

    
    
        jarid: 7e5ae720-9151-11e0-eff2-00238bce4216 recv: 00:00:02,217 ack: 00:00:04,537
    

I think the best way to accomplish this is with a dictionary (or maybe not!,
please comment). I have written the following script, which is somewhat
working, but it is not giving me the desired output:-

    
    
    #!/opt/SP/bin/python
    
        log = file(/opt/SP/logs/generic.log, "r")
        filecontent = log.xreadlines()
        storage = {}
        for line in filecontent:
            line = line.strip()
            jarid, JARID, status, STATUS, timestamp, TIME = line.split(" ")
            if JARID not in storage:
                storage[JARID] = {}
            if STATUS not in storage[JARID]:
                storage[JARID][STATUS] = {}
            if TIME not in storage[JARID][STATUS]:
                storage[JARID][STATUS][TIME] = {}
    
        jarids = storage.keys()
        jarids.sort()
        for JARID in jarids:
            stats = storage[JARID].keys()
            stats.sort()
            for STATUS in stats:
                times = storage[JARID][STATUS].keys()
                times.sort()
                for TIME in times:
                    all = storage[JARID][STATUS][TIME].keys()
                    all.sort()
    
        for JARID in jarids:
            if "1" in storage[JARID].keys() and "13" in storage[JARID].keys():
                print "MSG: %s, RECV: %s, ACK: %s" % (JARID, storage[JARID]["1"], storage[JARID]["13"])
            else:
                if "1" in storage[JARID].keys() and "14" in storage[JARID].keys():
                    print "MSG: %s, RECV: %s, NACK: %s" % (JARID, storage[JARID]["1"], storage[JARID]["14"])
    

When I run this script, I am getting the following output:-

    
    
        MSG: 7e5ae720-9151-11e0-eff2-00238bce4216, RECV: {'00:00:02,217': {}}, ACK: {'00:00:04,537': {}}
    

Please note that I am still learning python and that my scripting skills are
not all that!

Please, can you help me figure out how to get the desired output as I wrote
above?


Answer: Based on JBernardo's answer, but using defaultdict instead of setdefault. You
can print it exactly the same way, so I won't copy that code here

    
    
    from collections import defaultdict
    log = ['jarid: 7e5ae720-9151-11e0-eff2-00238bce4216 recv: 1 timestamp: 00:00:02,217',
           'jarid: 7e5ae720-9151-11e0-eff2-00238bce4216 ack: 13 timestamp: 00:00:04,537',
           'jarid: 462c6d11-9151-11e0-a72c-00238bbdc9e7 recv: 1 timestamp: 00:00:08,018',
           'jarid: 462c6d11-9151-11e0-a72c-00238bbdc9e7 nack: 14 timestamp: 00:00:10,338']
    
    d = defaultdict(dict)
    for i in (line.split() for line in log):
        d[i[1]][i[2]] = i[-1]
    

You can also unpack into meaningful names. for example

    
    
    for label1, jarid, jartype, x, label2, timestamp in (line.split() for line in log):
        d[jarid][jartype] = timestamp
    

