Python unpacking binary stream from tcp socket
Question: ok, So I thought it would be a good idea to get familiar with Python. (I have
had experience with Java, php, perl, VB, etc. not a master of any, but
intermediate knowledge)

so I am attempting to write a script that will take a the data from a socket,
and translate it to the screen. rough beginning code to follow:

my code seems to correctly read the binary info from the socket, but I can't
unpack it since I don't have access to the original structure.

I have the output for this stream with a different program, (which is terribly
written which is why I am tackling this)

when I do print out the recv, it's like this...

    
    
    b'L\x00k\x07vQ\n\x01\xffh\x00\x04NGIN\x04MAIN6Product XX finished reprocessing cdc XXXXX at jesadr 0c\x00k\x07vQ\n\x01\xffF\x00\x06CSSPRD\x0cliab_checkerCCheckpointed to XXXXXXXXXXXXXXXX:XXXXXXX.XXX at jesadr 0 (serial 0)[\x00l\x07vQ\n\x00\xff\x01\x00\x05MLIFE\x06dayendBdayend 1 Copyright XXXX XXXXXXX XXXXXXX XXXXX XXX XXXXXX XXXXXXXX.
    

from looking at this, and comparing it to the output of the other program, I
would surmise that it should be broken up like..

    
    
    b'L\x00k\x07vQ\n\x01\xffh\x00\x04NGIN\x04MAIN6Product XX finished reprocessing cdc XXXXX at jesadr 0'
    

with corresponding info

    
    
    04-23
    00:00:43
    10
    1
    NGIN
    MAIN
    255
    104
    Product XX finished reprocessing cdc XXXXX at jesadr 0
    

Now, based on my research, it looks like I need to use the "struct" and unpack
it, however I have no idea of the original structure of this, I only know what
info is available from it, and to be honest, I'm having a hell of a time
figuring this out.

I have used the python interpreter to attempt to unpack bits and pieces of the
line, however it is an exercise in frustration.

If anyone can at least help me get started, I would very much appreciate it.

Thanks


Answer: Okay. I think I've managed to decode it, although I'm not sure about the
intermediate 16-bit value.

This Python 2.7 code...

    
    
    from cStringIO import StringIO
    import struct
    import time
    
    def decode(f):
    
        def read_le16(f):
            return struct.unpack('<h', f.read(2))[0]
    
        def read_timestamp(f):
            ts = struct.unpack('<l', f.read(4))[0]
            return time.ctime(ts)
    
        def read_byte(f):
            return ord(f.read(1))
    
        def read_pascal(f):
            l = ord(f.read(1))
            return f.read(l)
    
        result = []
    
        # Read total length
        result.append('Total message length is %d bytes' % read_le16(f))
    
        # Read timestamp
        result.append(read_timestamp(f))
    
        # Read 3 x byte
        result.append(read_byte(f))
        result.append(read_byte(f))
        result.append(read_byte(f))
    
        # Read 1 x LE16
        result.append(read_le16(f))
    
        # Read 3 x pascal string
        result.append(read_pascal(f))
        result.append(read_pascal(f))
        result.append(read_pascal(f))
    
        return result
    
    s = 'L\x00k\x07vQ\n\x01\xffh\x00\x04NGIN\x04MAIN6Product XX finished reprocessing cdc XXXXX at jesadr 0c\x00k\x07vQ\n\x01\xffF\x00\x06CSSPRD\x0cliab_checkerCCheckpointed to XXXXXXXXXXXXXXXX:XXXXXXX.XXX at jesadr 0 (serial 0)[\x00l\x07vQ\n\x00\xff\x01\x00\x05MLIFE\x06dayendBdayend 1 Copyright XXXX XXXXXXX XXXXXXX XXXXX XXX XXXXXX XXXXXXXX.'
    
    f = StringIO(s)
    print decode(f)
    print decode(f)
    print decode(f)
    

...yields...

    
    
    ['Total message length is 76 bytes', 'Tue Apr 23 05:00:43 2013', 10, 1, 255, 104, 'NGIN', 'MAIN', 'Product XX finished reprocessing cdc XXXXX at jesadr 0']
    ['Total message length is 99 bytes', 'Tue Apr 23 05:00:43 2013', 10, 1, 255, 70, 'CSSPRD', 'liab_checker', 'Checkpointed to XXXXXXXXXXXXXXXX:XXXXXXX.XXX at jesadr 0 (serial 0)']
    ['Total message length is 91 bytes', 'Tue Apr 23 05:00:44 2013', 10, 0, 255, 1, 'MLIFE', 'dayend', 'dayend 1 Copyright XXXX XXXXXXX XXXXXXX XXXXX XXX XXXXXX XXXXXXXX.']
    

The timestamps are out by 5 hours, so I'm assuming it's a timezone thing.

