Efficiently convert a string of comma separated values to bytes
Question: My `python3` program is receiving data from elsewhere as a string in the
following format (the `...` means more data that I care to type out):

    
    
    data = "0,12,145,234;1,0,0,128;2,255,255,255;...;909,100,100,100;"
    

I want to convert this to packed binary data where I disregard the `,` and `;`
characters. Currently, I am doing the following:

    
    
    splitData = data.split(';')[:-1] # ignore the last ';'
    buff = []
    for item in splitData:
        addr, R, G, B = item.split(',')
        addr = int(addr) # two bytes
        R    = int(R)    # one byte
        G    = int(G)    # one byte
        B    = int(B)    # one byte
        packed = struct.pack('HBBB', addr, R, G, B)
        buff.append(packed)
    dataBytes = b''.join(buff)
    

For my example data above, this process gives me the following:

    
    
    dataBytes = b'\x00\x00\x0c\x91\xea\x01\x00\x00\x00\x80...\x8d\x03ddd'
    

which is what I want (and is about one third the size of the original string).

However, this process is taking about `0.002` seconds. I need to do this
process 33 times per frame, which leads to about `0.05` seconds to compute,
amounting to about 20 frames per second. I would like to speed this up, if
possible.

Is there a way to convert from the string data to byte data which is faster
than the method above?


Answer: Using itertools, doing a replace then splitting, mapping to int and finally
zipping in fours is about 25 percent faster:

    
    
     In [82]: data = "0,12,145,234;1,0,0,128;2,255,255,255;909,100,100,100;" * 1000
     In [83]: from itertools import  imap, izip
     [84]: %%timeit  
    splitData = data.split(';')[:-1] # ignore the last ';'
    buff = []
    for item in splitData:
        addr, R, G, B = item.split(',')
        addr = int(addr) # two bytes
        R    = int(R)    # one byte
        G    = int(G)    # one byte
        B    = int(B)    # one byte
        packed = struct.pack('HBBB', addr, R, G, B)
        buff.append(packed)
    dataBytes = b''.join(buff)
       ....: 
    100 loops, best of 3: 8.61 ms per loop
    
    In [85]: %%timeit     
    mapped = imap(int, data[:-1].replace(";", ",").split(","))
    b"".join([struct.pack('HBBB', *sub) for sub in izip(mapped, mapped, mapped, mapped)])
       ....: 
    100 loops, best of 3: 6.27 ms per loop
    

Using python3, just use map and zip:

    
    
    In [4]: %%timeit
    mapped = map(int, data[:-1].replace(";", ",").split(","))
    b"".join([struct.pack('HBBB', *sub) for sub in zip(mapped, mapped, mapped, mapped)])
       ...: 
    100 loops, best of 3: 3.61 ms per loop
    
    In [5]: %%timeit        
    splitData = data.split(';')[:-1] # ignore the last ';'
    buff = []                                                                  for item in splitData:
        addr, R, G, B = item.split(',')
        addr = int(addr) # two bytes
        R    = int(R)    # one byte
        G    = int(G)    # one byte
        B    = int(B)    # one byte
        packed = struct.pack('HBBB', addr, R, G, B)
        buff.append(packed)
    dataBytes = b''.join(buff)
       ...: 
    100 loops, best of 3: 4.89 ms per loop
    

