Sending corrupted csv line over the socket in Python 3
Question: I build a sensor unit where data are gathered at one Raspberry Pi and send to
another's over the network. My first Pi creates a line with multiple readings
from different sensors. It supposed to create a server and send it to clients.
The client Pis needs to receive the sentence, do further processing or
visualisation.

To test my solutions I want to read data from a txt file, which was build in
an experiment. The problem is that sometimes data are corrupted, has different
format depending on sensor and rows can be different for different set-ups.

I have build a function which suppose to change the input line to bytes. (I
tried different methods but only this clunky function is the closest to any
results). But it does not convert over the network

    
    
    import struct
    
    message = ['First sensor', 'second data', 'third',1, '19.04.2016', 0.1]
    
    def packerForNet(message): 
        pattern = ''
        newMessage = []
        for cell in message:
            if isinstance( cell, int ):
                pattern += ('I')
                newMessage.append(cell)
            elif isinstance( cell, float ):
                pattern += ('d')
                newMessage.append(cell)
            elif isinstance(cell, str):
                pattern += (str(len(cell)))
                pattern += ('s')
                newMessage.append(cell.encode('UTF-8'))
            else:
                cell = str(cell)
                pattern += (len(cell))
                pattern += ('s')
                newMessage.append(cell.encode('UTF-8'))            
        return (newMessage, pattern)
    
    newMessage, pattern = packerForNet(message)
    patternStruct = struct.Struct(pattern)
    packedM = patternStruct.pack(*newMessage)  
    

The output from the function does not unpack correctly:

    
    
    packedM = b'First sensorsecond datathird\x01\x00\x00\x0019.04.2016\x00\x00\x00\x00\x00\x00\x9a\x99\x99\x99\x99\x99\xb9?'
    56
        print('unpacked = %s' % patternStruct.unpack(packedM))
    TypeError: not all arguments converted during string formatting    
    

In addition I need to know the `pattern` to unpack it on the client side so in
general it doesn't have sense.

In final version the server needs to work in the way that after client
connects it will send to the client line from the sensors every millisecond.
Sensors parsing is implemented in C and at the moment it creates a txt file
for off-line processing. I can't change the way the sensor's line are made.

I don't know how to pack the list of different types and constantly send such
lists to the client.


Answer: Actually it does unpack correctly. The problem is not with the unpacking, but
with the print.

Try:

    
    
    unpackedM = patternStruct.unpack(packedM)
    print(unpackedM)
    

unpackedM is a tuple of multiple values. The formatting of the string with the
tuple failed.

EDIT: To convert entire objects you can use python
[msgpack](https://pypi.python.org/pypi/msgpack-python). That is what we use in
communication python to python and php to python.

