Reading a serial port in python with unknown data length
Question: Hello I am trying to read data from a pic32 microcontroller configured as a
serial port.

The pic32 sends "binary" data variable in length (14 to 26 bytes long). I want
to read in the data and separate the bits then convert them to their decimal
equivalent.

    
    
    import serial
    import csv
    
    #open the configuartion file
    with open('config.txt') as configFile:
       #save the config parameters to an array called parameters
       parameters = configFile.read().split()
    
    #Function to Initialize the Serial Port
    def init_serial():
        global ser
        ser = serial.Serial()
        ser.baudrate = 9600
        ser.port = 'COM7'
        ser.timeout = 10
        ser.open()
        if ser.isOpen():
            print ('Open: ' + ser.portstr)
    
    #call the serial initilization function
    init_serial()
    
    #writes the lines from the config file to the serial port
    counter = 0
    while counter<4:
        ser.write(chr(int(parameters[counter])).encode('utf-8') + chr(int(parameters[counter+1])).encode('utf-8'))
        counter = counter + 2
    
    #opens the csv file to append to 
    resultsFile = open('results.csv', 'wt')
    
    #writes the titles of the four columns to the csv file
    resultsFile.write("{} {} {} {}\n".format('ChannelAI', 'ChannelAQ', 'ChannelBI', 'ChannelBQ')) 
    
    
    count=0
    while count < 10:    
    
        #read from serial port
        incoming = ser.read(26)
    
        #decodes incoming bytes to a string
        incoming = incoming.decode('cp1252')
    
    
        #will select element 4, 5 & 6 from incoming data
        channelAIstr = incoming[4:6]
        #converts slected elements to an integer
        channelAI=int(channelAIstr, 16)
    
        channelAQstr = incoming[7:10]
        #channelAQ=int(channelAQstr, 16)
    
        channelBIstr = incoming[10:13]
        #channelBI=int(channelBIstr, 16)
    
        channelBQstr = incoming[13:16]
        #channelBQ=int(channelBQstr, 16)
    
        #writes to csv file
        resultsFile.write("{} {} {} {}\n".format(str(channelAI), str(channelAQ), str(channelBI), str(channelBQ)))             
    
        count = count + 1
    
    #close the file to save memory
    resultsFile.close()
    

I am having some trouble properly reading and converting the bits from the
serial port. Any help on how to do this would be appreciated.

I know I am reading the serial port correctly and am getting data that looks
something like this "\x00\x7f\x7f" as an example. I then want to convert this
3 byte long string to an integer.


Answer: I am not sure how you determine if the data is 14 or 26 bytes long or anything
in-between.

In all cases, you might want to use a wrapper class which wraps the IO.

On every request of data, you can have the wrapper either read a certain
number of bytes or all bytes which are available, until you have enough to
decode. Then the wrapper decodes them and returns them as a tuple.

This is a first guess about how you can proceed; I do not see where exactly
you are having trouble in your code.

* * *

But this is only syntactig sugar and for a more advanced state of your
program.

For the start, let's locate the more obvious things: You seem to be ripping
the data apart in a wrong way. Your code

    
    
    #will select element 4, 5 & 6 from incoming data
    channelAIstr = incoming[4:6]
    

does not fit to each other: indexing `[4:6]` means 4 inclusive to 6 exclusive.
If you really want 4, 5 and 6, you have to write `[4:7]`.

The next step is to convert this into an integer. If you have `\x00\x7f\x7f`,
this can mean a lot of things:

  * If it is a 24 bit integer, it can be little or big endian. In the first case, it is `0x7F7F00`, in the second case it is `0x007F7F`.

These cases can be dealt with

    
        >>> a='\x00\x7f\x7f'
    >>> import struct
    >>> struct.unpack("<I", a+"\x00")[0]
    8355584
    >>> 0x7f7f00
    8355584
    >>> struct.unpack(">I", "\x00"+a)[0]
    32639
    >>> 0x7f7f
    32639
    

  * If it is a different kind of data format (maybe some floating point?), you have to be more verbose what these data mean.

So, if I am right about the integer solution, just do any of

    
    
    channelAI = struct.unpack(">I", "\x00" + channelAIstr)[0]
    channelAI = struct.unpack("<I", channelAIstr + "\x00")[0]
    

