Python Socket Programming - Messages Getting Truncated
Question: I have a GPS modem (Sixnet BT-5800) that attempts to broadcast GPS NMEA
messages over ethernet to my Linux client on a timed interval.

On the client I have a python script running. I was hoping if someone could
identify if I'm doing something wrong here.

    
    
    '''
    The main program waits on the TCP socket to receive data, parses the data into
    GPS NMEA sentences and writes them to the MySQL database
    '''
    # python library
    import configparser
    import select
    import socket
    import sys
    from time import sleep
    
    # custom library
    #import gpsnmeapacketparser
    
    # open the configuration files
    config = configparser.ConfigParser()
    files = ['.config.host', '.config', '.config.mysql']
    dataset = config.read(files)
    if (len(files) != len(dataset)):
        print("Error: Failed to open/find configuration files. Has this package been installed?")
        exit()
    
    
    def main():
        host_address = config['HOST']['IPAddress']
        host_gps_port = config['HOST']['GPSPort']
    
        # create a tcp/ip socket
        sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    
        sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    
        server_address = (host_address, int(host_gps_port, 10))
    
        print("Binding socket:", server_address)
        sock.bind(server_address)
    
        # listen for incoming connections
        sock.listen(5)
    
    
        while True:
            block = ""
    
            # wait for a connection
            print('waiting for a connection')
            connection, client_address = sock.accept()
    
            print('connection from', client_address)
    
            data = bytearray([])
            buf = bytearray([])
    
            while True:
                buf = connection.recv(10)
    
                if buf != b'':
                    data += buf
                else:
                    break
    
            if data != b'':
                block = data.decode("utf-8")
                print(block)
                print()
    
            else:
                connection.close()   
    
    if __name__ == "__main__":
        sys.exit(main())
    

The modem seems to max out at 512 byte messages. Running the python script I
will see output like the following:

    
    
    waiting for a connection
    connection from ('192.168.0.1', 4433)
    waiting for a connection
    connection from ('192.168.0.1', 4434)
    $GPRMC,210458.00,A,4437.35460,N,07545.93616,W,000.0,000.0,180315,13.4,W,A*0F
    $GPGGA,210458.00,4437.35460,N,07545.93616,W,1,08,0.91,00121,M,-034,M,,*54
    $GPGLL,4437.35460,N,07545.93616,W,210458.00,A,A*79
    $GPVTG,000.0,T,013.4,M,000.0,N,000.0,K,A*25
    $GPGSV,3,1,09,31,32,089,36,03,19,236,29,16,77,229,36,23,57,292,35*75
    $GPGSV,3,2,09,10,07,326,23,29,08,032,17,08,58,067,41,09,29,312,36*73
    $GPGSV,3,3,09,27,26,164,37,,,,,,,,,,,,*46
    $GPGSA,A,3,31,03,16,23,10,29,09,27,,,,,1.61,0.91,1.33*0C
    $GPZDA,210458.00,18,0
    
    waiting for a connection
    connection from ('192.168.0.1', 4435)
    waiting for a connection
    connection from ('192.168.0.1', 4436)
    $GPRMC,210528.00,A,4437.35458,N,07545.93617,W,000.0,000.0,180315,13.4,W,A*03
    $GPGGA,210528.00,4437.35458,N,07545.93617,W,1,07,1.05,00121,M,-034,M,,*5B
    $GPGLL,4437.35458,N,07545.93617,W,210528.00,A,A*75
    $GPVTG,000.0,T,013.4,M,000.0,N,000.0,K,A*25
    $GPGSV,3,1,09,31,32,089,36,03,19,236,30,16,77,229,36,23,57,292,35*7D
    $GPGSV,3,2,09,10,07,326,22,29,08,032,06,08,58,067,42,09,29,312,35*72
    $GPGSV,3,3,09,27,26,164,35,,,,,,,,,,,,*44
    $GPGSA,A,3,31,03,16,23,10,09,27,,,,,,1.77,1.05,1.42*0A
    $GPZDA,210528.00,18,03,
    
    waiting for a connection
    connection from ('192.168.0.1', 4437)
    

During execution the `connection.recv(10)` runs to gather data until it
returns the empty array, and then it runs once more and times out. (**This is
a secondary problem, how can I make sure I've received all data without having
to wait on this timeout?**)

Here is the output of a tcpdump

    
    
    17:09:27.907495 IP 192.168.0.1.4621 > 192.168.0.5.8763: Flags [F.], seq 1, ack 1, win 2920, options [nop,nop,TS val 5166242 ecr 6220255], length 0
    17:09:27.907667 IP 192.168.0.5.8763 > 192.168.0.1.4621: Flags [F.], seq 1, ack 2, win 202, options [nop,nop,TS val 6224000 ecr 5166242], length 0
    17:09:27.908091 IP 192.168.0.1.4621 > 192.168.0.5.8763: Flags [.], ack 2, win 2920, options [nop,nop,TS val 5166242 ecr 6224000], length 0
    17:09:27.910329 IP 192.168.0.1.4622 > 192.168.0.5.8763: Flags [S], seq 2455146170, win 5840, options [mss 1460,sackOK,TS val 5166244 ecr 0,nop,wscale 1], length 0
    17:09:27.910390 IP 192.168.0.5.8763 > 192.168.0.1.4622: Flags [S.], seq 3558179681, ack 2455146171, win 25760, options [mss 1300,sackOK,TS val 6224000 ecr 5166244,nop,wscale 7], length 0
    17:09:27.910796 IP 192.168.0.1.4622 > 192.168.0.5.8763: Flags [.], ack 1, win 2920, options [nop,nop,TS val 5166245 ecr 6224000], length 0
    17:09:27.914219 IP 192.168.0.1.4622 > 192.168.0.5.8763: Flags [P.], seq 1:513, ack 1, win 2920, options [nop,nop,TS val 5166248 ecr 6224000], length 512
    17:09:27.914309 IP 192.168.0.5.8763 > 192.168.0.1.4622: Flags [.], ack 513, win 210, options [nop,nop,TS val 6224001 ecr 5166248], length 0
    17:09:42.895197 IP 192.168.0.1.4622 > 192.168.0.5.8763: Flags [F.], seq 513, ack 1, win 2920, options [nop,nop,TS val 5181229 ecr 6224001], length 0
    17:09:42.897588 IP 192.168.0.1.4623 > 192.168.0.5.8763: Flags [S], seq 2470830214, win 5840, options [mss 1460,sackOK,TS val 5181231 ecr 0,nop,wscale 1], length 0
    17:09:42.897643 IP 192.168.0.5.8763 > 192.168.0.1.4623: Flags [S.], seq 2665688556, ack 2470830215, win 25760, options [mss 1300,sackOK,TS val 6227747 ecr 5181231,nop,wscale 7], length 0
    17:09:42.898114 IP 192.168.0.1.4623 > 192.168.0.5.8763: Flags [.], ack 1, win 2920, options [nop,nop,TS val 5181232 ecr 6227747], length 0
    17:09:42.898383 IP 192.168.0.5.8763 > 192.168.0.1.4622: Flags [F.], seq 1, ack 514, win 210, options [nop,nop,TS val 6227747 ecr 5181229], length 0
    17:09:42.898773 IP 192.168.0.1.4622 > 192.168.0.5.8763: Flags [.], ack 2, win 2920, options [nop,nop,TS val 5181232 ecr 6227747], length 0
    

It looks as though the modem never gets properly signalled to send the
remaining characters and those enormous sequence numbers could be error code?

**I can't figure out if the error is in my code, or if the modem is using some
non-standard TCP?**


Answer: TCP is a streaming protocol, you are not receiving the message packet based.
Increase your buffersize drastically and be ready to receive messages that
might be broken and spread over several receive actions.

So yes this is a TCP issue, but it is normal. Because as I said TCP is a
streaming protocol.

If a client sender does this without any pause :

  * send 10 bytes
  * send 10 bytes

the receiver is very likely to get this result :

  * receive 20 bytes

now if the receiver's buffer is too small, chance is very high that he will
loose part of the information. Not what you want of course. The way the
receiver is going to receive the data is not predictable, you should not rely
on it. In theory you should be prepared to even receive the message byte per
byte. But because TCP is implemented to make use of the MTU in the most
intelligent way, it is not very likely that you will receive byte per byte.
But the thing is you just don't know which message is going to be spread over
different receive calls.

Bottom line make your receive buffer a few k's in size.

