Python pyserial slow and crashes after given period of time
Question: When I use Python’s pyserial to obtain voltage and time information from an
Arduino, the Python with Pyserial appears to be slow. When I look in the
Arduino serial monitor, everything is fine, the time and voltage values are as
expected and can go on for as long as is possible of the Arduino. However on
Python, the first thousand or so values (samples) are fine, after a period of
15 - 20 seconds,the plot crashes and data values are corrupted.

The following text file containing the results from the plot illustrate what I
describe:

    
    
    7180.0 2.9521
    7187.0 4.1349
    7193.0 4.51124
    7200.0 3.68035
    7206.0 2.58065
    7214.0 1.86217
    7221.0 1.51026
    7227.0 1.44184
    7234.0 1.39296
    7240.0 1.35386
    7247.0 1.4174
    7255.0 1.43695
    7261.0 1.43695
    7268.0 1.52004
    7274.0 1.53959
    7281.0 1.55425
    7288.0 1.63734
    7294.0 1.65689
    7302.0 1.67155
    7308.0 1.75464
    7315.0 1.76442
    7322.0 1.78397
    7328.0 1.86706
    7335.0 1.87683
    7342.0 1.91593
    7349.0 2.0088
    7356.0 1.99902
    7362.0 2.02346
    7369.0 2.09677
    7375.0 2.087
    7383.0 2.10166
    7390.0 2.15054
    7396.0 2.10655
    7403.0 2.09677
    7409.0 2.10655
    7416.0 2.02346
    7424.0 1.97947
    7430.0 1.95015
    7437.0 1.83773
    7443.0 1.77908
    7450.0 1.77908
    7457.0 1.6911
    7463.0 1.64223
    7471.0 1.65689
    7477.0 1.59335
    7484.0 1.60802
    7491.0 1.63734
    7497.0 1.60313
    7504.0 1.64712
    7511.0 1.68622
    7518.0 1.65689
    7525.0 1.69599
    7531.0 1.73509
    7538.0 1.70577
    7544.0 1.74976
    7552.0 1.80352
    7559.0 1.74487
    7565.0 1.82796
    7572.0 1.81329
    7578.0 1.77908
    7585.0 1.82796
    7592.0 1.83773
    7599.0 1.80352
    7606.0 1.84751
    7612.0 1.85239
    7619.0 1.81329
    7626.0 1.85239
    7632.0 1.84751
    7640.0 1.80841
    7646.0 1.85728
    7653.0 1.86706
    7660.0 1.83773
    7666.0 1.86706
    7673.0 1.85728
    7680.0 1.83284
    7687.0 1.8915
    7694.0 1.88661
    7700.0 1.87683
    7707.0 1.95015
    7713.0 1.92571
    7720.0 1.92571
    7728.0 2.01857
    7734.0 2.03324
    7741.0 2.01857
    7747.0 2.03324
    7754.0 1.9306
    7761.0 1.86706
    7768.0 1.85728
    7775.0 1.79863
    7781.0 1.78397
    7788.0 1.82307
    7795.0 1.78397
    7801.0 1.77908
    7809.0 1.82796
    7815.0 1.78886
    7822.0 1.77419
    7829.0 1.76442
    7835.0 1.74487
    7842.0 2.10166
    7848.0 3.04497
    7856.0 4.174
    7863.0 4.50635
    7869.0 3.65591
    7876.0 2.48778
    7882.0 1.80352
    7889.0 1.52981
    7897.0 1.39785
    7903.0 1.38807
    7910.0 1.40762
    7916.0 1.36364
    7923.0 1.42717
    7930.0 1.48583
    7937.0 1.47605
    7944.0 1.53959
    7950.0 1.59335
    7957.0 1.57869
    7964.0 1.64712
    7970.0 1.69599
    7977.0 1.68622
    7984.0 1.76442
    7991.0 1.80841
    7998.0 1.80841
    8004.0 1.88172
    8011.0 1.92571
    8018.0 1.92082
    8025.0 1.99902
    8032.0 2.03812
    8038.0 2.03324
    8045.0 2.10166
    8052.0 2.11632
    8058.0 2.087
    8066.0 2.11632
    8072.0 2.09677
    8079.0 2.03812
    8086.0 2.05279
    8092.0 1.98925
    8099.0 1.89638
    8105.0 1.88172
    8113.0 1.79863
    8120.0 1.71554
    8126.0 1.71554
    8133.0 1.66178
    8139.0 1.61779
    8146.0 1.66178
    8154.0 1.62268
    8160.0 1.6129
    8167.0 1.67155
    8173.0 1.65689
    8180.0 1.65689
    8187.0 1.71554
    8194.0 1.70088
    8201.0 1.70577
    8207.0 1.76442
    8214.0 1.73998
    8221.0 1.75464
    8227.0 1.80841
    8235.0 1.78397
    8241.0 1.79863
    8248.0 1.83773
    8255.0 1.81818
    8261.0 1.80352
    8268.0 1.84751
    8274.0 1.80352
    8282.0 1.81818
    8289.0 1.85239
    8295.0 1.80841
    8302.0 1.82796
    8308.0 1.85728
    8315.0 1.81329
    8323.0 1.83773
    8329.0 1.86706
    8336.0 1.82307
    8342.0 1.84751
    8349.0 1.88172
    8356.0 1.82307
    8363.0 1.85728
    8370.0 1.88661
    8376.0 1.85239
    8383.0 1.90127
    8390.0 1.9306
    8396.0 1.89638
    8403.0 1.9306
    8410.0 1.96481
    8417.0 1.96481
    8424.0 2.04301
    8430.0 2.04301
    8437.0 1.96481
    8443.0 1.95015
    8451.0 1.89638
    8458.0 1.0
    8667.0 1.75953
    8674.0 1.75464
    8680.0 1.78886
    8687.0 1.86706
    8694.0 1.85728
    8700.0 16892.0
    8640.0 1.58358
    8646.0 1.652
    8653.0 1.65689
    8660.0 1.68622
    8667.0 1.75953
    8674.0 1.75464
    8680.0 1.78886
    8687.0 1.86706
    8694.0 1.85728
    8700.0 1.90616
    8708.0 1.98436
    8714.0 1.98925
    8721.0 2.03324
    8728.0 299066.0
    9072.0 1.86706
    9079.0 1.88172
    9086.0 1.94037
    25.0 1.87683
    9032.0 855.0
    9438.0 2.10166
    9444.0 2.15054
    9451.0 2.10655
    9458.0 2.07722
    9464.0 2.09677
    9472.0 2.01369
    9478.0 1.95503
    9485.0 13812.0
    9424.0 2.11632
    9431.0 2.10655
    9438.0 2.10166
    9444.0 2.15054
    9451.0 2.10655
    9458.0 2.07722
    9464.0 2.09677
    9472.0 2.01369
    9478.0 1.95503
    9485.0 1.912
    9424.0 2.11632
    9431.0 2.10655
    9438.0 2.10166
    9444.0 2.15054
    9451.0 2.10655
    9458.0 2.07722
    9464.0 2.09677
    9472.0 2.01369
    9478.0 1.95503
    9485.0 1.92082
    9492.0 1.81329
    9498.0 1.75464
    9505.0 1.74487
    9850.0 1.79374
    9857.0 1.78397
    9864.0 1.83284
    9870.0 1.78886
    9877.0 1.735
    10632.0 1.41251
    10639.0 1.48094
    10646.0 1.5347
    10653.0 1.53959
    10660.0 1.62268
    10668.0 1.64712
    10675.0 1.67155
    10683.0 1.75464
    10690.0 1.75953
    10697.0 1.80352
    10704.0 1.876861
    11046.0 1.87683
    11054.0 1.85239
    11061.0 1.92571
    11068.0 1.91105
    11075.0 1.93548
    11083.0 1.98436
    11090.0 1.94037
    11098.0 2.0039262
    11039.0 1.88661
    11046.0 1.87683
    11054.0 1.85239
    11061.0 1.92571
    11068.0 1.91105
    11075.0 1.93548
    11083.0 1.98436
    11090.0 1.94037
    11098.0 2.003912
    11440.0 2.1652
    11447.0 2.11632
    11454.0 2.06256
    11461.0 2.05279
    11468.0 1.94037
    11475.0 1.8768256
    11418.0 2.13099
    11425.0 2.15543
    11432.0 2.1261
    11440.0 2.1652
    11447.0 2.11632
    11454.0 2.06256
    11461.0 2.05279
    11468.0 1.94037
    11475.0 1.87683
    11484.0 1.82307
    11491.0 1.71065
    168.0 2.06745
    12175.0 2.0479
    12182.0 2.00391
    12189.0 1.87683
    12196.0 1.8279329
    12873.0 1.87195
    12881.0 1.94037
    12889.0 1.9306
    12896.0 2.01369
    12903.0 2.04301
    12910.0 2.06256
    12917.0 2.14565
    12924.0 2.1261
    1293866.0 1.81329
    12873.0 1.87195
    12881.0 1.94037
    12889.0 1.9306
    12896.0 2.01369
    12903.0 2.04301
    12910.0 2.06256
    12917.0 2.14565
    12924.0 2.1261
    12932.0 6.0
    12873.0 1.87195
    12881.0 1.94037
    12889.0 1.9306
    12896.0 2.01369
    12903.0 2.04301
    12910.0 2.06256
    12917.0 2.14565
    12924.0 2.1261
    12932.0 2.1261
    12939.0 2.16031
    123282.0 1.85728
    13289.0 1.8284751
    13231.0 1.83284
    13238.0 1.87195
    13245.0 1.82796
    13252.0 1.83284
    13259.0 1.87195
    13266.0 1.81818
    13274.0 1.85728
    13282.0 1.85728
    13289.0 1.8230334
    13631.0 1.67644
    13638.0 1.74976
    13645.0 1.73998
    13652.0 7.0
    1329.0 1.36364
    14337.0 1.43695
    14345.0 1.43695
    14352.0 1.51026
    14359.0 1.56892
    14366.0 1.56892
    14373.0 1.65689
    14381.0 1.68622
    14388.0 1.70088
    1439329.0 1.36364
    14337.0 1.43695
    14345.0 1.43695
    14352.0 1.51026
    14359.0 1.56892
    14366.0 1.56892
    14373.0 1.65689
    14381.0 1.68622
    14388.0 1.70088
    14395.0 1.79374
    14402.0 1.78886
    144145.0 1.82796
    14752.0 1.86706
    14759.0 1.86706
    14708.0 1.84751
    14715.0 1.90616
    14723.0 1.84262
    14730.0 1.85728
    14737.0 1.87195
    14745.0 1.82796
    14752.0 1.86706
    14759.0 1.86706
    14767.0 1.832392
    15109.0 1.3392
    15116.0 1.39785
    15123.0 1.3783
    15130.0 1.45161
    473.0 1.86217
    15480.0 1.83284
    15488.0 1.83773
    15495.0 1.87683
    15502.0 1.82307
    15509.0 1.84751
    15516.0 1.86706
    15523.0 1.82307
    1553165.0 1.79863
    15473.0 1.86217
    15480.0 1.83284
    15488.0 1.83773
    15495.0 1.87683
    15502.0 1.82307
    15509.0 1.84751
    15516.0 1.86706
    15523.0 1.82307
    15531.0 1.0
    15473.0 1.86217
    15480.0 1.83284
    15488.0 1.83773
    15495.0 1.87683
    15502.0 1.82307
    15509.0 1.84751
    15516.0 1.86706
    15523.0 1.82307
    15531.0 1.81304
    15874.0 3.15738
    15881.0 2.07722
    15888.0 1.56892
    15895.0 1.43695
    15902.0 1.34897
    15909.0 1.37341
    1591751.0 3.43597
    15858.0 4.45748
    15865.0 4.3304
    15874.0 3.15738
    15881.0 2.07722
    15888.0 1.56892
    15895.0 1.43695
    15902.0 1.34897
    15909.0 1.37341
    15917.0 16.80352
    16259.0 1.77908
    16266.0 1.78886
    16274.0 1.83773
    16281.0 1.79863
    1616223.0 1.68133
    16230.0 1.74487
    16237.0 1.75464
    16244.0 1.74487
    16251.0 1.80352
    16259.0 1.77908
    16266.0 1.78886
    16274.0 1.83773
    16281.0 1.79863
    116623.0 1.74487
    16630.0 1.89638
    16637.0 2.42424
    16645.0 3.60215
    16652.0 4.5063417394
    17401.0 4.32063
    17410.0 4.58455
    17417.0 3.47019
    17424.0 2.25806
    17431.0 1.6422352
    17373.0 1.79374
    17380.0 1.74487
    17387.0 2.13587
    17394.0 3.13294
    17401.0 4.32063
    17410.0 4.58455
    17417.0 3.47019
    17424.0 2.25806
    17431.0 1.64223
    17373.0 1.79374
    17380.0 1.74487
    17387.0 2.13587
    17394.0 3.13294
    17401.0 4.32063
    17410.0 4.58455
    17417.0 3.47019
    17424.0 2.25806
    17431.0 1.64223
    18181.0 4.5259
    18188.0 4.4086
    18195.0 3.10362
    18203.0 2.05279
    18210.0 1.5738
    18217.0 1.36852
    1822159.0 1.82796
    18166.0 2.37537
    18173.0 3.55816
    18181.0 4.5259
    18188.0 4.4086
    18195.0 3.10362
    18203.0 2.05279
    18210.0 1.5738
    18217.0 1.36852
    18967.0 1.85728
    18974.0 1.81818
    18981.0 1.83284
    18989.0 1.83284
    18996.0 1.75464
    19004.0 1.98925
    19011.0 2.7664751
    18953.0 1.82796
    18960.0 1.81329
    18967.0 1.85728
    18974.0 1.81818
    18981.0 1.83284
    18989.0 1.83284
    18996.0 1.75464
    19004.0 1.98925
    19011.0 2.7667
    4.0 1.54936
    20161.0 1.58358
    20168.0 1.6129
    20175.0 1.58358
    20183.0 1.65689
    20190.0 1.66178
    20197.0 1.64223
    20205.0 1.70577
    20212.0 1.7057747
    20154.0 1.54936
    20161.0 1.58358
    20168.0 1.6129
    20175.0 1.58358
    20183.0 1.65689
    20190.0 1.66178
    20197.0 1.64223
    20205.0 1.70577
    20212.0 1.70577
    20219.0 81.79374
    20561.0 1.82796
    20569.0 1.78886
    20576.0 1.73998
    20583.0 1.98925
    20999.0 1.76931
    21006.0 1.81818
    21013.0 1.78397
    21020.0 1.82796
    21027.0 1.84751
    21035.0 1.0
    20976.0 1.69599
    20983.0 1.76442
    20990.0 1.73509
    20999.0 1.76931
    21006.0 1.81818
    21013.0 1.78397
    21020.0 1.82796
    21027.0 1.84751
    21377.0 4.56989
    21384.0 3.47507
    21391.0 2.24829
    21399.0 1.62268
    21406.0 1.4174
    21413.0 1.31476
    21421.0 1.33431
    21428.0 1.33431
    21435.0 9.0
    21377.0 4.56989
    21384.0 3.47507
    21391.0 2.24829
    21399.0 1.62268
    21406.0 1.4174
    21413.0 1.31476
    21421.0 1.33431
    21428.0 1.33431
    

After about 8500 you can see some corrupted data. This ruins the plot
completely, I see my options as either changing the way I do things in Python
or using an exception handler to ignore such values, but it would be nice to
know why they happen and therefore to eliminate such issues.

For reference here is the implementation of pyserial on Python:

    
    
    from matplotlib.backends.backend_qt4agg import FigureCanvasQTAgg as FigureCanvas
    from matplotlib.backends.backend_qt4agg import NavigationToolbar2QTAgg as NavigationToolbar
    from matplotlib.figure import Figure
    import PyQt4.QtGui as gui, PyQt4.QtCore as core
    import collections
    import time
    import random
    import numpy as np 
    
    import serial
    ser = serial.Serial('/dev/tty.usbmodem1411', 19200)
    time.sleep(2)
    ser.write('S')
    
    # def get_line():
    #     x = time.time()%10000
    #     y = random.random() * 5
    #     return "%d %.3f\n" % (x,y)
    
    refreshMillis = 5
    N = 1000
    xs = collections.deque(maxlen=N)
    ys = collections.deque(maxlen=N) 
    
    xsLong = []
    ysLong = []
    
    app = gui.QApplication([])
    
    fig = Figure() # container for Axes 
    canvas = FigureCanvas(fig) # holds Figure instance
    
    ax = fig.add_subplot(111) # creates and adds Axes instance to Figure instance 'fig'
    ax.set_ylim([0,5])
    # ax.set_xlim([0,12000])
    ax.set_title('Arduino Electrocardiogram') 
    ax.set_xlabel("Time (ms)")
    ax.set_ylabel("Voltage (V)")
    ax.grid(True)
    line2D, = ax.plot(xs,ys) # adds plot by calling required function 'plot' on axes instance 'ax'
    canvas.show()
    
    waiting =  ser.inWaiting()
    print waiting
    
    
    def process_line():
        line = ser.readline()
        data = map(float,line.split(" ")) 
        xs.append(data[0])
        ys.append(data[1])
        xsLong.append(data[0])
        ysLong.append(data[1])
        line2D.set_data(xs,ys)
        print data
        xmin, xmax = min(xs),max(xs)
        if xmin == xmax:
            ax.set_xlim([xmin,xmin+1])
        else:
            ax.set_xlim([xmin,xmax])
    
        canvas.draw()
    
    
        zipString = zip(xsLong,ysLong)
        f = open("plot_store.txt","w")
        for line in zipString:
            f.write(" ".join(str(x) for x in line) + "\n")
        f.close()
    
    
    
    
    timer = core.QTimer()
    timer.timeout.connect(process_line)
    timer.start(refreshMillis)
    
    app.exec_()
    
    
    ser.flush()
    ser.close()
    


Answer: In process_line() you append data to xsLong as well as ysLong. Thus these
become longer with each pass. Both are then zippend and fed into
plot_store.txt That is instead of appending a line to plot_store.txt you
rewrite it with each pass. Since the data gets longer and longer this will
take more and more time. At one point in time it will make your handler slow
enough to start missing data.

Solution: do not rewrite the file each time but only append the new data.

