title: io.BytesIO is very slow. Alternatives? Optimizations?
body: <p>I am running a Python v3.5 script on a Raspberry Pi with a camera.  The program involves recording video from the <code>picamera</code> and taking a sample frame from the video stream to perform operations on.  Sometimes, it takes a very long time (20+ s) to deal with the byte buffer.  A simplified version of the code is containing the problem area is:</p>
  
  <pre><code>import io
  import picamera
  
  camera = picamera.PiCamera()
  camera.start_recording("/path/to/file.h264")
  cnt = 0
  while True:
      if cnt &gt; 30:
          stream = io.BytesIO()
          camera.capture(stream, use_video_port=True, resize=(1920, 1080), format='rgba')
          cnt = 0
      else:
          cnt += 1
  </code></pre>
  
  <p>After a while, the time it takes to open the bytestream goes crazy.  In my latest run, one instance took over 48 seconds!  <a href="https://i.stack.imgur.com/LiQfm.png" rel="nofollow noreferrer">This figure shows a plot of the times to open the byte stream for each cycle.</a>  I performed a timing test for each line in the problematic area of the code, and I can confirm that it is the <code>stream = io.BytesIO()</code> line causing the delays.</p>
  
  <p>When I monitor the CPU and memory of the Raspberry Pi during this task using <code>psutils</code>, I observe no obvious problems.  CPU usage sits at 10-15%, virtual memory use ~24.2%, and 0 swap is being used.</p>
  
  <p>Aside from the Python program, no other user-executed processes are running on the Pi.  The hardware is running a default Raspbian installation with GUI.  </p>
  
  <p>Since the Python program is 1000+ lines, I am not going to include anything beyond the minimal example in this question text.  If you would like to take a look at it for contextual information, <a href="https://gist.github.com/BlueNalgene/be6293848dd1f62101b05b39b04c8303" rel="nofollow noreferrer">please take a look at this Gist with the code</a>.</p>
  
  <p>Preliminary searches suggest that this is a known issue with BytesIO.  Some old bugtracking (ca. 2014) for Python suggests that this was improved for some cases in the 3.5 release.  </p>
  
  <p>The questions are:</p>
  
  <ul>
  <li>Why is <code>BytesIO</code> slow here?</li>
  <li>Is there an alternative way to stream bytes which is faster?</li>
  <li>Is there a better way to use <code>BytesIO</code> to get what I need?</li>
  </ul>
  
  <p><strike>EDIT: I added a line to the loop forcing the stream to close at the end of each process using <code>stream.close()</code>, but this appears to have been ineffective.  I still had stream opening times of 20+ seconds.</strike></p>
  
  <p>EDIT_2: I misread values in the test from the edited information and missed that values had scientific notation.</p>