title: File streaming encounters buffer underrun/underflow?
body: <h2>Background</h2>
  
  <p>I'm developing a progressive download media streaming server in C++ using Boost.  A typical configuration is an Android rendering device running Android 4.2.2, using a stock Gallery player as the media player, and the media streaming server running on a Windows desktop.  The Android device requests a media file via an HTTP URL, with the media server streams the file using progressive download.</p>
  
  <h2>The Problem</h2>
  
  <p>When attempting to stream a video file with an internal bit rate of 20 Mbps, the renderer consistently stalls numerous times.  A typical streaming experience consists essentially of multiple occurences of the steps:</p>
  
  <ol>
  <li>Render smoothly for 3-5 seconds</li>
  <li>Video stalls for 5-10 seconds</li>
  <li>Go to step 1</li>
  </ol>
  
  <p>The most logical explanation is that the renderer is experiencing "buffer underrun" or "buffer underflow"</p>
  
  <h2>The Question</h2>
  
  <p>Is there any way to solve the buffer underrun problem, improving the media streamers output rate, and preventing visual stalling/blocking?</p>
  
  <h2>Technical Information</h2>
  
  <p>The server code looks like this:</p>
  
  <pre><code>void StreamFile (boost::asio::ip::tcp::socket *socket, const wchar_t *path)
  {
      . . .
      for (long offset=startOffset; offset &lt;= endOffset; offset+=streamingBlockSize)
      {
          long numBytesToRead = (std::min&lt;long&gt;) (endOffset - offset + 1, streamingBlockSize);
          fread (buffer, 1, numBytesToRead, f);
          if (RawSocketWrite (socket, buffer, numBytesToRead) == 0)
          {
              // RawSocketWrite() encountered a serious error, exit
              break;
          }
      }
      . . .
  }
  
  size_t RawSocketWrite (boost::asio::ip::tcp::socket *socket, const char *data, size_t len)
  {
      size_t numCharsWritten = 0;
  
      try
      {
          numCharsWritten = boost::asio::write (socket, boost::asio::buffer (data, len));
      }
      catch (boost::system::system_error&amp; e)
      {
          LOG_ERROR (("error", "write() failed in RawSocketWrite (socket %d) %s", socket-&gt;native (), e.what()));
          numCharsWritten = 0;
      }
  
      return numCharsWritten;
  }
  </code></pre>
  
  <p>I'm trying to stream a 39 MB, 16 second video file with the following file data (courtesy of MediaInfo):</p>
  
  <pre><code>Video information
  
  General
  Complete name                            : TestVideo.mp4
  Format                                   : MPEG-4
  Format profile                           : Base Media
  Codec ID                                 : isom
  File size                                : 38.6 MiB
  Duration                                 : 16s 102ms
  Overall bit rate                         : 20.1 Mbps
  
  Video
  ID                                       : 1
  Format                                   : AVC
  Format/Info                              : Advanced Video Codec
  Format profile                           : High@L4.0
  Format settings, CABAC                   : Yes
  Format settings, ReFrames                : 1 frame
  Format settings, GOP                     : M=1, N=61
  Codec ID                                 : avc1
  Codec ID/Info                            : Advanced Video Coding
  Duration                                 : 15s 701ms
  Bit rate                                 : 20.0 Mbps
  Width                                    : 1 920 pixels
  Height                                   : 1 080 pixels
  Display aspect ratio                     : 16:9
  Frame rate mode                          : Variable
  Frame rate                               : 30.000 fps
  Minimum frame rate                       : 29.732 fps
  Maximum frame rate                       : 30.313 fps
  Color space                              : YUV
  Chroma subsampling                       : 4:2:0
  Bit depth                                : 8 bits
  Scan type                                : Progressive
  Bits/(Pixel*Frame)                       : 0.322
  Stream size                              : 37.5 MiB (97%)
  Title                                    : VideoHandle
  Language                                 : English
  mdhd_Duration                            : 15701
  
  Audio
  ID                                       : 2
  Format                                   : AAC
  Format/Info                              : Advanced Audio Codec
  Format profile                           : LC
  Codec ID                                 : 40
  Duration                                 : 16s 102ms
  Source duration                          : 16s 131ms
  Bit rate mode                            : Constant
  Bit rate                                 : 192 Kbps
  Nominal bit rate                         : 96.0 Kbps
  Channel(s)                               : 2 channels
  Channel positions                        : Front: L R
  Sampling rate                            : 48.0 KHz
  Compression mode                         : Lossy
  Stream size                              : 374 KiB (1%)
  Source stream size                       : 375 KiB (1%)
  Title                                    : SoundHandle
  Language                                 : English
  mdhd_Duration                            : 16102
  </code></pre>
  
  <p>The StreamFile() function streams blocks of 'streamingBlockSize' bytes in a tight loop to the output socket ('streamingBlockSize' is set through a configuration file, and was introduced while researching and debugging the current problem of buffer underrun).</p>
  
  <p>Tracking the packets using Wireshark shows packets with 1448 bytes of streaming data being sent at an even pace:</p>
  
  <pre><code>|Time     | 192.168.0.197                         |
  |         |                   | 192.168.0.199     |                   
  |14.420722000|         SYN, ACK  |                   |Seq = 0 Ack = 1|         |(10243)  ------------------&gt;  (58358)  |
  |14.437750000|         PSH, ACK - Len: 266           |Seq = 1 Ack = 188|         |(10243)  ------------------&gt;  (58358)  |
  |14.437924000|         ACK - Len: 1448               |Seq = 267 Ack = 188|         |(10243)  ------------------&gt;  (58358)  |
  |14.437939000|         ACK - Len: 1448               |Seq = 1715 Ack = 188|         |(10243)  ------------------&gt;  (58358)  |
  |14.437950000|         ACK - Len: 1448               |Seq = 3163 Ack = 188|         |(10243)  ------------------&gt;  (58358)  |
  |14.442016000|         ACK - Len: 1448               |Seq = 4611 Ack = 188|         |(10243)  ------------------&gt;  (58358)  |
  |14.444269000|         ACK - Len: 1448               |Seq = 6059 Ack = 188|         |(10243)  ------------------&gt;  (58358)  |
  |14.444293000|         ACK - Len: 1448               |Seq = 7507 Ack = 188|         |(10243)  ------------------&gt;  (58358)  |
  |14.444358000|         ACK - Len: 1448               |Seq = 8955 Ack = 188|         |(10243)  ------------------&gt;  (58358)  |
  |14.444373000|         ACK - Len: 1448               |Seq = 10403 Ack = 188|         |(10243)  ------------------&gt;  (58358)  |
  |14.444389000|         ACK - Len: 1448               |Seq = 11851 Ack = 188|         |(10243)  ------------------&gt;  (58358)  |
  . . .
  |72.768739000|         ACK - Len: 1448               |Seq = 40488067 Ack = 188|         |(10243)  ------------------&gt;  (58358)  |
  |72.768766000|         ACK - Len: 1448               |Seq = 40489515 Ack = 188|         |(10243)  ------------------&gt;  (58358)  |
  |72.772484000|         ACK - Len: 1448               |Seq = 40490963 Ack = 188|         |(10243)  ------------------&gt;  (58358)  |
  |72.772521000|         PSH, ACK - Len: 895           |Seq = 40492411 Ack = 188|         |(10243)  ------------------&gt;  (58358) 
  </code></pre>
  
  <p>Wireshark provides very useful summary information about the packets above via the Statistics>Summary menu item:</p>
  
  <pre><code>Packets                        27997
  Between first and last packet  58.352 sec
  Avg. packets/sec               479.797
  Avg. packet size               1513.586 bytes
  Bytes                          42375867
  Avg. bytes/sec                 726213.548
  Avg. MBit/sec                  5.810
  </code></pre>
  
  <p>This tells us it took 58.352 seconds to transfer a 39 MB video that has a playing time of 16.102 seconds and whose renderer encounters frequent stalling.  This sounds like a classic case of buffer underrun.</p>
  
  <p>In addition, the average Mbps rate detected by Wireshark was 5.81 Mbps.  By definition this can never satisfy a renderer that needs to render a video at a bit rate of 20.1 Mbps.</p>
  
  <h2>Possible Fixes</h2>
  
  <p>While researching the problem I've come across numerous technical issues that might contribute to the problem, and would appreciate your thoughts.</p>
  
  <h3>Increase size of buffer passed to write()</h3>
  
  <p>I've tried varying the amount of bytes passed to the write() function (e.g., 4096, 8192, 16384) to see if increasing the data size can speed up the transfer.  It does not seem to make a difference (see discussion of MTU and MSS for possible explanation).</p>
  
  <h3>Increase Ethernet MTU (Maximum Transmission Unit) and/or TCP MSS (Maximum Segment Size)</h3>
  
  <p>Wireshark shows each TCP packet carries 1448 of the video raw data.  Would increasing the MTU or MSS improve streaming throughput?  <a href="http://www.stratus.com/blog/openvos/?p=1459" rel="nofollow">http://www.stratus.com/blog/openvos/?p=1459</a> has an interesting comparison between MTU and MSS.</p>
  
  <h3>TCP_NODELAY</h3>
  
  <p>There are several pages discussing the socket setting TCP_NODELAY (see <a href="http://en.wikipedia.org/wiki/Transmission_Control_Protocol" rel="nofollow">http://en.wikipedia.org/wiki/Transmission_Control_Protocol</a>).  My understanding is that it will improve multiple file transfer, which typically results in the last bytes of a file not filling the output buffer.  By default TCP will wait 200 ms for the buffer to fill up.  With TCP_NODELAY there will be no delay.  In a single video file streaming situation I would not expect an improvement.  Is this correct?</p>
  
  <h3>Network load variability</h3>
  
  <p>Could the network being used cause the data to stream too slowly?</p>
  
  <h3>boost::asio::write() is a blocking write -- would a non-blocking write help?</h3>
  
  <p>The boost::asio::write() at the very bottom is a blocking write:</p>
  
  <pre><code>try
  {
      numCharsWritten = boost::asio::write (socket, boost::asio::buffer (data, len));
  }
  </code></pre>
  
  <p>Is there maybe an intrinsic delay when using a blocking write() as opposed to a non-blocking write()?  Would using a non-blocking write improve throughput?</p>
  
  <p>Many thanks in advance for your help.</p>