title: Replacing av_read_frame() to reduce delay
body: <p>I am implementing a (very) low latency video streaming C++ application using ffmpeg. The client receives a video which is encoded with x264’s zerolatency preset, so there is no need for buffering. As described <a href="https://trac.ffmpeg.org/ticket/3354" rel="nofollow noreferrer">here</a>, if you use <em>av_read_frame</em>() to read packets of the encoded video stream, you will always have at least one frame delay because of internal buffering done in ffmpeg. So when I call <em>av_read_frame</em>() after frame n+1 has been sent to the client, the function will return frame n.</p>
  
  <p>Getting rid of this buffering by setting the AVFormatContext flags AVFMT_FLAG_NOPARSE | AVFMT_FLAG_NOFILLIN as suggested in the <a href="https://trac.ffmpeg.org/ticket/3354" rel="nofollow noreferrer">source</a> disables packet parsing and therefore breaks decoding, as noted in the <a href="https://trac.ffmpeg.org/ticket/3354" rel="nofollow noreferrer">source</a>. </p>
  
  <p>Therefore, I am writing my own packet receiver and parser. First, here are the relevant steps of the working solution (including one frame delay) using <em>av_read_frame</em>():</p>
  
  <pre><code>AVFormatContext *fctx;
  AVCodecContext *cctx;
  AVPacket *pkt;
  AVFrame *frm;
  
  //Initialization of AV structures
  //…
  
  //Main Loop
  while(true){
  
      //Receive packet
      av_read_frame(fctx, pkt);
  
      //Decode:
      avcodec_send_packet(cctx, pkt);
      avcodec_receive_frame(cctx, frm);
  
      //Display frame
      //…
  }
  </code></pre>
  
  <p>And below is my solution, which mimics the behavior of <em>av_read_frame</em>(), as far as I could reproduce it. I was able to track the source code of <a href="https://ffmpeg.org/doxygen/trunk/libavformat_2utils_8c_source.html#l01767" rel="nofollow noreferrer"><em>av_read_frame</em></a>() down to <a href="https://ffmpeg.org/doxygen/trunk/libavformat_2utils_8c_source.html#l00831" rel="nofollow noreferrer"><em>ff_read_packet</em></a>(),but  I cannot find the source of <a href="https://ffmpeg.org/doxygen/trunk/avformat_8h_source.html#l00714" rel="nofollow noreferrer"><em>AVInputformat.read_packet</em></a>().</p>
  
  <pre><code>int tcpsocket;
  AVCodecContext *cctx;
  AVPacket *pkt;
  AVFrame *frm;
  uint8_t recvbuf[(int)10e5];
  memset(recvbuf,0,10e5);
  int pos = 0;
  
  AVCodecParserContext * parser = av_parser_init(AV_CODEC_ID_H264);
  parser-&gt;flags |= PARSER_FLAG_COMPLETE_FRAMES;
  parser-&gt;flags |= PARSER_FLAG_USE_CODEC_TS;
  
  //Initialization of AV structures and the tcpsocket
  //…
  
  //Main Loop
  while(true){
  
      //Receive packet
      int length = read(tcpsocket, recvbuf, 10e5);
      if (length &gt;= 0) {
  
          //Creating temporary packet
          AVPacket * tempPacket = new AVPacket;
          av_init_packet(tempPacket);
          av_new_packet(tempPacket, length);
          memcpy(tempPacket-&gt;data, recvbuf, length);
          tempPacket-&gt;pos = pos;
          pos += length;
          memset(recvbuf,0,length);
  
          //Parsing temporary packet into pkt
          av_init_packet(pkt);
          av_parser_parse2(parser, cctx,
              &amp;(pkt-&gt;data), &amp;(pkt-&gt;size),
              tempPacket-&gt;data, tempPacket-&gt;size,
              tempPacket-&gt;pts, tempPacket-&gt;dts, tempPacket-&gt;pos
              );
  
          pkt-&gt;pts = parser-&gt;pts;
          pkt-&gt;dts = parser-&gt;dts;
          pkt-&gt;pos = parser-&gt;pos;
  
          //Set keyframe flag
          if (parser-&gt;key_frame == 1 ||
              (parser-&gt;key_frame == -1 &amp;&amp;
              parser-&gt;pict_type == AV_PICTURE_TYPE_I))
              pkt-&gt;flags |= AV_PKT_FLAG_KEY;
          if (parser-&gt;key_frame == -1 &amp;&amp; parser-&gt;pict_type == AV_PICTURE_TYPE_NONE &amp;&amp; (pkt-&gt;flags &amp; AV_PKT_FLAG_KEY))
              pkt-&gt;flags |= AV_PKT_FLAG_KEY;
          pkt-&gt;duration = 96000; //Same result as in av_read_frame()
  
          //Decode:
          avcodec_send_packet(cctx, pkt);
          avcodec_receive_frame(cctx, frm);
          //Display frame
          //…
      }
  }
  </code></pre>
  
  <p>I checked the <a href="https://www.ffmpeg.org/doxygen/3.2/structAVPacket.html" rel="nofollow noreferrer">fields</a> of the resulting packet (<em>pkt</em>) just before <em>avcodec_send_packet</em>() in both solutions. They are as far as I can tell identical. The only difference might be the actual content of <em>pkt->data</em>. My solution decodes I-Frames fine, but the references in P-Frames seem to be broken, causing heavy artifacts and error messages such as “invalid level prefix”, “error while decoding MB xx”, and similar.</p>
  
  <p>I would be very grateful for any hints.</p>
  
  <p><strong>Edit 1:</strong> I have developed a workaround for the time being: in the video server, after sending the packet containing the encoded data of a frame, I send one dummy packet which only contains the delimiters marking beginning and end of the packet. This way, I push the actual video data frames through <em>av_read_frame</em>(). I discard the dummy packets immediately after <em>av_frame_read</em>().</p>
  
  <p><strong>Edit 2:</strong> Solved <a href="https://github.com/Genymobile/scrcpy/pull/646" rel="nofollow noreferrer">here</a> by rom1v, as written in his comment to this question.</p>