title: Compute PTS and DTS correctly to sync audio and video ffmpeg C++
body: <p>I am trying to mux H264 encoded data and G711 PCM data into <code>mov</code> multimedia container. I am creating <code>AVPacket</code> from encoded data and initially the PTS and DTS value of video/audio frames is equivalent to <code>AV_NOPTS_VALUE</code>. So I calculated the DTS using current time information. My code - </p>
  
  <pre><code>bool AudioVideoRecorder::WriteVideo(const unsigned char *pData, size_t iDataSize, bool const bIFrame) {
      .....................................
      .....................................
      .....................................
      AVPacket pkt = {0};
      av_init_packet(&amp;pkt);
      int64_t dts = av_gettime();
      dts = av_rescale_q(dts, (AVRational){1, 1000000}, m_pVideoStream-&gt;time_base);
      int duration = 90000 / VIDEO_FRAME_RATE;
      if(m_prevVideoDts &gt; 0LL) {
          duration = dts - m_prevVideoDts;
      }
      m_prevVideoDts = dts;
  
      pkt.pts = AV_NOPTS_VALUE;
      pkt.dts = m_currVideoDts;
      m_currVideoDts += duration;
      pkt.duration = duration;
      if(bIFrame) {
          pkt.flags |= AV_PKT_FLAG_KEY;
      }
      pkt.stream_index = m_pVideoStream-&gt;index;
      pkt.data = (uint8_t*) pData;
      pkt.size = iDataSize;
  
      int ret = av_interleaved_write_frame(m_pFormatCtx, &amp;pkt);
  
      if(ret &lt; 0) {
          LogErr("Writing video frame failed.");
          return false;
      }
  
      Log("Writing video frame done.");
  
      av_free_packet(&amp;pkt);
      return true;
  }
  
  bool AudioVideoRecorder::WriteAudio(const unsigned char *pEncodedData, size_t iDataSize) {
      .................................
      .................................
      .................................
      AVPacket pkt = {0};
      av_init_packet(&amp;pkt);
  
      int64_t dts = av_gettime();
      dts = av_rescale_q(dts, (AVRational){1, 1000000}, (AVRational){1, 90000});
      int duration = AUDIO_STREAM_DURATION; // 20
      if(m_prevAudioDts &gt; 0LL) {
          duration = dts - m_prevAudioDts;
      }
      m_prevAudioDts = dts;
      pkt.pts = AV_NOPTS_VALUE;
      pkt.dts = m_currAudioDts;
      m_currAudioDts += duration;
      pkt.duration = duration;
  
      pkt.stream_index = m_pAudioStream-&gt;index;
      pkt.flags |= AV_PKT_FLAG_KEY;
      pkt.data = (uint8_t*) pEncodedData;
      pkt.size = iDataSize;
  
      int ret = av_interleaved_write_frame(m_pFormatCtx, &amp;pkt);
      if(ret &lt; 0) {
          LogErr("Writing audio frame failed: %d", ret);
          return false;
      }
  
      Log("Writing audio frame done.");
  
      av_free_packet(&amp;pkt);
      return true;
  }
  </code></pre>
  
  <p>And I added stream like this -</p>
  
  <pre><code>AVStream* AudioVideoRecorder::AddMediaStream(enum AVCodecID codecID) {
      ................................
      .................................   
      pStream = avformat_new_stream(m_pFormatCtx, codec);
      if (!pStream) {
          LogErr("Could not allocate stream.");
          return NULL;
      }
      pStream-&gt;id = m_pFormatCtx-&gt;nb_streams - 1;
      pCodecCtx = pStream-&gt;codec;
      pCodecCtx-&gt;codec_id = codecID;
  
      switch(codec-&gt;type) {
      case AVMEDIA_TYPE_VIDEO:
          pCodecCtx-&gt;bit_rate = VIDEO_BIT_RATE;
          pCodecCtx-&gt;width = PICTURE_WIDTH;
          pCodecCtx-&gt;height = PICTURE_HEIGHT;
          pStream-&gt;time_base = (AVRational){1, 90000};
          pStream-&gt;avg_frame_rate = (AVRational){90000, 1};
          pStream-&gt;r_frame_rate = (AVRational){90000, 1}; // though the frame rate is variable and around 15 fps
          pCodecCtx-&gt;pix_fmt = STREAM_PIX_FMT;
          m_pVideoStream = pStream;
          break;
  
      case AVMEDIA_TYPE_AUDIO:
          pCodecCtx-&gt;sample_fmt = AV_SAMPLE_FMT_S16;
          pCodecCtx-&gt;bit_rate = AUDIO_BIT_RATE;
          pCodecCtx-&gt;sample_rate = AUDIO_SAMPLE_RATE;
          pCodecCtx-&gt;channels = 1;
          m_pAudioStream = pStream;
          break;
  
      default:
          break;
      }
  
      /* Some formats want stream headers to be separate. */
      if (m_pOutputFmt-&gt;flags &amp; AVFMT_GLOBALHEADER)
          m_pFormatCtx-&gt;flags |= CODEC_FLAG_GLOBAL_HEADER;
  
      return pStream;
  }
  </code></pre>
  
  <p>There are several problems with this calculation:</p>
  
  <ol>
  <li><p>The video is laggy and lags behind than audio increasingly with time.</p></li>
  <li><p>Suppose, an audio frame is received (<code>WriteAudio(..)</code>) little lately like 3 seconds, then the late frame should be started playing with 3 second delay, but it's not. The delayed frame is played consecutively with previous frame.</p></li>
  <li><p>Sometimes I recorded for ~40 seconds but the file duration is much like 2 minutes, but audio/video is played only few moments like 40 seconds and rest of the file contains nothing and seekbar jumps at en immediately after 40 seconds (tested in VLC).</p></li>
  </ol>
  
  <p>EDIT:</p>
  
  <p>According to Ronald S. Bultje's suggestion, what I've understand:</p>
  
  <pre><code>m_pAudioStream-&gt;time_base = (AVRational){1, 9000}; // actually no need to set as 9000 is already default value for audio as you said
  m_pVideoStream-&gt;time_base = (AVRational){1, 9000};
  </code></pre>
  
  <p>should be set as now both audio and video streams are now in same time base units.</p>
  
  <p>And for video:</p>
  
  <pre><code>...................
  ...................
  
  int64_t dts = av_gettime(); // get current time in microseconds
  dts *= 9000; 
  dts /= 1000000; // 1 second = 10^6 microseconds
  pkt.pts = AV_NOPTS_VALUE; // is it okay?
  pkt.dts = dts;
  // and no need to set pkt.duration, right?
  </code></pre>
  
  <p>And for audio: (exactly same as video, right?)</p>
  
  <pre><code>...................
  ...................
  
  int64_t dts = av_gettime(); // get current time in microseconds
  dts *= 9000; 
  dts /= 1000000; // 1 second = 10^6 microseconds
  pkt.pts = AV_NOPTS_VALUE; // is it okay?
  pkt.dts = dts;
  // and no need to set pkt.duration, right?
  </code></pre>
  
  <p>And I think they are now like sharing same <code>currDts</code>, right? Please correct me if I am wrong anywhere or missing anything.</p>
  
  <p>Also, if I want to use video stream time base as <code>(AVRational){1, frameRate}</code> and audio stream time base as <code>(AVRational){1, sampleRate}</code>, how the correct code should look like?</p>
  
  <p>EDIT 2.0:</p>
  
  <pre><code>m_pAudioStream-&gt;time_base = (AVRational){1, VIDEO_FRAME_RATE};
  m_pVideoStream-&gt;time_base = (AVRational){1, VIDEO_FRAME_RATE};
  </code></pre>
  
  <p>And </p>
  
  <pre><code>bool AudioVideoRecorder::WriteAudio(const unsigned char *pEncodedData, size_t iDataSize) {
      ...........................
      ......................
      AVPacket pkt = {0};
      av_init_packet(&amp;pkt);
  
      int64_t dts = av_gettime() / 1000; // convert into millisecond
      dts = dts * VIDEO_FRAME_RATE;
      if(m_dtsOffset &lt; 0) {
          m_dtsOffset = dts;
      }
  
      pkt.pts = AV_NOPTS_VALUE;
      pkt.dts = (dts - m_dtsOffset);
  
      pkt.stream_index = m_pAudioStream-&gt;index;
      pkt.flags |= AV_PKT_FLAG_KEY;
      pkt.data = (uint8_t*) pEncodedData;
      pkt.size = iDataSize;
  
      int ret = av_interleaved_write_frame(m_pFormatCtx, &amp;pkt);
      if(ret &lt; 0) {
          LogErr("Writing audio frame failed: %d", ret);
          return false;
      }
  
      Log("Writing audio frame done.");
  
      av_free_packet(&amp;pkt);
      return true;
  }
  
  bool AudioVideoRecorder::WriteVideo(const unsigned char *pData, size_t iDataSize, bool const bIFrame) {
      ........................................
      .................................
      AVPacket pkt = {0};
      av_init_packet(&amp;pkt);
      int64_t dts = av_gettime() / 1000;
      dts = dts * VIDEO_FRAME_RATE;
      if(m_dtsOffset &lt; 0) {
          m_dtsOffset = dts;
      }
      pkt.pts = AV_NOPTS_VALUE;
      pkt.dts = (dts - m_dtsOffset);
  
      if(bIFrame) {
          pkt.flags |= AV_PKT_FLAG_KEY;
      }
      pkt.stream_index = m_pVideoStream-&gt;index;
      pkt.data = (uint8_t*) pData;
      pkt.size = iDataSize;
  
      int ret = av_interleaved_write_frame(m_pFormatCtx, &amp;pkt);
  
      if(ret &lt; 0) {
          LogErr("Writing video frame failed.");
          return false;
      }
  
      Log("Writing video frame done.");
  
      av_free_packet(&amp;pkt);
      return true;
  }
  </code></pre>
  
  <p>Is the last change okay? The video and audio seems synced. Only problem is - the audio is played without the delay regardless the packet arrived in delay.
  Like - </p>
  
  <p>packet arrival: 1 2 3 4... (then next frame arrived after 3 sec) .. 5</p>
  
  <p>audio played: 1 2 3 4 (no delay) 5</p>
  
  <p>EDIT 3.0:</p>
  
  <p>zeroed audio sample data:</p>
  
  <pre><code>AVFrame* pSilentData;
  pSilentData = av_frame_alloc();
  memset(&amp;pSilentData-&gt;data[0], 0, iDataSize);
  
  pkt.data = (uint8_t*) pSilentData;
  pkt.size = iDataSize;
  
  av_freep(&amp;pSilentData-&gt;data[0]);
  av_frame_free(&amp;pSilentData);
  </code></pre>
  
  <p>Is this okay? But after writing this into file container, there are <strong><em>dot dot</em></strong> noise during playing the media. Whats the problem?</p>
  
  <p>EDIT 4.0:</p>
  
  <p>Well, <a href="https://stackoverflow.com/a/32007095/1162233">For µ-Law audio the zero value is represented as <code>0xff</code></a>. So -</p>
  
  <pre><code>memset(&amp;pSilentData-&gt;data[0], 0xff, iDataSize);
  </code></pre>
  
  <p>solve my problem.</p>