title: How can I sync video and audio in real time with the primitive Media Foundation APIs?
body: <p>I am trying to sync Audio and Video into a presentation using the <strong>Primitive Media Foundation Interfaces</strong>: <code>IMFSourceReader</code>; <code>EVR</code>; <code>SAR</code>. I've tried some techniques and yet Audio is ahead of Video.</p>
  <p>To summarize the structure of playback of Video and Audio: I have <strong>two buffers</strong> (Audio and Video) that hold 6 samples each, each sample processed, plus one is requested from <code>IMFSourceReader</code>. Each type (Audio or Video) contains its <strong>TASK</strong> for processing buffer samples.
  The presentation clock, according to <strong>MSDN</strong>, is good to use SAR to <strong>improve synchronization</strong>, and that's what I'm doing here.</p>
  <p>Attempt implemented for synchronization: To try to display the video at the correct time, I am performing a delay that I calculated as follows:</p>
  <pre><code>//Calculates the Media Frame Rate. The Numerator and Denominator were obtained from IMFMediaType from the video file loaded locally by IMFSourceReader.
  var FrameRate = Math.Round((Numerator / Denominator)).
  
  //Calculates the delay that will be used to call the next sample. Video processing time is 8 to 11ms.
  var Delay = Math.Round((1000 - Video Sample Processing Time) / FrameRate) - 12.
  </code></pre>
  <p><strong>Video sample processing time</strong> = + or - 10ms.</p>
  <p>The subtracted value from every division (<strong>-12</strong>) is because if I implemented the resulting delay value, the video would crash. In my tests, what is possible to perceive to human eyes, about 13ms, was the best result that I achieved in sync with Audio. It starts well, but it falls behind in relation to Audio.</p>
  <p>I don't have much to show code. But I'll show you how I'm trying to play the media loaded from a local file.</p>
  <h1>Video Processing Task:</h1>
  <pre><code>//Get the next video sample available in the buffer.
  IMFSample VideoSample = BufferVideo.Next();
  
  //Calls the method that will copy the read sample data to the IMFVideoSampleAllocator sample. The sample is accessed for the IMF2DBuffer2 interface as recommended by MSDN.
  ProcessVideoSample(VideoSample);
  
  //I call IMFSourceReader to read the next available video sample. This method is thread safe. The sample reader is in Asynchronous mode.
  RequestNextSample(TypeSample.Video);
  
  //Delay to read the next sample. The delay is calculated as shown above.
  await Task.Delay(DelayFrameRate);
  
  </code></pre>
  <p>In the case of audio, the process is the same, except that it calls the audio buffer.</p>
  <p>The delay time here is a problem, I don't know the time I should implement. In several tests, I noticed that a delay over <strong>16ms</strong> starts to make squeaks in the audio playback. Therefore, the Audio delay I left as <strong>15ms</strong>.</p>
  <p>I have tried to synchronize Audio processing with Video processing, I did not get a good result, the audio crashes too much during playback.</p>
  <h1>1 Update - 08/24/2019</h1>
  <p>I modified the code according to what <strong>VuVirt</strong> gave me, but unfortunately I was unsuccessful at some points.
  Now I can keep Audio synchronized properly, because I'm rendering at the correct rendering time.
  Unfortunately, the video did not follow the same process. It keeps crashing during playback, and as I suspected, if you use a delay it can display acceptably. Which makes me wonder, maybe the EVR isn't delaying what it should.</p>
  <p>presentation time of the samples is correct as follows, then discarded the process that selects the samples.</p>
  <p><img src="https://i.stack.imgur.com/lVbrw.jpg" alt="Image Debug Player" /></p>
  <p>Video samples are processed so quickly by EVR that the sample buffer does not track, in a test, I could not even get the ms time it took for EVR to render the buffer. Speed was below millisecond range</p>
  <p>Below I have the code inside the task that is checking and requesting samples for the buffer:</p>
  <pre><code>//Variable that will contain the Stream type
  StreamType Tp = StreamType.TP_Audio;
  
  //wait for the presentation clock to start.
  while (StatusPresentationClock != EnumStatusClock.Started)
  {
  
  }
  
  // Create the loop that will be checking and ordering the samples.
  while (VideoPlayer.Permission)
  {
  //Checks if the amount of audio buffer samples is below the total size.
  if (BufferSamplesAudio.Count &lt;= (LengthBuffer))
  {
                              //Continue
  }
  else
  {
  //The buffer has the maximum number of samples. Even taking this delay, at various times evr renders faster.
  await Task.Delay(1);
  goto Done;
  }
  
  //Checks if the audio stream has not reached its end
  if(VideoPlayer.EndStreamAudio)
  {
  //The audio stream is over
  SendCon.Send(&quot;Audio Task&quot;, &quot;Stream End!!&quot;);
  
  //Exit
  break;
  }
  
  //Defines that you are reading an audio sample
  AudioRender.ReadingAudioSample = true;
  
  //Calls the method that will request a sample from the reader.
  ResultCode Result = RequestNewSample(Tp);
  
  //Check call result
  if (Result != ResultCode.SS_OK)
  {
  //Call Exception
  throw new Exception(&quot;ERROR&quot;);
  }
                       
  //Waits for the reading process to be completed.
  while (AudioRender.ReadingAudioSample)
  {
                             
  }
                          
  Done:;
  }
  </code></pre>
  <p>Following is the code I used to create the EVR:</p>
  <pre><code>                //Calls the method for creating a Media Sink trigger that will be used to render the video.
                  Result = CarenEVR.CreateInstanceVideoRender(SurfaceVideo.Handle, out VideoRender.AtivadorVideoSink);
  
                  //Checks for no operation error
                  if (Result != ResultCode.SS_OK)
                  {
                      //Exit Method
                      goto Done;
                  }
  
                  //Calls the method to create an instance of EVR.
                  Result = VideoRender.ActivatorVideoSink.ActivateObject(InterfacesMediaFoundation.IID_IMFMediaSink, out ICarenQuery MediaSinkEVR);
  
                  //Checks for no operation error
                  if (Result != ResultCode.SS_OK)
                  {
                      //Exit Method
                      goto Done;
                  }
  
                  //Converts the interface to the MediaSink.
                  VideoRender.VideoSinkRender = ConvertInterfaceBase&lt;ICaren_MidiaSink&gt;(ref MediaSinkEVR);
  
                  //Get interface that represents the real video renderer
                  Result = VideoRender.VideoSinkRender.QueryInterface(InterfacesMediaFoundation.IID_IVideoRenderer, out ICarenQuery OutInterfaceSolictada);
  
                  //Checks for no operation error
                  if (Result != ResultCode.SS_OK)
                  {
                      //Exit Method
                      goto Done;
                  }
  
                  //Convert interface to real video renderer
                  VideoRender.EVR = ConvertInterfaceBase&lt;ICaren_EVR&gt;(ref OutInterfaceSolictada);
  
                  //Initializes the video renderer in standard mode.
                  Result = VideoRender.EVR.InitializeVideoRender(null, null);
  
                  //Checks for no operation error
                  if (Result != ResultCode.SS_OK)
                  {
                      //Exit Method
                      goto Done;
                  }
  
                  //Queries the service interface for the video renderer
                  Result = VideoRender.EVR.QueryInterface(InterfacesMediaFoundation.IID_IMFGetService, out ICarenQuery OutInterfaceServico);
  
                  //Checks for no operation error
                  if (Result != ResultCode.SS_OK)
                  {
                      //Exit Method
                      goto Done;
                  }
  
                  //Converts and defines the interface.
                  VideoRender.ServiceInterface = ConvertInterfaceBase&lt;ICaren_ObterInterface&gt;(ref OutInterfaceServico);
  
                  //Calls the method to get the EVR from the service interface exposed by the video renderer.
                  Result = VideoRender.ServiceInterface.GetService(MFServiceInterfaces.MR_VIDEO_RENDER_SERVICE, InterfacesMediaFoundation.IID_IMFVideoDisplayControl, out ICarenQuery OutInterfaceDisplayControl);
  
                  //Checks for no operation error
                  if (Result != ResultCode.SS_OK)
                  {
                      //Exit Method
                      goto Done;
                  }
  
                  //Converts and define the interface.
                  VideoRender.VideoControler = ConvertInterfaceBase&lt;ICaren_ControladorVideo&gt;(ref OutInterfaceDisplayControl);
  
                  //Defines the surface handle that will display the video frames.
                  Result = VideoRender.VideoControler.SetHandleSurface(SufarceVideo.Handle);
  
                  //Checks for no operation error
                  if (Result != ResultCode.SS_OK)
                  {
                      //Exit Method
                      goto Done;
                  }
  
                  //Sets video window position
                  Result = VideoRender.VideoControler.SetVideoPosition(null, new CA_Rectangle() { Rigth = SurfaceVideo.Width, Bottom = SurfaceVideo.Height });
  
                  //Checks for no operation error
                  if (Result != ResultCode.SS_OK)
                  {
                      //Exit Method
                      goto Done;
                  }
  
                  //Get Direct3D Manager
                  Result = VideoRender.VideoSinkRender.ConsultarServiceInterface(MFServiceInterfaces.MR_VIDEO_ACCELERATION_SERVICE, InterfacesDirectX.IID_IDirect3DDeviceManager9, out ICarenQuery OutInterfaceD3DManager);
  
                  //Checks for no operation error
                  if (Result != ResultCode.SS_OK)
                  {
                      //Exit Method
                      goto Done;
                  }
  
                  //Convert interface
                  VideoRender.Direct3DManager = ConvertInterfaceBase&lt;ICaren_Direct3DDeviceManager9&gt;(ref OutInterfaceD3DManager);
  
  
  
                  //Sets the presentation clock for the media source.
  
                  //Defines the presentation clock that will be used by the player.
                  Result = VideoRender.VideoSinkRender.SetPresentationClock(VideoPlayer.PresentationClock);
  
                  //Checks for no operation error
                  if (Result != ResultCode.SS_OK)
                  {
                      //Exit Method
                      goto Done;
                  }
  
  
  
                  //Configures the Media Stream Sink and the sample allocator.
  
                  //Configures the Media Stream Sink and the sample allocator.
                  Result = VideoRender.VideoSinkRender.GetStreamSinkByIndex(0, out VideoRender.StreamVideo);
  
                  //Checks for no operation error
                  if (Result != ResultCode.SS_OK)
                  {
                      //Exit Method
                      goto Done;
                  }
  
                  //Get media type handler
                  Result = VideoRender.StreamVideo.GetMediaTypeHandle(out VideoRender.MediaTypeHandler);
  
                  //Checks for no operation error
                  if (Result != ResultCode.SS_OK)
                  {
                      //Exit Method
                      goto Done;
                  }
  </code></pre>
  <p>The code for the presentation clock:</p>
  <pre><code> //Create the presentation clock
              ResultCode Result = CarenPresentationClock.CreateIntance(out VideoPlayer.PresentationClock);
  
              //Check Result of Operation
              if (Result != ResultCode.SS_OK)
              {
                  //Get out of the method
                  goto Done;
              }
  
              //Create time source for presentation clock
              Result = CarenPresentationClockSourceTime.CreateIntance(out VideoPlayer.SourcePresentationClock);
  
              //Check Result of Operation
              if (Result != ResultCode.SS_OK)
              {
                  //Get out of the method
                  goto Done;
              }
  
              //Creates the status notification clock for the presentation clock.
              Result = CarenClockStateSink.CreateIntance(out VideoPlayer.NotificationPresentationClock);
  
              //Check Result of Operation
              if (Result != ResultCode.SS_OK)
              {
                  //Get out of the method
                  goto Done;
              }
  
  
              //Configures the clock events.
              VideoPlayer.NotificationPresentationClock.OnClockStart += NotificationPresentationClock_OnClockStart;
              VideoPlayer.NotificationPresentationClock.OnClockPause += NotificationPresentationClock_OnClockPause;
              VideoPlayer.NotificationPresentationClock.OnClockRestart += NotificationPresentationClock_OnClockRestart;
              VideoPlayer.NotificationPresentationClock.OnClockStop += NotificationPresentationClock_OnClockStop;
              VideoPlayer.NotificationPresentationClock.OnClockSetRate += NotificationPresentationClock_OnClockSetRate;
  
              //This method will log the events of the generated Clock in the native interface and send to managed code.
              VideoPlayer.NotificationPresentationClock.RegisterCallback();
  
  
              //Sets the presentation clock.
  
  
              //Sets the clock time source in the presentation clock.
              Result = VideoPlayer.PresentationClock.SetSourceTime(VideoPlayer.SourcePresentationClock);
  
              //Check Result of Operation
              if (Result != ResultCode.SS_OK)
              {
                  //Get out of the method
                  goto Done;
              }
         
              //Sets the notification clock.
              Result = VideoPlayer.PresentationClock.RegisterObjectNotification(VideoPlayer.NotificationPresentationClock);
  
              //Check Result of Operation
              if (Result != ResultCode.SS_OK)
              {
                  //Get out of the method
                  goto Done;
              }
              
          Done:;
              //Return the result
              return Result;
  </code></pre>
  <p>Note: I'm using DXVA2 for decoding acceleration in IMFSourceReader, could this have anything to do with video crashes during playback?</p>
  <h1>2 Update - 08/25/2019</h1>
  <p>The problem with Video crashing was because I was not using <strong>IMFSinkWriter</strong> to render Video samples. Looking at your information, I noticed that it prevents samples from being rendered too fast.
  <strong>But the video is still out of sync with Audio.</strong></p>