title: MediaCodec simultaneous encoding and decoding
body: <p>I am trying to apply effects to the frames of a video using the GPU and then to re-encode those frames into a new result video.</p>
  
  <p>In the interest of performance I have implemented the following flow:</p>
  
  <p>There are 3 different threads, each with it's own OpenGL context. These contexts are set up in such a way that they share textures between them.</p>
  
  <p><strong>Thread 1</strong> extracts frames from the video and holds them in the GPU memory as textures, similar to <strong><a href="http://bigflake.com/mediacodec/ExtractMpegFramesTest_egl14.java.txt" rel="nofollow">this</a></strong> example.</p>
  
  <p><strong>Thread 2</strong> processes the textures using a modified version of GPUImage that also outputs textures in the GPU memory.</p>
  
  <p>Finally, <strong>thread 3</strong> writes the textures obtained from thread 2 into a new video file similar to the method described <strong><a href="https://android.googlesource.com/platform/cts/+/jb-mr2-release/tests/tests/media/src/android/media/cts/EncodeDecodeTest.java" rel="nofollow">here</a></strong></p>
  
  <p>Frame order is maintained using queues between threads 1 and 2, and threads 2 and 3. Textures are deleted from memory manually <strong>after</strong> they are used for processing / writing.</p>
  
  <p>The whole point of this flow is to separate each process in hopes that the final performance will be that of the slowest of the 3 threads.</p>
  
  <p><strong>THE PROBLEM:</strong></p>
  
  <p>The final video is 90% black frames, only some of them being correct. </p>
  
  <p>I have checked the individual results of extraction, and processing and they work as expected. Also note that the 3 components described in the 3 threads work just fine together in a single thread.</p>
  
  <p>I have tried to synchronise thread 1 and thread 3, and after adding an extra 100ms sleep time to thread 1 the video turns out just fine, with maybe 1 or 2 black frames. Seems to me like the two instance of the decoder and encoder are unable to work simultaneously.</p>
  
  <p>I will edit this post with any extra requested details.</p>