title: Using GPUImage with AVVideoCompositing
body: <p>I'm trying to use a combination of <code>GPUImage</code> and the <code>AVVideoCompositing</code> to implement a live chroma-key filter between two videos. Doing this naively using <code>CIImage</code> <code>imageFromCVPixelBuffer</code> into <code>CGImage</code> into <code>GPUImage</code> out to <code>CGImage</code> to <code>CIImage</code> to <code>CVPixelBuffer</code>, is highly inefficient and leads to memory problems. </p>
  
  <p>I noticed there are texture objects, render targets, and frame buffers in the <code>GPUImage</code> Framework. I was hoping to be able to leverage <code>CVOpenGLESTextureCacheCreateTextureFromImage</code> in iOS to keep everything on the GPU. </p>
  
  <p>I don't think I'm quite understanding the framework's inner workings, because I assumed I could set up a filter chain on a <code>GPUImageTextureInput</code> object and then get the filter's <code>renderTarget</code>, which is a <code>CVPixelBufferRef</code>. The <code>renderTarget</code> below is always nil, and calling <code>imageFromCurrentFrameBuffer</code> will yield me a gray box, which is not my image.</p>
  
  <p>Please note that the below example is not chroma-key, but a simple brightness filter on a single video to try and prove the concept.</p>
  
  <pre><code>@implementation MyCustomCompositor : NSObject &lt;AVVideoCompositing&gt;
  
  - (instancetype)init
  {
      self = [super init];
      if (self) {
          CVOpenGLESTextureCacheCreate(kCFAllocatorDefault, NULL, [GPUImageContext sharedImageProcessingContext].context, NULL, &amp;_textureCache);
      }
      return self;
  }
  
  - (NSDictionary&lt;NSString *,id&gt; *)requiredPixelBufferAttributesForRenderContext
  {
      return @{(NSString *)kCVPixelBufferPixelFormatTypeKey : @[@(kCVPixelFormatType_32BGRA)],
               (NSString *)kCVPixelBufferOpenGLCompatibilityKey : @YES};
  }
  
  - (NSDictionary&lt;NSString *,id&gt; *)sourcePixelBufferAttributes
  {
      return @{(NSString *)kCVPixelBufferPixelFormatTypeKey : @[@(kCVPixelFormatType_32BGRA)],
               (NSString *)kCVPixelBufferOpenGLCompatibilityKey : @YES};
  }
  
  - (void)startVideoCompositionRequest:(AVAsynchronousVideoCompositionRequest *)asyncVideoCompositionRequest
  {
      @autoreleasepool {
          CVPixelBufferRef mePixelBuffer = [asyncVideoCompositionRequest sourceFrameByTrackID:200];
          CVPixelBufferLockBaseAddress(mePixelBuffer, kCVPixelBufferLock_ReadOnly);
  
          CVOpenGLESTextureRef meTextureRef = NULL;
          size_t width = CVPixelBufferGetWidth(mePixelBuffer);
          size_t height = CVPixelBufferGetHeight(mePixelBuffer);
          CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault, _textureCache, mePixelBuffer, NULL, GL_TEXTURE_2D, GL_BGRA, (int)width, (int)height, GL_BGRA, GL_UNSIGNED_BYTE, 0, &amp;meTextureRef);
  
          GPUImageTextureInput *meTextureInput = [[GPUImageTextureInput alloc] initWithTexture:CVOpenGLESTextureGetName(meTextureRef) size:CGSizeMake(width, height)];
  
          GPUImageBrightnessFilter *filter = [[GPUImageBrightnessFilter alloc] init];
          filter.brightness = 0.5;
          [meTextureInput addTarget:filter];
  
          [filter setFrameProcessingCompletionBlock:^(GPUImageOutput *imageOutput, CMTime time) {
              [asyncVideoCompositionRequest finishWithComposedVideoFrame:((GPUImageBrightnessFilter *)imageOutput).renderTarget];
          }];
  
          [meTextureInput processTextureWithFrameTime:kCMTimeZero];
  
          CFRelease(meTextureRef);
          CVOpenGLESTextureCacheFlush(_textureCache, 0);
  
          CVPixelBufferUnlockBaseAddress(mePixelBuffer, kCVPixelBufferLock_ReadOnly);
      }
  }
  </code></pre>
  
  <p>I am not using the <code>GPUMovieWriter</code> or video APIs in <code>GPUImage</code> because I need more granular control of my composition. The composition can be made up of multiple chroma-key instructions that reference a different green video overlay at different time ranges, and it seems to me that the movie APIs in <code>GPUImage</code> are limited to filtering an entire video file. I also need the composition's abilities to manipulate tracks and audio mixes.</p>
  
  <p>I've tried wrapping my head around doing all of this in GL with custom shaders, but I figured I'd leverage existing frameworks that seem to do what I'm trying to do.</p>