title: Slow buffering when streaming multiple remote videos with AVPlayer and AVMutableComposition
body: <p><strong>[Edit: I was able to figure out a workaround for this, see below.]</strong></p>
  
  <p>I'm trying to stream multiple remote MP4 clips from S3, playing them in a sequence as one continuous video (to enable scrubbing within and between clips) with no stuttering, without explicitly downloading them to the device first. However, I find that the clips buffer very slowly (even on a fast network connection) and have been unable to find an adequate way to address that.</p>
  
  <p>I've been trying to use <code>AVPlayer</code> for this, since <code>AVPlayer</code> with <code>AVMutableComposition</code> plays the supplied video tracks as one continuous track (unlike <code>AVQueuePlayer</code>, which I gather plays each video separately and thus doesn't support continuous scrubbing between the clips).</p>
  
  <p>When I stick one of the assets directly into an <code>AVPlayerItem</code> and play that (with no <code>AVMutableComposition</code>), it buffers fast. But using <code>AVMutableComposition</code>, the video starts stuttering very badly on the second clip (my test case has 6 clips, each around 6 seconds), while the audio keeps going. After it plays through once, it plays perfectly smoothly if I rewind to the beginning, so I assume the problem lies in the buffering.</p>
  
  <p>My current attempt to fix this problem feels convoluted, given that this seems like a rather basic use-case for <code>AVPlayer</code> - I do hope there's a simpler solution to all this that works properly. Somehow I doubt that the buffering player I use below is really necessary, but I'm running out of ideas.</p>
  
  <p>Here's the main code that sets up the <code>AVMutableComposition</code>:</p>
  
  <pre><code>// Build an AVAsset for each of the source URIs
  - (void)prepareAssetsForSources:(NSArray *)sources
  {
    NSMutableArray *assets = [[NSMutableArray alloc] init]; // the assets to be used in the AVMutableComposition
    NSMutableArray *offsets = [[NSMutableArray alloc] init]; // for tracking buffering progress
    CMTime currentOffset = kCMTimeZero;
    for (NSDictionary* source in sources) {
      bool isNetwork = [RCTConvert BOOL:[source objectForKey:@"isNetwork"]];
      bool isAsset = [RCTConvert BOOL:[source objectForKey:@"isAsset"]];
      NSString *uri = [source objectForKey:@"uri"];
      NSString *type = [source objectForKey:@"type"];
  
      NSURL *url = isNetwork ?
        [NSURL URLWithString:uri] :
        [[NSURL alloc] initFileURLWithPath:[[NSBundle mainBundle] pathForResource:uri ofType:type]];
  
      AVURLAsset *asset = [AVURLAsset URLAssetWithURL:url options:nil];
      currentOffset = CMTimeAdd(currentOffset, asset.duration);
  
      [assets addObject:asset];
      [offsets addObject:[NSNumber numberWithFloat:CMTimeGetSeconds(currentOffset)]];
    }
    _clipAssets = assets;
    _clipEndOffsets = offsets;
  }
  
  // Called with _clipAssets
  - (AVPlayerItem*)playerItemForAssets:(NSMutableArray *)assets
  {
    AVMutableComposition* composition = [AVMutableComposition composition];
    for (AVAsset* asset in assets) {
      CMTimeRange editRange = CMTimeRangeMake(CMTimeMake(0, 600), asset.duration);
      NSError *editError;
  
      [composition insertTimeRange:editRange
                           ofAsset:asset
                            atTime:composition.duration
                             error:&amp;editError];
    }
    AVPlayerItem* playerItem = [AVPlayerItem playerItemWithAsset:composition];
    return playerItem; // this is used to initialize the main player
  }
  </code></pre>
  
  <p>My initial thought was: Since it buffers fast with a vanilla <code>AVPlayerItem</code>, why not maintain a separate buffering player that's loaded with each asset in turn (with no <code>AVMutableComposition</code>) to buffer the assets for the main player?</p>
  
  <pre><code>- (void)startBufferingClips
  {
    _bufferingPlayerItem = [AVPlayerItem playerItemWithAsset:_clipAssets[0] 
                                automaticallyLoadedAssetKeys:@[@"tracks"]];
    _bufferingPlayer = [AVPlayer playerWithPlayerItem:_bufferingPlayerItem];
    _currentlyBufferingIndex = 0;
  }
  
  // called every 250 msecs via an addPeriodicTimeObserverForInterval on the main player
  - (void)updateBufferingProgress
  {
    // If the playable (loaded) range is within 100 milliseconds of the clip
    // currently being buffered, load the next clip into the buffering player.
    float playableDuration = [[self calculateBufferedDuration] floatValue];
    CMTime totalDurationTime = [self playerItemDuration :_bufferingPlayer];
    Float64 totalDurationSeconds = CMTimeGetSeconds(totalDurationTime);
    bool bufferingComplete = totalDurationSeconds - playableDuration &lt; 0.1;
    float bufferedSeconds = [self bufferedSeconds :playableDuration];
  
    float playerTimeSeconds = CMTimeGetSeconds([_player currentTime]);
    __block NSUInteger playingClipIndex = 0;
  
    // find the index of _player's currently playing clip
    [_clipEndOffsets enumerateObjectsUsingBlock:^(id offset, NSUInteger idx, BOOL *stop) {
      if (playerTimeSeconds &lt; [offset floatValue]) {
        playingClipIndex = idx;
        *stop = YES;
      }
    }];
  
    // TODO: if  bufferedSeconds - playerTimeSeconds &lt;= 0, pause the main player
  
    if (bufferingComplete &amp;&amp; _currentlyBufferingIndex &lt; [_clipAssets count] - 1) {
      // We're done buffering this clip, load the buffering player with the next asset
      _currentlyBufferingIndex += 1;
      _bufferingPlayerItem = [AVPlayerItem playerItemWithAsset:_clipAssets[_currentlyBufferingIndex]
                                  automaticallyLoadedAssetKeys:@[@"tracks"]];
      _bufferingPlayer = [AVPlayer playerWithPlayerItem:_bufferingPlayerItem];
    }
  }
  
  - (float)bufferedSeconds:(float)playableDuration {
    __block float seconds = 0.0; // total duration of clips already buffered
    if (_currentlyBufferingIndex &gt; 0) {
      [_clipEndOffsets enumerateObjectsUsingBlock:^(id offset, NSUInteger idx, BOOL *stop) {
        if (idx + 1 &gt;= _currentlyBufferingIndex) {
          seconds = [offset floatValue];
          *stop = YES;
        }
      }];
    }
    return seconds + playableDuration;
  }
  
  - (NSNumber *)calculateBufferedDuration {
    AVPlayerItem *video = _bufferingPlayer.currentItem;
    if (video.status == AVPlayerItemStatusReadyToPlay) {
      __block float longestPlayableRangeSeconds;
      [video.loadedTimeRanges enumerateObjectsUsingBlock:^(id obj, NSUInteger idx, BOOL *stop) {
        CMTimeRange timeRange = [obj CMTimeRangeValue];
        float seconds = CMTimeGetSeconds(CMTimeRangeGetEnd(timeRange));
        if (seconds &gt; 0.1) {
          if (!longestPlayableRangeSeconds) {
            longestPlayableRangeSeconds = seconds;
          } else if (seconds &gt; longestPlayableRangeSeconds) {
            longestPlayableRangeSeconds = seconds;
          }
        }
      }];
      Float64 playableDuration = longestPlayableRangeSeconds;
      if (playableDuration &amp;&amp; playableDuration &gt; 0) {
        return [NSNumber numberWithFloat:longestPlayableRangeSeconds];
      }
    }
    return [NSNumber numberWithInteger:0];
  }
  </code></pre>
  
  <p>It initially seemed that this worked like a charm, but then I switched to another set of test clips and then the buffering was very slow again (the buffering player helped, but not enough). It seems like the <code>loadedTimeRange</code>s for the assets as loaded into the buffering player didn't match the <code>loadedTimeRange</code>s for the same assets inside the <code>AVMutableComposition</code>: Even after the <code>loadedTimeRange</code>s for each item loaded into the buffering player indicated that the whole asset had been buffered, the main player's video continued stuttering (while the audio played seamlessly to the end). Again, the playback was seamless after rewinding once the main player had run through all the clips once.</p>
  
  <p>I hope the answer to this, whatever it is, will prove useful as a starting point for other iOS developers trying to implement this basic use-case. Thanks!</p>
  
  <p><strong>Edit: Since I posted this question, I made the following workaround for this. Hopefully this will save whoever runs into this some headache.</strong></p>
  
  <p>What I ended up doing was maintaining <em>two</em> buffering players (both <code>AVPlayer</code>s) that started buffering the first two clips, moving on to the lowest-indexed unbuffered clip after their <code>loadedTimeRanges</code> indicated that buffering for their current clip was done. I made the logic pause/unpause playback based on the clips currently buffered, and the <code>loadedTimeRanges</code> of the buffering players, plus a small margin. This needed a few bookkeeping variables, but wasn't too complicated.</p>
  
  <p>This is how the buffering players were initialized (I'm omitting the bookkeeping logic here):</p>
  
  <pre><code>- (void)startBufferingClips
  {
    _bufferingPlayerItemA = [AVPlayerItem playerItemWithAsset:_clipAssets[0] 
                                 automaticallyLoadedAssetKeys:@[@"tracks"]];
    _bufferingPlayerA = [AVPlayer playerWithPlayerItem:_bufferingPlayerItemA];
    _currentlyBufferingIndexA = [NSNumber numberWithInt:0];
    if ([_clipAssets count] &gt; 1) {
      _bufferingPlayerItemB = [AVPlayerItem playerItemWithAsset:_clipAssets[1] 
                                   automaticallyLoadedAssetKeys:@[@"tracks"]];
      _bufferingPlayerB = [AVPlayer playerWithPlayerItem:_bufferingPlayerItemB];
      _currentlyBufferingIndexB = [NSNumber numberWithInt:1];
      _nextIndexToBuffer = [NSNumber numberWithInt:2];
    } else {
      _nextIndexToBuffer = [NSNumber numberWithInt:1];
    }
  }
  </code></pre>
  
  <p>In addition, I needed to make sure that the video and audio tracks weren't being merged as they were added to <code>AVMutableComposition</code>, as this apparently interfered with the buffering (perhaps they didn't register as the same video/audio tracks as those the buffering players were loading, and thus didn't receive the new data). Here's the code where the <code>AVMutableComposition</code> is built from an array of <code>NSAsset</code>s:</p>
  
  <pre><code>- (AVPlayerItem*)playerItemForAssets:(NSMutableArray *)assets
  {
    AVMutableComposition* composition = [AVMutableComposition composition];
    AVMutableCompositionTrack *compVideoTrack = [composition addMutableTrackWithMediaType:AVMediaTypeVideo
                                                                     preferredTrackID:kCMPersistentTrackID_Invalid];
    AVMutableCompositionTrack *compAudioTrack = [composition addMutableTrackWithMediaType:AVMediaTypeAudio
                                                                     preferredTrackID:kCMPersistentTrackID_Invalid];
    CMTime timeOffset = kCMTimeZero;
    for (AVAsset* asset in assets) {
      CMTimeRange editRange = CMTimeRangeMake(CMTimeMake(0, 600), asset.duration);
      NSError *editError;
  
      NSArray *videoTracks = [asset tracksWithMediaType:AVMediaTypeVideo];
      NSArray *audioTracks = [asset tracksWithMediaType:AVMediaTypeAudio];
  
      if ([videoTracks count] &gt; 0) {
      AVAssetTrack *videoTrack = [videoTracks objectAtIndex:0];
        [compVideoTrack insertTimeRange:editRange
                                ofTrack:videoTrack
                                 atTime:timeOffset
                                  error:&amp;editError];
      }
  
      if ([audioTracks count] &gt; 0) {
        AVAssetTrack *audioTrack = [audioTracks objectAtIndex:0];
        [compAudioTrack insertTimeRange:editRange
                                ofTrack:audioTrack
                                 atTime:timeOffset
                                  error:&amp;editError];
      }
  
      if ([videoTracks count] &gt; 0 || [audioTracks count] &gt; 0) {
        timeOffset = CMTimeAdd(timeOffset, asset.duration);
      }
    }
    AVPlayerItem* playerItem = [AVPlayerItem playerItemWithAsset:composition];
    return playerItem;
  }
  </code></pre>
  
  <p>With this approach, buffering while using <code>AVMutableComposition</code> for the main player works nice and fast, at least in my setup.</p>