title: Memory exhausted while processing big size video using tensorflow
body: <p>I am using a simple semantic segmentation network where I am using pre-trained models to predict the classes on the images. Everything works well on images and even small videos. However, the video file which I want to pass through the network and output a processed video is around 900MB. Whenever I try to use it, it can also process around 200 frames and then everything crashes with following error:</p>
  <pre><code>(0) Resource exhausted:  OOM when allocating tensor with shape[327635100] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc    
           [[{{node StatefulPartitionedCall/StatefulPartitionedCall/while/body/_628/while/cond_1/else/_924/while/cond_1/norm/Sum}}]]
  Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
  </code></pre>
  <p>I am using opencv to read the video:</p>
  <pre><code>cap = cv2.VideoCapture(filename)
  
  # Define the codec and create VideoWriter object
  fourcc = cv2.VideoWriter_fourcc(*'MP42')
  out = cv2.VideoWriter('output.avi', fourcc, 20.0, (1656, 949))
  count = 0
  while(cap.isOpened()):
      ret, image = cap.read()
      # plt.imshow(image)
      if ret:
        print(image.shape)
        plt.imshow(image)
        image = cv2.resize(image, (1656, 949))
        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)  
        print(image.shape)
        plt.imshow(image)
      
        output = LOADED_MODEL(tf.cast(image, tf.uint8))
        panoptic_map, used_colors = color_panoptic_map(output['panoptic_pred'][0], DATASET_INFO, 60)
        dst = cv2.addWeighted(image, 0.5, panoptic_map, 1,1)
        print('Read a new frame: ', ret)
        count += 1
        #plt.imshow(dst)
        dst = cv2.resize(dst, (1656, 949))
        out.write(dst)
      
       
  out.release()
  cap.release()
  </code></pre>
  <p>On small size videos, everything works perfectly. When I check the GPU memory usage, its definitely exhausted. I have 1080 Ti with 8GB dedicated memory, in addition to 32GB RAM.</p>
  <p>Is there anyway, I can process this video by saving parts on the disk and merge them later? Or any other way which can avoid memory overflow?</p>
  <p>Thanks!</p>
  <p>EDIT:
  Currently, I am trying to force the video processing logic to cpu instead of gpu.
  Forcing to cpu using <code>with tf.device(&quot;/cpu:0&quot;):</code> also doesnt work. I still get the same error:</p>
  <pre><code>2021-09-08 09:44:02.367723: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 2772297216 totalling 2.58GiB
  2021-09-08 09:44:02.367855: I tensorflow/core/common_runtime/bfc_allocator.cc:1058] Sum Total of in-use chunks: 3.99GiB
  2021-09-08 09:44:02.368003: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] total_region_allocated_bytes_: 6930300928 memory_limit_: 6930300928 available bytes: 0 curr_region_allocation_bytes_: 13860601856
  2021-09-08 09:44:02.368153: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Stats:
  Limit:                      6930300928
  InUse:                      4285491712
  MaxInUse:                   6300585472
  NumAllocs:                      618063
  MaxAllocSize:               3192342016
  Reserved:                            0
  PeakReserved:                        0
  LargestFreeBlock:                    0
  
  2021-09-08 09:44:02.369071: W tensorflow/core/common_runtime/bfc_allocator.cc:467] **************__**********************************************________________****__________________
  2021-09-08 09:44:02.369234: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at reduction_ops_common.h:186 : Resource exhausted: OOM when allocating tensor with shape[346537125] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
  Traceback (most recent call last):
  </code></pre>