title: How to free up all memory pytorch is taken from gpu memory
body: <p>I have some kind of high level code, so model training and etc. are wrapped by <code>pipeline_network</code> class. My main goal is to train new model every new fold.</p>
  
  <pre><code>for train_idx, valid_idx in cv.split(meta_train[DEPTH_COLUMN].values.reshape(-1)):
  
          meta_train_split, meta_valid_split = meta_train.iloc[train_idx], meta_train.iloc[valid_idx]
  
          pipeline_network = unet(config=CONFIG, suffix = 'fold' + str(fold), train_mode=True)
  </code></pre>
  
  <p>But then I move on to 2nd fold everything fails out of gpu memory:</p>
  
  <pre><code>RuntimeError: cuda runtime error (2) : out of memory at /pytorch/torch/lib/THC/generic/THCStorage.cu:58
  </code></pre>
  
  <p>At the end of epoch I tried to manually delete that pipeline with no luck:  </p>
  
  <pre><code> def clean_object_from_memory(obj): #definition
      del obj
      gc.collect()
      torch.cuda.empty_cache()
  
  clean_object_from_memory( clean_object_from_memory) # calling
  </code></pre>
  
  <p>Calling this didn't help as well:</p>
  
  <pre><code>def dump_tensors(gpu_only=True):
          torch.cuda.empty_cache()
          total_size = 0
          for obj in gc.get_objects():
              try:
                  if torch.is_tensor(obj):
                      if not gpu_only or obj.is_cuda:
                          del obj
                          gc.collect()
                  elif hasattr(obj, "data") and torch.is_tensor(obj.data):
                      if not gpu_only or obj.is_cuda:
                          del obj
                          gc.collect()
              except Exception as e:
                  pass
  </code></pre>
  
  <p>How can reset pytorch then I move on to the next fold?</p>