title: pytorch: RuntimeError: CUDA out of memory. with enough GPU memory
body: <p>Torch Error:</p>
  <pre><code>RuntimeError: CUDA out of memory. Tried to allocate 392.00 MiB (GPU 0; 10.73 GiB total capacity; 9.47 GiB already allocated; 347.56 MiB free; 9.51 GiB reserved in total by PyTorch)
  </code></pre>
  <p>I checked GPU resource by nvidia-smi, showing no other running process and memory-usage: 10/10989MiB.</p>
  <pre><code>+-----------------------------------------------------------------------------+
  | NVIDIA-SMI 418.56       Driver Version: 418.56       CUDA Version: 10.1     |
  |-------------------------------+----------------------+----------------------+
  | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
  | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
  |===============================+======================+======================|
  |   0  GeForce RTX 208...  On   | 00000000:04:00.0 Off |                  N/A |
  | 22%   30C    P8    10W / 230W |     10MiB / 10989MiB |      0%      Default |
  +-------------------------------+----------------------+----------------------+
  
  +-----------------------------------------------------------------------------+
  | Processes:                                                       GPU Memory |
  |  GPU       PID   Type   Process name                             Usage      |
  |=============================================================================|
  |  No running processes found                                                 |
  +-----------------------------------------------------------------------------+
  </code></pre>
  <p>I have tried following 3 methods:</p>
  <ol>
  <li>with torch.no_grad();</li>
  <li>torch.cuda.empty_cache();</li>
  <li>reduce batch_size;</li>
  </ol>
  <p>All of them did not work.</p>