title: PyTorch - RuntimeError: transform: failed to synchronize: cudaErrorIllegalAddress
body: <p>I have the problem that when I run my model on Google Colab I very frequently get this error:</p>
  
  <pre><code>RuntimeError: transform: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered
  </code></pre>
  
  <p>I does not happen all the time. Sometimes it works fine but then again it doesn't. <br>
  <em>(If it works or not seems to be related to the batch size, see edit at the bottom.)</em> </p>
  
  <p>When running only on <strong>CPU</strong> there is no problem, so it seems related to <strong>GPU/CUDA</strong>.</p>
  
  <p>The error traceback shows that the error happens at some point during <code>backward()</code>.</p>
  
  <pre><code>---------------------------------------------------------------------------
  RuntimeError                              Traceback (most recent call last)
  &lt;ipython-input-1-c8bbb05fb58a&gt; in &lt;module&gt;()
       65 out   = model(chars, pos)
       66 loss  = F.binary_cross_entropy_with_logits(out, labels)
  ---&gt; 67 loss.backward()
  
  1 frames
  /usr/local/lib/python3.6/dist-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
      164                 products. Defaults to ``False``.
      165         """"""
  --&gt; 166         torch.autograd.backward(self, gradient, retain_graph, create_graph)
      167 
      168     def register_hook(self, hook):
  
  /usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
       97     Variable._execution_engine.run_backward(
       98         tensors, grad_tensors, retain_graph, create_graph,
  ---&gt; 99         allow_unreachable=True)  # allow_unreachable flag
      100 
      101 
  
  RuntimeError: transform: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered
  </code></pre>
  
  <p>Once this error has occurred I cannot do anything related with GPU on that runtime. Whenever I try then to create some tensors or so on GPU then I get a slightly different error message:</p>
  
  <pre><code>RuntimeError: CUDA error: an illegal memory access was encountered
  </code></pre>
  
  <p>I need to restart the runtime/notebook in order to do something on the GPU (with Pytorch haven't tried other frameworks).</p>
  
  <p>Here is a code snippet that should be able to reproduce the issue in Google Colab:</p>
  
  <pre><code>import os
  #os.environ['CUDA_LAUNCH_BLOCKING'] = '1'
  
  import torch
  device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
  print(torch.__version__, torch.cuda.get_device_name(0))
  from torch import nn
  from torch.nn import functional as F
  class MyModel(nn.Module):
      def __init__(self, char_vocab, num_pos, dim, hidden, dropout):
          super().__init__()
          self.emb_char  = nn.Embedding(char_vocab, dim)
          self.cnn1 = nn.Conv1d(dim, dim // 2, kernel_size=3)
          self.bn1  = nn.BatchNorm1d(dim // 2)
          self.cnn2 = nn.Conv1d(dim // 2, dim // 4, kernel_size=2)
          self.bn2  = nn.BatchNorm1d(dim // 4)
          self.pooling = nn.MaxPool1d(2)
          self.lin1 = nn.Linear(11, hidden)
          self.bn3  = nn.BatchNorm1d(hidden)
          self.lin2 = nn.Linear(hidden, dim // 4)
          self.bn4  = nn.BatchNorm1d(dim // 4)
  
  
  
          self.out  = nn.Linear(dim // 4, 1)
          self.drop = nn.Dropout(dropout)
  
      def forward(self, chars, pos):
          x = self.emb_char(chars).transpose(1, 2)
          x = self.drop(x)
  
          x = self.cnn1(x)
          x = self.bn1(x)
          x = self.pooling(x)
          x = F.relu(x)
  
          x = self.cnn2(x)
          x = self.bn2(x)
          x = self.pooling(x).squeeze(-1)
          x = F.relu(x)
          x = self.drop(x)
  
  
          y = F.relu(self.lin1(pos))
          y = self.bn3(y)
          y = self.drop(y)
          y = F.relu(self.lin2(y))
          y = self.bn4(y)
          y = self.drop(y)
  
          return self.out(x+y)
  
  model = MyModel(char_vocab=80, num_pos=11, dim=32, hidden=8, dropout=0.5).to(device)
  
  batch_size = 160000
  
  # inputs
  chars = torch.randint(0, 79, (batch_size, 8), device=device)
  pos   = torch.rand(batch_size, 11, device=device)
  
  # labels
  labels = torch.ones(batch_size, 1, dtype=torch.float, device=device)
  
  # forward
  out   = model(chars, pos)
  loss  = F.binary_cross_entropy_with_logits(out, labels)
  
  # backward
  loss.backward()
  </code></pre>
  
  <p>Pytorch version is <code>1.3.1</code> the GPU it was running on was <code>P100-PCIE-16GB</code>.</p>
  
  <p><strong>Any ideas how to get rid of this error?</strong></p>
  
  <hr>
  
  <p><strong>Edit:</strong></p>
  
  <ul>
  <li><p>I added a piece of code with generated input that can reproduce the issue.</p></li>
  <li><p>It seems to be related to the high batch size I'm using. The batch size of <code>160000</code> I'm using in the code seems to reliably cause this error, however I noticed that it does not solely depend on the batch size. I tried to reduce it down to <code>80000</code> and still got this error. But I know that in my previous runs I have had higher batch sizes up to <code>180000</code> without problems. Also starting with a batch size of <code>60000</code> worked, then using the same runtime I could run it again with an increased batch size of <code>140000</code> without error (that failed before). <br><em>Note that the memory usage is very low / only ~1-2 GB</em></p></li>
  <li><p>Setting <code>CUDA_LAUNCH_BLOCKING=1</code> leads to a different error message:</p>
  
  <pre><code>RuntimeError: fractional_max_pool2d_backward_out_cuda failed with error code 0
  </code></pre>
  
  <p>But at the same point in code and with same traceback.</p></li>
  </ul>