title: PyTorch Model Training: RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR
body: <p>After training a PyTorch model on a GPU for several hours, the program fails with the error</p>
  
  <blockquote>
    <p>RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR</p>
  </blockquote>
  
  <p><strong>Training Conditions</strong></p>
  
  <ul>
  <li>Neural Network: PyTorch 4-layer <code>nn.LSTM</code> with <code>nn.Linear</code> output</li>
  <li>Deep Q Network Agent (Vanilla DQN with Replay Memory)</li>
  <li><code>state</code> passed into <code>forward()</code> has the shape <code>(32, 20, 15)</code>, where <code>32</code> is the batch size</li>
  <li>50 seconds per episode</li>
  <li>Error occurs after about 583 episodes (8 hours) or 1,150,000 steps, where each step involves a forward pass through the LSTM model.</li>
  </ul>
  
  <p>My code also has the following values set before the training began</p>
  
  <pre><code>torch.manual_seed(0)
  torch.backends.cudnn.deterministic = True
  torch.backends.cudnn.benchmark = False
  np.random.seed(0)
  </code></pre>
  
  <p>How can we troubleshoot this problem? Since this occurred 8 hours into the training, some educated guess will be very helpful here!</p>
  
  <p>Thanks!</p>
  
  <hr>
  
  <p><strong>Update:</strong></p>
  
  <p>Commenting out the 2 <code>torch.backends.cudnn...</code> lines did not work. <code>CUDNN_STATUS_INTERNAL_ERROR</code> still occurs, but much earlier at around Episode 300 (585,000 steps).</p>
  
  <pre><code>torch.manual_seed(0)
  #torch.backends.cudnn.deterministic = True
  #torch.backends.cudnn.benchmark = False
  np.random.seed(0)
  </code></pre>
  
  <hr>
  
  <p><strong>System</strong></p>
  
  <ul>
  <li>PyTorch 1.6.0.dev20200525</li>
  <li>CUDA 10.2</li>
  <li>cuDNN 7604</li>
  <li>Python 3.8</li>
  <li>Windows 10</li>
  <li>nVidia 1080 GPU</li>
  </ul>
  
  <p><strong>Error Traceback</strong></p>
  
  <pre><code>RuntimeError                              Traceback (most recent call last)
  &lt;ipython-input-18-f5bbb4fdfda5&gt; in &lt;module&gt;
       57 
       58     while not done:
  ---&gt; 59         action = agent.choose_action(state)
       60         state_, reward, done, info = env.step(action)
       61         score += reward
  
  &lt;ipython-input-11-5ad4dd57b5ad&gt; in choose_action(self, state)
       58         if np.random.random() &gt; self.epsilon:
       59             state = T.tensor([state], dtype=T.float).to(self.q_eval.device)
  ---&gt; 60             actions = self.q_eval.forward(state)
       61             action = T.argmax(actions).item()
       62         else:
  
  &lt;ipython-input-10-94271a92f66e&gt; in forward(self, state)
       20 
       21     def forward(self, state):
  ---&gt; 22         lstm, hidden = self.lstm(state)
       23         actions = self.fc1(lstm[:,-1:].squeeze(1))
       24         return actions
  
  ~\AppData\Local\Continuum\anaconda3\envs\rl\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
      575             result = self._slow_forward(*input, **kwargs)
      576         else:
  --&gt; 577             result = self.forward(*input, **kwargs)
      578         for hook in self._forward_hooks.values():
      579             hook_result = hook(self, input, result)
  
  ~\AppData\Local\Continuum\anaconda3\envs\rl\lib\site-packages\torch\nn\modules\rnn.py in forward(self, input, hx)
      571         self.check_forward_args(input, hx, batch_sizes)
      572         if batch_sizes is None:
  --&gt; 573             result = _VF.lstm(input, hx, self._flat_weights, self.bias, self.num_layers,
      574                               self.dropout, self.training, self.bidirectional, self.batch_first)
      575         else:
  
  RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR
  </code></pre>
  
  <p>Update: Tried <code>try... except</code> on my code where this error occurs at, and in addition to <code>RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR</code>, we also get a second traceback for the error <code>RuntimeError: CUDA error: unspecified launch failure</code></p>
  
  <pre><code>During handling of the above exception, another exception occurred:
  
  RuntimeError                              Traceback (most recent call last)
  &lt;ipython-input-4-e8f15cc8cf4f&gt; in &lt;module&gt;
       61 
       62     while not done:
  ---&gt; 63         action = agent.choose_action(state)
       64         state_, reward, done, info = env.step(action)
       65         score += reward
  
  &lt;ipython-input-3-1aae79080e99&gt; in choose_action(self, state)
       58         if np.random.random() &gt; self.epsilon:
       59             state = T.tensor([state], dtype=T.float).to(self.q_eval.device)
  ---&gt; 60             actions = self.q_eval.forward(state)
       61             action = T.argmax(actions).item()
       62         else:
  
  &lt;ipython-input-2-6d22bb632c4c&gt; in forward(self, state)
       25         except Exception as e:
       26             print('error in forward() with state:', state.shape, 'exception:', e)
  ---&gt; 27             print('state:', state)
       28         actions = self.fc1(lstm[:,-1:].squeeze(1))
       29         return actions
  
  ~\AppData\Local\Continuum\anaconda3\envs\rl\lib\site-packages\torch\tensor.py in __repr__(self)
      152     def __repr__(self):
      153         # All strings are unicode in Python 3.
  --&gt; 154         return torch._tensor_str._str(self)
      155 
      156     def backward(self, gradient=None, retain_graph=None, create_graph=False):
  
  ~\AppData\Local\Continuum\anaconda3\envs\rl\lib\site-packages\torch\_tensor_str.py in _str(self)
      331                 tensor_str = _tensor_str(self.to_dense(), indent)
      332             else:
  --&gt; 333                 tensor_str = _tensor_str(self, indent)
      334 
      335     if self.layout != torch.strided:
  
  ~\AppData\Local\Continuum\anaconda3\envs\rl\lib\site-packages\torch\_tensor_str.py in _tensor_str(self, indent)
      227     if self.dtype is torch.float16 or self.dtype is torch.bfloat16:
      228         self = self.float()
  --&gt; 229     formatter = _Formatter(get_summarized_data(self) if summarize else self)
      230     return _tensor_str_with_formatter(self, indent, formatter, summarize)
      231 
  
  ~\AppData\Local\Continuum\anaconda3\envs\rl\lib\site-packages\torch\_tensor_str.py in __init__(self, tensor)
       99 
      100         else:
  --&gt; 101             nonzero_finite_vals = torch.masked_select(tensor_view, torch.isfinite(tensor_view) &amp; tensor_view.ne(0))
      102 
      103             if nonzero_finite_vals.numel() == 0:
  
  RuntimeError: CUDA error: unspecified launch failure
  </code></pre>