title: RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle)` with GPU only
body: <p>I'm working on the CNN with one-dimensional signal. It works totally fine with CPU device. However, when I training model in GPU, CUDA error occurred. I set <code>os.environ['CUDA_LAUNCH_BLOCKING'] = &quot;1&quot;</code> command after I got <code>RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED</code> when calling <code>cublasCreate(handle)</code>. With doing this, a <code>cublasSgemm</code> error occurred instead of <code>cublasCreate</code> error.
  Though the nvidia document doubt the hardware problem, I can training other CNN with images without any error. Below is my code for the data loading and set data in training model.</p>
  <pre><code>    idx = np.arange(len(dataset))  # dataset &amp; label shuffle in once
      np.random.shuffle(idx)
  
      dataset = dataset[idx]
      sdnn = np.array(sdnn)[idx.astype(int)]        
  
      train_data, val_data = dataset[:int(0.8 * len(dataset))], dataset[int(0.8 * len(dataset)):]
      train_label, val_label = sdnn[:int(0.8 * len(sdnn))], sdnn[int(0.8 * len(sdnn)):]
      train_set = DataLoader(dataset=train_data, batch_size=opt.batch_size, num_workers=opt.workers)
  
      for i, data in enumerate(train_set, 0):  # data.shape = [batch_size, 3000(len(signal)), 1(channel)] tensor
  
          x = data.transpose(1, 2)
          label = torch.Tensor(train_label[i * opt.batch_size:i * opt.batch_size + opt.batch_size])
          x = x.to(device, non_blocking=True)
          label = label.to(device, non_blocking=True) # [batch size]
          label = label.view([len(label), 1])
          optim.zero_grad()
  
          # Feature of signal extract
          y_predict = model(x) # [batch size, fc3 output] # Error occurred HERE
          loss = mse(y_predict, label)
  </code></pre>
  <p>Below is the error message from this code.</p>
  <pre><code>File C:/Users/Me/Desktop/Me/Study/Project/Analysis/Regression/main.py&quot;, line 217, in Processing
      y_predict = model(x) # [batch size, fc3 output]
    File &quot;C:\Anaconda\envs\torch\lib\site-packages\torch\nn\modules\module.py&quot;, line 722, in _call_impl
      result = self.forward(*input, **kwargs)
    File &quot;C:\Users\ME\Desktop\ME\Study\Project\Analysis\Regression\cnn.py&quot;, line 104, in forward
      x = self.fc1(x)
    File &quot;C:\Anaconda\envs\torch\lib\site-packages\torch\nn\modules\module.py&quot;, line 722, in _call_impl
      result = self.forward(*input, **kwargs)
    File &quot;C:\Anaconda\envs\torch\lib\site-packages\torch\nn\modules\linear.py&quot;, line 91, in forward
      return F.linear(input, self.weight, self.bias)
    File &quot;C:\Anaconda\envs\torch\lib\site-packages\torch\nn\functional.py&quot;, line 1674, in linear
      ret = torch.addmm(bias, input, weight.t())
  RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &amp;alpha, a, lda, b, ldb, &amp;beta, c, ldc)`
  </code></pre>
  <p>I've tried to solve this error for weeks but can't find the solution. If you can see anything wrong here, please let me know.</p>