<p dir="auto">when the batch size is 1024, my model work fine, but when I change it to 1025, it throw error</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="RuntimeError: CUDNN_STATUS_NOT_SUPPORTED"><pre class="notranslate"><span class="pl-v">RuntimeError</span>: <span class="pl-v">CUDNN_STATUS_NOT_SUPPORTED</span></pre></div>
<p dir="auto">this is how I load data:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="val_dataloader=t.utils.data.DataLoader(val_dataset,1025,True,num_workers=8,\
        collate_fn=my_collate)"><pre class="notranslate"><span class="pl-s1">val_dataloader</span><span class="pl-c1">=</span><span class="pl-s1">t</span>.<span class="pl-s1">utils</span>.<span class="pl-s1">data</span>.<span class="pl-v">DataLoader</span>(<span class="pl-s1">val_dataset</span>,<span class="pl-c1">1025</span>,<span class="pl-c1">True</span>,<span class="pl-s1">num_workers</span><span class="pl-c1">=</span><span class="pl-c1">8</span>,\
        <span class="pl-s1">collate_fn</span><span class="pl-c1">=</span><span class="pl-s1">my_collate</span>)</pre></div>
<p dir="auto">the full trace is here:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&lt;ipython-input-4-91b240474ca9&gt; in forward(self, input)
     21         self.model.add_module('conv5',nn.Conv2d(opt.ndf*8,1,4,1,0,bias=False))
     22     def forward(self,input):
---&gt; 23         return self.model(input)
     24         gpuids=None
     25         if self.ngpu:

/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.pyc in __call__(self, *input, **kwargs)
    208 
    209     def __call__(self, *input, **kwargs):
--&gt; 210         result = self.forward(*input, **kwargs)
    211         for hook in self._forward_hooks.values():
    212             hook_result = hook(self, input, result)

/usr/local/lib/python2.7/dist-packages/torch/nn/modules/container.pyc in forward(self, input)
     61     def forward(self, input):
     62         for module in self._modules.values():
---&gt; 63             input = module(input)
     64         return input

/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.pyc in __call__(self, *input, **kwargs)
    208 
    209     def __call__(self, *input, **kwargs):
--&gt; 210         result = self.forward(*input, **kwargs)
    211         for hook in self._forward_hooks.values():
    212             hook_result = hook(self, input, result)

/usr/local/lib/python2.7/dist-packages/torch/nn/modules/batchnorm.pyc in forward(self, input)
     41         return F.batch_norm(
     42             input, self.running_mean, self.running_var, self.weight, self.bias,
---&gt; 43             self.training, self.momentum, self.eps)
     44 
     45     def __repr__(self):

/usr/local/lib/python2.7/dist-packages/torch/nn/functional.pyc in batch_norm(input, running_mean, running_var, weight, bias, training, momentum, eps)
    483     state = _functions.batchnorm.BatchNorm(
    484         running_mean, running_var, training, momentum, eps)
--&gt; 485     return weight and state(input, weight, bias) or state(input)
    486 
    487 

/usr/local/lib/python2.7/dist-packages/torch/nn/_functions/batchnorm.pyc in forward(self, input, weight, bias)
     34                 input, output, weight, bias,
     35                 self.running_mean, self.running_var, self._save_mean,
---&gt; 36                 self._save_std, self.training, self.momentum, self.eps)
     37         else:
     38             backend = type2backend[type(input)]

RuntimeError: CUDNN_STATUS_NOT_SUPPORTED
"><pre class="notranslate"><span class="pl-c1">&lt;</span><span class="pl-s1">ipython</span><span class="pl-c1">-</span><span class="pl-s1">input</span><span class="pl-c1">-</span><span class="pl-c1">4</span><span class="pl-c1">-</span><span class="pl-c1">91</span><span class="pl-s1">b240474ca9</span><span class="pl-c1">&gt;</span> <span class="pl-s1">in</span> <span class="pl-en">forward</span>(<span class="pl-s1">self</span>, <span class="pl-s1">input</span>)
     <span class="pl-c1">21</span>         <span class="pl-s1">self</span>.<span class="pl-s1">model</span>.<span class="pl-en">add_module</span>(<span class="pl-s">'conv5'</span>,<span class="pl-s1">nn</span>.<span class="pl-v">Conv2d</span>(<span class="pl-s1">opt</span>.<span class="pl-s1">ndf</span><span class="pl-c1">*</span><span class="pl-c1">8</span>,<span class="pl-c1">1</span>,<span class="pl-c1">4</span>,<span class="pl-c1">1</span>,<span class="pl-c1">0</span>,<span class="pl-s1">bias</span><span class="pl-c1">=</span><span class="pl-c1">False</span>))
     <span class="pl-c1">22</span>     <span class="pl-k">def</span> <span class="pl-en">forward</span>(<span class="pl-s1">self</span>,<span class="pl-s1">input</span>):
<span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">&gt;</span> <span class="pl-c1">23</span>         <span class="pl-s1">return</span> <span class="pl-s1">self</span>.<span class="pl-en">model</span>(<span class="pl-s1">input</span>)
     <span class="pl-c1">24</span>         <span class="pl-s1">gpuids</span><span class="pl-c1">=</span><span class="pl-c1">None</span>
     <span class="pl-c1">25</span>         <span class="pl-k">if</span> <span class="pl-s1">self</span>.<span class="pl-s1">ngpu</span>:

<span class="pl-c1">/</span><span class="pl-s1">usr</span><span class="pl-c1">/</span><span class="pl-s1">local</span><span class="pl-c1">/</span><span class="pl-s1">lib</span><span class="pl-c1">/</span><span class="pl-s1">python2</span>.<span class="pl-c1">7</span><span class="pl-c1">/</span><span class="pl-s1">dist</span><span class="pl-c1">-</span><span class="pl-s1">packages</span><span class="pl-c1">/</span><span class="pl-s1">torch</span><span class="pl-c1">/</span><span class="pl-s1">nn</span><span class="pl-c1">/</span><span class="pl-s1">modules</span><span class="pl-c1">/</span><span class="pl-s1">module</span>.<span class="pl-s1">pyc</span> <span class="pl-c1">in</span> <span class="pl-en">__call__</span>(<span class="pl-s1">self</span>, <span class="pl-c1">*</span><span class="pl-s1">input</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>)
    <span class="pl-c1">208</span> 
    <span class="pl-c1">209</span>     <span class="pl-k">def</span> <span class="pl-en">__call__</span>(<span class="pl-s1">self</span>, <span class="pl-c1">*</span><span class="pl-s1">input</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>):
<span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">&gt;</span> <span class="pl-c1">210</span>         <span class="pl-s1">result</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-en">forward</span>(<span class="pl-c1">*</span><span class="pl-s1">input</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>)
    <span class="pl-c1">211</span>         <span class="pl-s1">for</span> <span class="pl-s1">hook</span> <span class="pl-c1">in</span> <span class="pl-s1">self</span>.<span class="pl-s1">_forward_hooks</span>.<span class="pl-en">values</span>():
    <span class="pl-c1">212</span>             <span class="pl-s1">hook_result</span> <span class="pl-c1">=</span> <span class="pl-en">hook</span>(<span class="pl-s1">self</span>, <span class="pl-s1">input</span>, <span class="pl-s1">result</span>)

<span class="pl-c1">/</span><span class="pl-s1">usr</span><span class="pl-c1">/</span><span class="pl-s1">local</span><span class="pl-c1">/</span><span class="pl-s1">lib</span><span class="pl-c1">/</span><span class="pl-s1">python2</span>.<span class="pl-c1">7</span><span class="pl-c1">/</span><span class="pl-s1">dist</span><span class="pl-c1">-</span><span class="pl-s1">packages</span><span class="pl-c1">/</span><span class="pl-s1">torch</span><span class="pl-c1">/</span><span class="pl-s1">nn</span><span class="pl-c1">/</span><span class="pl-s1">modules</span><span class="pl-c1">/</span><span class="pl-s1">container</span>.<span class="pl-s1">pyc</span> <span class="pl-c1">in</span> <span class="pl-en">forward</span>(<span class="pl-s1">self</span>, <span class="pl-s1">input</span>)
     <span class="pl-c1">61</span>     <span class="pl-s1">def</span> <span class="pl-en">forward</span>(<span class="pl-s1">self</span>, <span class="pl-s1">input</span>):
     <span class="pl-c1">62</span>         <span class="pl-s1">for</span> <span class="pl-s1">module</span> <span class="pl-c1">in</span> <span class="pl-s1">self</span>.<span class="pl-s1">_modules</span>.<span class="pl-en">values</span>():
<span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">&gt;</span> <span class="pl-c1">63</span>             <span class="pl-s1">input</span> <span class="pl-c1">=</span> <span class="pl-en">module</span>(<span class="pl-s1">input</span>)
     <span class="pl-c1">64</span>         <span class="pl-s1">return</span> <span class="pl-s1">input</span>

<span class="pl-c1">/</span><span class="pl-s1">usr</span><span class="pl-c1">/</span><span class="pl-s1">local</span><span class="pl-c1">/</span><span class="pl-s1">lib</span><span class="pl-c1">/</span><span class="pl-s1">python2</span>.<span class="pl-c1">7</span><span class="pl-c1">/</span><span class="pl-s1">dist</span><span class="pl-c1">-</span><span class="pl-s1">packages</span><span class="pl-c1">/</span><span class="pl-s1">torch</span><span class="pl-c1">/</span><span class="pl-s1">nn</span><span class="pl-c1">/</span><span class="pl-s1">modules</span><span class="pl-c1">/</span><span class="pl-s1">module</span>.<span class="pl-s1">pyc</span> <span class="pl-c1">in</span> <span class="pl-en">__call__</span>(<span class="pl-s1">self</span>, <span class="pl-c1">*</span><span class="pl-s1">input</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>)
    <span class="pl-c1">208</span> 
    <span class="pl-c1">209</span>     <span class="pl-k">def</span> <span class="pl-en">__call__</span>(<span class="pl-s1">self</span>, <span class="pl-c1">*</span><span class="pl-s1">input</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>):
<span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">&gt;</span> <span class="pl-c1">210</span>         <span class="pl-s1">result</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-en">forward</span>(<span class="pl-c1">*</span><span class="pl-s1">input</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>)
    <span class="pl-c1">211</span>         <span class="pl-s1">for</span> <span class="pl-s1">hook</span> <span class="pl-c1">in</span> <span class="pl-s1">self</span>.<span class="pl-s1">_forward_hooks</span>.<span class="pl-en">values</span>():
    <span class="pl-c1">212</span>             <span class="pl-s1">hook_result</span> <span class="pl-c1">=</span> <span class="pl-en">hook</span>(<span class="pl-s1">self</span>, <span class="pl-s1">input</span>, <span class="pl-s1">result</span>)

<span class="pl-c1">/</span><span class="pl-s1">usr</span><span class="pl-c1">/</span><span class="pl-s1">local</span><span class="pl-c1">/</span><span class="pl-s1">lib</span><span class="pl-c1">/</span><span class="pl-s1">python2</span>.<span class="pl-c1">7</span><span class="pl-c1">/</span><span class="pl-s1">dist</span><span class="pl-c1">-</span><span class="pl-s1">packages</span><span class="pl-c1">/</span><span class="pl-s1">torch</span><span class="pl-c1">/</span><span class="pl-s1">nn</span><span class="pl-c1">/</span><span class="pl-s1">modules</span><span class="pl-c1">/</span><span class="pl-s1">batchnorm</span>.<span class="pl-s1">pyc</span> <span class="pl-c1">in</span> <span class="pl-en">forward</span>(<span class="pl-s1">self</span>, <span class="pl-s1">input</span>)
     <span class="pl-c1">41</span>         <span class="pl-s1">return</span> <span class="pl-v">F</span>.<span class="pl-en">batch_norm</span>(
     <span class="pl-c1">42</span>             <span class="pl-s1">input</span>, <span class="pl-s1">self</span>.<span class="pl-s1">running_mean</span>, <span class="pl-s1">self</span>.<span class="pl-s1">running_var</span>, <span class="pl-s1">self</span>.<span class="pl-s1">weight</span>, <span class="pl-s1">self</span>.<span class="pl-s1">bias</span>,
<span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">&gt;</span> <span class="pl-c1">43</span>             <span class="pl-s1">self</span>.<span class="pl-s1">training</span>, <span class="pl-s1">self</span>.<span class="pl-s1">momentum</span>, <span class="pl-s1">self</span>.<span class="pl-s1">eps</span>)
     <span class="pl-c1">44</span> 
     <span class="pl-c1">45</span>     <span class="pl-s1">def</span> <span class="pl-en">__repr__</span>(<span class="pl-s1">self</span>):

<span class="pl-c1">/</span><span class="pl-s1">usr</span><span class="pl-c1">/</span><span class="pl-s1">local</span><span class="pl-c1">/</span><span class="pl-s1">lib</span><span class="pl-c1">/</span><span class="pl-s1">python2</span>.<span class="pl-c1">7</span><span class="pl-c1">/</span><span class="pl-s1">dist</span><span class="pl-c1">-</span><span class="pl-s1">packages</span><span class="pl-c1">/</span><span class="pl-s1">torch</span><span class="pl-c1">/</span><span class="pl-s1">nn</span><span class="pl-c1">/</span><span class="pl-s1">functional</span>.<span class="pl-s1">pyc</span> <span class="pl-c1">in</span> <span class="pl-en">batch_norm</span>(<span class="pl-s1">input</span>, <span class="pl-s1">running_mean</span>, <span class="pl-s1">running_var</span>, <span class="pl-s1">weight</span>, <span class="pl-s1">bias</span>, <span class="pl-s1">training</span>, <span class="pl-s1">momentum</span>, <span class="pl-s1">eps</span>)
    <span class="pl-c1">483</span>     <span class="pl-s1">state</span> <span class="pl-c1">=</span> <span class="pl-s1">_functions</span>.<span class="pl-s1">batchnorm</span>.<span class="pl-v">BatchNorm</span>(
    <span class="pl-c1">484</span>         <span class="pl-s1">running_mean</span>, <span class="pl-s1">running_var</span>, <span class="pl-s1">training</span>, <span class="pl-s1">momentum</span>, <span class="pl-s1">eps</span>)
<span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">&gt;</span> <span class="pl-c1">485</span>     <span class="pl-s1">return</span> <span class="pl-s1">weight</span> <span class="pl-c1">and</span> <span class="pl-en">state</span>(<span class="pl-s1">input</span>, <span class="pl-s1">weight</span>, <span class="pl-s1">bias</span>) <span class="pl-c1">or</span> <span class="pl-en">state</span>(<span class="pl-s1">input</span>)
    <span class="pl-c1">486</span> 
    <span class="pl-c1">487</span> 

<span class="pl-c1">/</span><span class="pl-s1">usr</span><span class="pl-c1">/</span><span class="pl-s1">local</span><span class="pl-c1">/</span><span class="pl-s1">lib</span><span class="pl-c1">/</span><span class="pl-s1">python2</span>.<span class="pl-c1">7</span><span class="pl-c1">/</span><span class="pl-s1">dist</span><span class="pl-c1">-</span><span class="pl-s1">packages</span><span class="pl-c1">/</span><span class="pl-s1">torch</span><span class="pl-c1">/</span><span class="pl-s1">nn</span><span class="pl-c1">/</span><span class="pl-s1">_functions</span><span class="pl-c1">/</span><span class="pl-s1">batchnorm</span>.<span class="pl-s1">pyc</span> <span class="pl-c1">in</span> <span class="pl-en">forward</span>(<span class="pl-s1">self</span>, <span class="pl-s1">input</span>, <span class="pl-s1">weight</span>, <span class="pl-s1">bias</span>)
     <span class="pl-c1">34</span>                 <span class="pl-s1">input</span>, <span class="pl-s1">output</span>, <span class="pl-s1">weight</span>, <span class="pl-s1">bias</span>,
     <span class="pl-c1">35</span>                 <span class="pl-s1">self</span>.<span class="pl-s1">running_mean</span>, <span class="pl-s1">self</span>.<span class="pl-s1">running_var</span>, <span class="pl-s1">self</span>.<span class="pl-s1">_save_mean</span>,
<span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-&gt;</span> <span class="pl-c1">36</span>                 <span class="pl-s1">self</span>.<span class="pl-s1">_save_std</span>, <span class="pl-s1">self</span>.<span class="pl-s1">training</span>, <span class="pl-s1">self</span>.<span class="pl-s1">momentum</span>, <span class="pl-s1">self</span>.<span class="pl-s1">eps</span>)
     <span class="pl-c1">37</span>         <span class="pl-k">else</span>:
     <span class="pl-c1">38</span>             <span class="pl-s1">backend</span> <span class="pl-c1">=</span> <span class="pl-s1">type2backend</span>[<span class="pl-en">type</span>(<span class="pl-s1">input</span>)]

<span class="pl-v">RuntimeError</span>: <span class="pl-v">CUDNN_STATUS_NOT_SUPPORTED</span></pre></div>
<p dir="auto">Fatal CUDNN_NOT_SUPPORTED error. Reported by <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/namdar/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/namdar">@namdar</a></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import torch
import torch.nn as nn
from torch.autograd import Variable
class TestModel(nn.Module):
    def __init__(self,inp_size):
        super(TestModel, self).__init__()
        self.encoder = nn.Sequential(
            nn.Linear(inp_size, 200),
            nn.BatchNorm1d(200),
        )
    def forward(self, x):
        x = self.encoder(x)
        return x
batch_size = 10000
inp_size = 1
x = Variable(torch.randn(batch_size,inp_size)).cuda()
m = TestModel(inp_size)
m.cuda()
out = m(x)
print(out.size())
m.eval()
out2 = m(x)"><pre class="notranslate"><code class="notranslate">import torch
import torch.nn as nn
from torch.autograd import Variable
class TestModel(nn.Module):
    def __init__(self,inp_size):
        super(TestModel, self).__init__()
        self.encoder = nn.Sequential(
            nn.Linear(inp_size, 200),
            nn.BatchNorm1d(200),
        )
    def forward(self, x):
        x = self.encoder(x)
        return x
batch_size = 10000
inp_size = 1
x = Variable(torch.randn(batch_size,inp_size)).cuda()
m = TestModel(inp_size)
m.cuda()
out = m(x)
print(out.size())
m.eval()
out2 = m(x)
</code></pre></div>