<h2 dir="auto">Issue description</h2>
<p dir="auto">As it illustrates in the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.BatchNorm1d" rel="nofollow">doc</a>, <code class="notranslate">torch.nn.Batchnorm1d</code> supports both input of size <code class="notranslate">(N, C, L)</code> and <code class="notranslate">(N, C)</code> . Howevere, when the input size is <code class="notranslate">(1, C)</code>(batch size is 1), <code class="notranslate">pytorch</code> will produce an error. It seems that if <code class="notranslate">N &gt; 1</code>, there will be no error at all.</p>
<p dir="auto">Another interesting fact is, when I try to seperate a minibatch into multi cards(by using <code class="notranslate">torch.nn.DataParallel</code>), since the batch size for each gpu is not decided by me, this error may also happens with <code class="notranslate">N &gt; 1</code>(for a specific gpu, its batch size may be <code class="notranslate">1</code> under some circumstances). I know currently <code class="notranslate">PyTorch</code> does not support distributed batchnorm, but it still worth a report, I think.</p>
<h2 dir="auto">Code example</h2>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import torch

class Foo(torch.nn.Module):
    def __init__(self):
        super(Foo, self).__init__()
        self.bn = torch.nn.BatchNorm1d(3)
    
    def forward(self, x):
        return self.bn(x)

def main():
    x1 = torch.ones(1,3)
    x2 = torch.ones(2,3)
    model = Foo()
    y = model(x1) # this line will fail to run 
    y = model(x2) # this line can run without error

if __name__ == '__main__':
    main()	
"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">torch</span>

<span class="pl-k">class</span> <span class="pl-v">Foo</span>(<span class="pl-s1">torch</span>.<span class="pl-s1">nn</span>.<span class="pl-v">Module</span>):
    <span class="pl-k">def</span> <span class="pl-en">__init__</span>(<span class="pl-s1">self</span>):
        <span class="pl-en">super</span>(<span class="pl-v">Foo</span>, <span class="pl-s1">self</span>).<span class="pl-en">__init__</span>()
        <span class="pl-s1">self</span>.<span class="pl-s1">bn</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-s1">nn</span>.<span class="pl-v">BatchNorm1d</span>(<span class="pl-c1">3</span>)
    
    <span class="pl-k">def</span> <span class="pl-en">forward</span>(<span class="pl-s1">self</span>, <span class="pl-s1">x</span>):
        <span class="pl-k">return</span> <span class="pl-s1">self</span>.<span class="pl-en">bn</span>(<span class="pl-s1">x</span>)

<span class="pl-k">def</span> <span class="pl-en">main</span>():
    <span class="pl-s1">x1</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">ones</span>(<span class="pl-c1">1</span>,<span class="pl-c1">3</span>)
    <span class="pl-s1">x2</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">ones</span>(<span class="pl-c1">2</span>,<span class="pl-c1">3</span>)
    <span class="pl-s1">model</span> <span class="pl-c1">=</span> <span class="pl-v">Foo</span>()
    <span class="pl-s1">y</span> <span class="pl-c1">=</span> <span class="pl-en">model</span>(<span class="pl-s1">x1</span>) <span class="pl-c"># this line will fail to run </span>
    <span class="pl-s1">y</span> <span class="pl-c1">=</span> <span class="pl-en">model</span>(<span class="pl-s1">x2</span>) <span class="pl-c"># this line can run without error</span>

<span class="pl-k">if</span> <span class="pl-s1">__name__</span> <span class="pl-c1">==</span> <span class="pl-s">'__main__'</span>:
    <span class="pl-en">main</span>()	</pre></div>
<h2 dir="auto">System Info</h2>
<p dir="auto">PyTorch version: 0.4.0<br>
Is debug build: No<br>
CUDA used to build PyTorch: 9.1.85</p>
<p dir="auto">OS: Debian GNU/Linux unstable (sid)<br>
GCC version: (Debian 6.4.0-17) 6.4.0 20180424<br>
CMake version: version 3.11.1</p>
<p dir="auto">Python version: 3.6<br>
Is CUDA available: Yes<br>
CUDA runtime version: 9.1.85<br>
GPU models and configuration: GPU 0: GeForce GTX 1070<br>
Nvidia driver version: 390.48<br>
cuDNN version: Probably one of the following:<br>
/usr/lib/cuda/lib64/libcudnn.so.7.1.3<br>
/usr/lib/cuda/lib64/libcudnn_static.a</p>
<p dir="auto">Versions of relevant libraries:<br>
[pip3] numpy (1.14.2)<br>
[pip3] torch (0.4.0)<br>
[pip3] torchvision (0.2.0)<br>
[conda] Could not collect</p>
<p dir="auto">I find in some tasks , for example, semantic segmentation, detection,  sync batch norm is crucial for performance.In these tasks, batch size per gpu is small so we need sync the bn's mean and variance   to enlarge the bn's "batch size". It need synchronize parameters and gradient across multi gpus in forward and backward. This will obviously improve the performance. I believe this will attract more and more users. So can you implement it in pytorch?</p>