title: Matching PyTorch w/ CNTK (VGG on CIFAR)
body: <p>I am trying to understand how PyTorch works and want to replicate a simple CNN training on CIFAR. The <a href="https://github.com/ilkarman/Blog/blob/master/DL-Examples/CNTK_CIFAR.ipynb" rel="nofollow noreferrer">CNTK</a> script gets to <strong>0.76</strong> accuracy after 168 seconds of training (10 epochs), which is similar to my <a href="https://github.com/ilkarman/Blog/blob/master/DL-Examples/MXNet_CIFAR.ipynb" rel="nofollow noreferrer">MXNet</a> script (<strong>0.75</strong> accuracy after 153 seconds).</p>
  
  <p>However, my <a href="https://github.com/ilkarman/Blog/blob/master/DL-Examples/PyTorch_CIFAR.ipynb" rel="nofollow noreferrer">PyTorch</a> script is lagging behind a lot at <strong>0.71</strong> accuracy and 354 seconds. I appreciate I will get differences in accuracy due to stochastic weight initialisation, etc. However the difference across frameworks is much greater than difference within a framework, initialising randomly between runs.</p>
  
  <p>The reasons I can think of:</p>
  
  <ul>
  <li>MXNet and CNTK are initialized to xavier/glorot uniform; not sure how to do this in PyTorch and so perhaps the weights are initialised to 0</li>
  <li>CNTK does gradient-clipping by default; not sure if PyTorch has the equivalent</li>
  <li>Perhaps the bias is dropped in PyTorch by default</li>
  <li>I use SGD with momentum; perhaps the PyTorch implementation of momentum is a bit different</li>
  </ul>
  
  <p>Edit:</p>
  
  <p>I have tried specifying the weight initialisation, however it seems to have no big effect:</p>
  
  <pre><code>self.conv1 = nn.Conv2d(3, 50, kernel_size=3, padding=1)
  init.xavier_uniform(self.conv1.weight, gain=np.sqrt(2.0))
  init.constant(self.conv1.bias, 0)
  </code></pre>