title: Will larger batch size make computation time less in machine learning?
body: <p>I am trying to tune the hyper parameter i.e <strong>batch size</strong> in CNN.I have a computer of corei7,RAM 12GB and i am training a CNN network with CIFAR-10 dataset which can be found in this <a href="http://torch.ch/blog/2015/07/30/cifar.html" rel="nofollow noreferrer">blog</a>.<br><br><em>Now At first what i have read and learnt about batch size in machine learning:</em></p>
  
  <blockquote>
    <p>let's first suppose that we're doing online learning, i.e. that we're
    using a mini­batch size of 1. The obvious worry about online learning
    is that using mini­batches which contain just a single training
    example will cause significant errors in our estimate of the gradient.
    In fact, though, the errors turn out to not be such a problem. The
    reason is that the individual gradient estimates don't need to be
    super­accurate. All we need is an estimate accurate enough that our
    cost function tends to keep decreasing. It's as though you are trying
    to get to the North Magnetic Pole, but have a wonky compass that's
    10­-20 degrees off each time you look at it. Provided you stop to
    check the compass frequently, and the compass gets the direction right
    on average, you'll end up at the North Magnetic Pole just
    fine.<br><br></p>
    
    <p>Based on this argument, it sounds as though we should use online
    learning. In fact, the situation turns out to be more complicated than
    that.As we know we can use matrix techniques to compute the gradient
    update for all examples in a mini­batch simultaneously, rather than
    looping over them. Depending on the details of our hardware and linear
    algebra library this can make it quite a bit faster to compute the
    gradient estimate for a mini­batch of (for example) size 100 , rather
    than computing the mini­batch gradient estimate by looping over the
    100 training examples separately. It might take (say) only 50 times as
    long, rather than 100 times as long.Now, at first it seems as though
    this doesn't help us that much.<br><br></p>
    
    <p>With our mini­batch of size 100 the learning rule for the weights
    looks like:<a href="https://i.stack.imgur.com/QDhgB.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/QDhgB.png" alt="enter image description here"></a><br></p>
    
    <p>where the sum is over training examples in the mini­batch. This is
    versus<a href="https://i.stack.imgur.com/X4CkK.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/X4CkK.png" alt="enter image description here"></a><br> for online learning.
    Even if it only takes 50 times as long to do the mini­batch update, it
    still seems likely to be better to do online learning, because we'd be
    updating so much more frequently. Suppose, however, that in the
    mini­batch case we increase the learning rate by a factor 100, so the
    update rule becomes<br><a href="https://i.stack.imgur.com/KMnF1.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/KMnF1.png" alt="enter image description here"></a><br>
    That's a lot like doing separate instances of online learning with a
    learning rate of <code>η</code>. But it only takes 50 times as long as doing a
    single instance of online learning. Still, it seems distinctly
    possible that using the larger mini­batch would speed things up.</p>
  </blockquote>
  
  <p><br><br></p>
  
  <p>Now i tried with <code>MNIST digit dataset</code> and ran a sample program and set the batch size <code>1</code> at first.I noted down the training time needed for the full dataset.Then i increased the batch size and i noticed that it became faster.
  <br>
  But in case of training with this  <a href="https://github.com/szagoruyko/cifar.torch/blob/master/models/vgg_bn_drop.lua" rel="nofollow noreferrer">code</a> and <a href="https://github.com/szagoruyko/cifar.torch" rel="nofollow noreferrer">github link</a> changing the batch size doesn't decrease the training time.It remained same if i use 30 or 128 or 64.They are saying that they got <code>92%</code> accuracy.After two or three epoch they have got above <code>40%</code> accuracy.But when i ran the code in my computer without changing anything other than the batch size i got worse result after 10 epoch like only 28% and test accuracy stuck there in the next epochs.Then i thought since they have used batch size of 128 i need to use that.Then i used the same but it became more worse only give 11% after 10 epoch and stuck in there.<strong>Why is that??</strong></p>