title: Choosing minibatch size for deep learning
body: <p>In a blog post by Ilya Sutskever, <a href="http://yyue.blogspot.dk/2015/01/a-brief-overview-of-deep-learning.html" rel="noreferrer">A brief overview of Deep Learning</a>, he describes how it is important to choose the right minibatch size to train a deep neural network efficiently. He gives the advice "use the smaller minibatch that runs efficiently on your machine". See the full quote below.</p>
  
  <p>I've seen similar statements by other well-known deep learning researchers, but it is still unclear to me how to find the correct minibatch size. Seeing as a greater minibatch can allow for a greater learning rate, it seems like it requires a lot of experiments to determine whether a certain minibatch size yields a better performance in terms of training speed. </p>
  
  <p>I have a GPU with 4gb of RAM and use the libraries Caffe and Keras. What is in this case a practical heuristic for choosing a good minibatch size given that each observation has a certain memory footprint <code>M</code>?</p>
  
  <blockquote>
    <p>Minibatches: Use minibatches. Modern computers cannot be efficient if
    you process one training case at a time. It is vastly more efficient
    to train the network on minibatches of 128 examples, because doing so
    will result in massively greater throughput. It would actually be nice
    to use minibatches of size 1, and they would probably result in
    improved performance and lower overfitting; but the benefit of doing
    so is outweighed the massive computational gains provided by
    minibatches. But don’t use very large minibatches because they tend to
    work less well and overfit more. So the practical recommendation is:
    use the smaller minibatch that runs efficiently on your machine.</p>
  </blockquote>