Are there any rules for choosing the size of a mini-batch?
In On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima there are a couple of intersting statements:


  It has been observed in practice that
  when using a larger batch there is a degradation in the quality of the model, as measured by its ability to generalize [...]
  
  large-batch methods tend to converge to sharp minimizers of the
  training and testing functions—and as is well known, sharp minima lead to poorer
  generalization. n. In contrast, small-batch methods consistently converge to flat minimizers, and our experiments support a commonly held view that this is due to the inherent noise in the gradient estimation.


From my masters thesis: Hence the choice of the mini-batch size influences:


Training time until convergence: There seems to be a sweet spot. If the batch size is very small (e.g. 8), this time goes up. If the batch size is huge, it is also higher than the minimum.
Training time per epoch: Bigger computes faster (is efficient)
Resulting model quality: The lower the better due to better generalization (?)


It is important to note hyper-parameter interactions: Batch size may interact with other hyper-parameters, most notably learning rate. In some experiments this interaction may make it hard to isolate the effect of batch size alone on model quality. Another strong interaction is with early stopping for regularisation.

See also


this nice answer / related question
Efficient Mini-batch Training for Stochastic Optimization
this RNN study