Changing the batch size during training
Efficient use of resources
It is a balancing game with the learning rate, and one reason you don't normally see people do this is that you want to utilise as much of the GPU as possible.
It is commonly preferred to start with the maximum batch size you can fit in memory, then increase the learning rate accordingly. This applies to &quot;effective batch sizes&quot; e.g. when you have 4 GPUs, each running with batch_size=10, then you might have a global learning rate of nb_gpu * initial_lr (used with sum or average of all 4 GPUs).
The final &quot;best approach&quot; is usually problem specific - small batch sizes might not work for GAN type models, and large batch sizes might be slow and sub-optimal for certain vision based tasks.
Friends don't let friends use large batch sizes
There is literature to support usage of small batch sizes at almost all times. Even though this idea was supported by Yann Lecun, there are differences of opinion.
Super convergence
There are also other tricks that you might consider, if you are interested in faster convergence, playing with learning rate cycling.