Performance gap between `batch_size==32` and `batch_size==8, gradient_accumulation==4`
I tried to use gradient accumulation in my project. To my understanding, the gradient accumulation is the same as increasing the batch size by x times. I tried batch_size==32 and batch_size==8, gradient_accumulation==4 in my project, however the result varies even when I disabled shuffle in dataloader. The batch_size==8, accumulation==4 variant's result is significantly poorer.
I wonder why?
Here is my snippet:
loss = model(x)
epoch_loss += float(loss)

loss.backward()

# step starts from 1
if (step % accumulate_step == 0) or (step == len(dataloader)):

    if clip_grad_norm &gt; 0:
        nn.utils.clip_grad_norm_(model.parameters(), max_norm=clip_grad_norm)

    optimizer.step()
    if scheduler:
        scheduler.step()

    optimizer.zero_grad()


Assuming your loss is mean-reduced, then you need to scale the loss by 1/accumulate_step
The default behavior of most loss functions is to return the average loss value across each batch element. This is referred to as mean-reduction, and has the property that batch size does not affect the magnitude of the loss (or the magnitude of the gradients of loss). However, when implementing gradient accumulation, each time you call backward you are adding gradients to the existing gradients. Therefore, if you call backward four times on quarter-sized batches, you are actually producing gradients that are four-times larger than if you had called backward once on a full-sized batch. To account for this behavior you need to divide the gradients by accumulate_step, which can be accomplished by scaling the loss by 1/accumulate_step before back-propagation.
loss = model(x) / accumulate_step

loss.backward()

# step starts from 1
if (step % accumulate_step == 0) or (step == len(dataloader)):

    if clip_grad_norm &gt; 0:
        nn.utils.clip_grad_norm_(model.parameters(), max_norm=clip_grad_norm)

    optimizer.step()
    if scheduler:
        scheduler.step()

    optimizer.zero_grad()

