PyTorch mini batch, when to call optimizer.zero_grad()
When we use mini batch, should I call optimizer.zero_grad() before starting the iteration? Or inside the iteration? I think the second code is correct, but I'm not sure.
nb_epochs = 20
    for epoch in range(nb_epochs + 1):
      optimizer.zero_grad() # THIS PART!!
      for batch_idx, samples in enumerate(dataloader):
        
        
        x_train, y_train = samples
        
        prediction = model(x_train)
    
        
        cost = F.mse_loss(prediction, y_train)
    
        
        
        cost.backward()
        optimizer.step()
    
        print('Epoch {:4d}/{} Batch {}/{} Cost: {:.6f}'.format(
            epoch, nb_epochs, batch_idx+1, len(dataloader),
            cost.item()
            ))

or
nb_epochs = 20
    for epoch in range(nb_epochs + 1):
      
      for batch_idx, samples in enumerate(dataloader):


        x_train, y_train = samples

        prediction = model(x_train)

        optimizer.zero_grad() #THIS PART!!
        cost = F.mse_loss(prediction, y_train)



        cost.backward()
        optimizer.step()

        print('Epoch {:4d}/{} Batch {}/{} Cost: {:.6f}'.format(
            epoch, nb_epochs, batch_idx+1, len(dataloader),
            cost.item()
            ))

Which one is correct? The only difference is location of optimizer.zero_grad().

Gradients accumulates by default everytime you call .backward() on the computational graph.
On the first snippet, you are resetting the gradients once per epoch so all gradients will accumulate their values over time. With a total of len(dataloader) accumulated gradients, only resseting the gradients when the next epoch starts. On the second snippet, you are doing the right thing, which is to reset the gradient after every backward pass.
So your assumptions were right.
There are some instances where accumulating gradients is needed, but most times it's not.
