Clarifying when a loss/cost function calculation and backprop take place
Epoch: One Epoch is when an ENTIRE dataset is passed forward and backward through the neural network only ONCE.
  
  Batch Size: Total number of training examples present in a single
  batch. In real life scenarios of utilising neural nets, the dataset
  needs to be as large as possible, for the network to learn better. So
  you can’t pass the entire dataset into the neural net at once (due to
  computation power limitation). So, you divide dataset into Number of
  Batches.
  
  Iterations: Iterations is the number of batches needed to complete one
  epoch. We can divide the dataset of 2000 examples into batches of 500
  then it will take 4 iterations to complete 1 epoch.


This is for the most part correct, except there are other reasons you would sometimes want to use batches (even if you could fit the whole thing in memory). One is that its less likely to overfit in a stochastic setting then in the full setting. Another is that it can achieve similar extrema with faster convergence. 

Now regarding your question, yes you apply the gradient descent step at the end of each batch or desired batch (what I mean by desired batch, is if you want to use a batch-size of 24 but your device can only process 8, you may use gradient accumulation of 3 pseudo-batches to achieve an emulated batch of 24). 

Though I do think its worth mentioning, your goal is still to find the global minimum. Even if your batch size is the entire dataset, that does not mean you will not fall into a local minimum, its actually in most cases still the most probable outcome