Clarifying when a loss/cost function calculation and backprop take place I read different articles and keep getting confused on this point. Not sure if the literature is giving mixed information or I'm interpreting it incorrectly. 

So from reading articles my understanding (loosely) for the following terms are as follows:

Epoch:
One Epoch is when an ENTIRE dataset is passed forward and backward through the neural network only ONCE.

Batch Size:
Total number of training examples present in a single batch. In real life scenarios of utilising neural nets, the dataset needs to be as large as possible, for the network to learn better. So you can’t pass the entire dataset into the neural net at once (due to computation power limitation). So, you divide dataset into Number of Batches.

Iterations:
Iterations is the number of batches needed to complete one epoch. We can divide the dataset of 2000 examples into batches of 500 then it will take 4 iterations to complete 1 epoch.

So if all is correct, then my question is, at what point does the loss/cost function and the subsequent backprop processes take place (assuming from my understanding that backprop takes place straight after the loss/cost is calculated)? Does the cost/loss function gets calculated:


At the end of each batch where the data samples in that batch have been forward-fed to the network (i.e. at each "Iteration, not each Epoch")? If so, then the loss/cost functions gets the average loss of all losses of all data samples in that batch, correct?
At the end of each Epoch? Meaning all the data samples of all the batches are forward-fed first, before the a cost/loss function is calculated. 


My understanding is that it's the first point, i.e. at the end of each batch (passed to the network), hence at each iteration (not Epoch). At least when it comes to SGD optimisation. My understanding is - the whole point is that you calculate loss/cost and backprop for each batch. That way you're not calculating the average loss of the entire data samples. Otherwise you would get a very universal minima value in the cost graph, rather than local minima with lower cost from each batch you train on separately. Once all iterations have taken place, then that would count as 1 Epoch. 
But then I was watching a YouTube video explaining Neural Nets, which mentioned that the cost/loss function is calculated at the end of each Epoch, which confused me. Any clarification would be really appreciated.
Epoch: One Epoch is when an ENTIRE dataset is passed forward and backward through the neural network only ONCE.
  
  Batch Size: Total number of training examples present in a single
  batch. In real life scenarios of utilising neural nets, the dataset
  needs to be as large as possible, for the network to learn better. So
  you can’t pass the entire dataset into the neural net at once (due to
  computation power limitation). So, you divide dataset into Number of
  Batches.
  
  Iterations: Iterations is the number of batches needed to complete one
  epoch. We can divide the dataset of 2000 examples into batches of 500
  then it will take 4 iterations to complete 1 epoch.


This is for the most part correct, except there are other reasons you would sometimes want to use batches (even if you could fit the whole thing in memory). One is that its less likely to overfit in a stochastic setting then in the full setting. Another is that it can achieve similar extrema with faster convergence. 

Now regarding your question, yes you apply the gradient descent step at the end of each batch or desired batch (what I mean by desired batch, is if you want to use a batch-size of 24 but your device can only process 8, you may use gradient accumulation of 3 pseudo-batches to achieve an emulated batch of 24). 

Though I do think its worth mentioning, your goal is still to find the global minimum. Even if your batch size is the entire dataset, that does not mean you will not fall into a local minimum, its actually in most cases still the most probable outcome