When are weights updated in CNN?
Whenever you train the network using batch means that you have chosen to train using batch gradient descent. There are three variants for gradient descent algorithm:


Gradient Descent
Stochastic Gradient Descent
Batch Gradient Descent 


The first one passes the whole data through the network and finds the error rate for all of them and finds the gradients with respect to all the data samples and updates the weights after passing the whole data-set. That means for each epoch, passing the whole data-set through the network, one update occurs. This update is accurate toward descending gradient. 

The second one, updates the weights after passing each data which means if your data sample has one thousand samples, one thousand updates will happen whilst the previous method updates the weights one time per the whole data-sample. This method is not accurate but is so much faster than the previous one.

The last one tries to find a trade-off between the above approaches. You specify a batch size and you will update the weights after passing the data samples in each batch, means the gradients are calculated after passing each batch. Suppose you have one thousand data sample and you have specified a batch size with one hundred data sample. You will have 10 weight update for each epoch. This method is more accurate than the second approach and is more faster than the first approach. 


  Do I back propagate after each batch has been presented to network or after each image?


Your method is the last one. Consequently, after passing the entire batch, you would update the weights.


Based on the comments of one of our friends, the above approaches are named as follows, respectively:


[Batch] Gradient Descent (batch size = all training samples)
True SGD (batch size = 1 - weights update for each training sample)
Mini-batch SGD (batch size = m out of n training samples).