Faster R-CNN and 4-step Alternating Training
Since no one answered this, I will post my attempt after further research.
Here is my understanding after having thought about it more. Lets say we are using VGG-16 as the backbone.
There are three fundamental components, the backbone, the RPN heads (objectness and RPN regression), and the ROI heads (classification and regression). Using the RPN heads, we can sample positive and negative ROIs.
Step 1) Train a region proposal network (backbone + RPN head) with the VGG-16 backbone pre-trained on imagenet with the head popped off. Some layers of the backbone will be trainable.
Step 2) Using the object proposals (300 or so per image) generated from the network obtained in step 1) (the backbone trained with the rpn), train a second network with another separate VGG-16 backbone pre-trained on imagenet, and a network head as in fast R-CNN (we will call ROI head). Then as in fast R-CNN we train for object classification of a region, as well as regression to learn the displacement from the proposal to a bounding box around the object (if the highest activation in the softmax does not correspond to a &quot;background&quot; class, i.e. no object).
Step 3) Use the backbone from step 2) for RPN training, which will be fixed (= not trainable), with the RPN head trained from step 1), and fine tune the RPN head.
Step 4) Finally keeping the backbone from step 2) fixed, and the RPN heads from step 3) fixed, fine tune the ROI Heads from step 2) to train bounding box regression and classification once more on the region proposals generated from
the backbone and RPN head.
Edit: Original answer had many errors so I came back to correct them, also original question doesn't really make sense now that I read it (I was new to this topic). Please just read this answer.