Faster R-CNN: Labels regarding the positive anchors when there are many classes
As well as the algorithm used for region proposals in R-CNN and in Fast R-CNN, RPN task in Faster R-CNN is to propose regions that may or may not be an object, so its 
cls head with 2k scores corresponds to the classification of every anchor as an object or as a non object (this way all of them may be non objects): 

But it could also be a k scores logistic regression (k sigmoids 0 to 1) as they stated in the fourth note of the paper:


  
  For simplicity we implement the cls layer as a two-class
  softmax layer. Alternatively, one may use logistic regression to
  produce k scores
  


After proposal generation, Fast R-CNN is used to classify and improve every selected proposal, i.e. grey area in the following image:



So you may better conceptualize the difference between Fast and Faster R-CNN considering the region proposal generation algorithm as a black box, the one could be filled with a RPN head connected to the output convolutional feature map of the deep net used as backbone, or the standard Selective Search used by R-CNN and Fast R-CNN