Faster R-CNN wrapper for the number of RPNs in the layer dimensions?
OK so the trick was in the RoI Pooling Layer: it is initialized with dimensions (1,512,7,7) and outputs the tensor (#RoIs,512,7,7), which in turn is an input in a fully-connected layer fc6 that treats the input as batch size #RoIs and hence processes them in the normal way, like it would have processed any batch.