low object confidence scores while training yolov2 for face-non face dataset
i am posting the answer here , in case if someone goes through same issue they can follow the same debugging steps , 
So first i figured what was the cause of low combined scores , and it turned out that it was the object scores which were very less , 
So i looked at the distribution of the activations after each loss and also the behaviour of object loss , seemed like it was fluctuating a lot , which suggested that the loss from this particular term was not according to the desired pattern . 
hence i recursed back to the loss function and fortunately the mistake was in the IOU calculation of the loss function . 
The problem was basically with the bounds of the iou calculation , i.e there was no bounds on what the intersection of two boxes can be , so the range  of the intersection can from -inf to +inf. Hence the iou was coming to be huge lot of times , but intuitively iou can be only from 0 to 1 and . 
the solution to it was to truncate the localisation of predictions from 0 to image height and image width during preprocessing for iou calculation. also, while calculating the iou which is intersection / union , i made sure that intersection is greater or equal to 0 (not less than which means that there is not intersection at all).
Doing this helped the model learn in a lot correct and better way to predict boxes whenever it can .
Currently i am working on an object detection problem which involves detecting faces in an images  and creating boxes around them.
To address this issue , i have created a yolov2 model as described in the paper with loss function as mentioned in the paper itlself. At earlier stages i was facing the issue of Nan losses for localization loss and hence unstable training. Hence  found a workaround by making some small changes in loss function. Presently the issue i am facing is with very low object confidence score, as i am dealing with face-no_face dataset ,the class score during inference is 1.0 but the object confidence score is very lesss for many of the instances which contain only one face , let alone multiple faces in an image. Though while training the cummulative loss is saturating around 0.06 , but while inference , there is whole new picture of very less confidence score.
Loss function : The loss function i am using for calculating confidence score is cummumative loss of two terms : no_object_loss and object_loss and both are MSE losses for grid cells which has object (object_loss) and rest contribute to no_object_loss.
Note : My dataset contains 4996 instances of multi face images and ONLY 4 INSTANCES OF NO FACE. and so i was wondering if this is the issue as i why i am getting low object confidence scores.