title: Tensorflow Object detection model evaluation on Test Dataset
body: <p>I have fine-tuned a faster_rcnn_resnet101 model available on the <a href="https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md" rel="nofollow noreferrer">Model Zoo</a> to detect my custom objects. I had the data split into train and eval set, and I used them in the config file while training. 
  Now after training has completed, I want to test my model on an unseen data (I call it the test data). I used a couple of functions but can not figure out for certain which code to use from the tensorflow's API to evaluate the performance on the test dataset. Below are the things that I tried:</p>
  
  <ol>
  <li>I used the object_detection/metrics/offline_eval_map_corloc.py function to get evaluation on test dataset. The code runs fine but I negative values or AR and AP for large and medium sized bounding boxes.</li>
  </ol>
  
  <p>Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.459</p>
  
  <p>Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.601</p>
  
  <p>Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.543</p>
  
  <p>Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.459</p>
  
  <p>Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000</p>
  
  <p>Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000</p>
  
  <p>Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.543</p>
  
  <p>Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.627</p>
  
  <p>Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.628</p>
  
  <p>Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.628</p>
  
  <p>Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000</p>
  
  <p>Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000</p>
  
  <p>Now, I know that mAP and AR can't be negative and there is something wrong. I want to know why do I see negative values when I run the offline evaluation on the test dataset? </p>
  
  <p>The query that I used to run this pipeline is:
  SPLIT=test</p>
  
  <pre><code>echo "
  label_map_path: '/training_demo/annotations/label_map.pbtxt'
  tf_record_input_reader: { input_path: '/training_demo/Predictions/test.record' }
  " &gt; /training_demo/${SPLIT}_eval_metrics/${SPLIT}_input_config.pbtxt
  
  echo "
  metrics_set: 'coco_detection_metrics'
  " &gt; /training_demo/${SPLIT}_eval_metrics/${SPLIT}_eval_config.pbtxt 
  
  python object_detection/metrics/offline_eval_map_corloc.py \
    --eval_dir='/training_demo/test_eval_metrics' \
    --eval_config_path='training_demo/test_eval_metrics/test_eval_config.pbtxt' \
    --input_config_path='/training_demo/test_eval_metrics/test_input_config.pbtxt'
  </code></pre>
  
  <ol start="2">
  <li>I also tried the object_detection/legacy/eval.py but I get values for evaluation metrics as negative:</li>
  </ol>
  
  <p>DetectionBoxes_Recall/AR@100 (medium): -1.0
  DetectionBoxes_Recall/AR@100 (small): -1.0
  DetectionBoxes_Precision/mAP@.50IOU: -1.0
  DetectionBoxes_Precision/mAP (medium): -1.0
  etc. </p>
  
  <p>I used the pipeline,
      python eval.py \
      --logtostderr \
      --checkpoint_dir=trained-inference-graphs/output_inference_graph/ \
      --eval_dir=test_eval_metrics \
      --pipeline_config_path=training/faster_rcnn_resnet101_coco-Copy1.config </p>
  
  <p>The eval_input_reader in the faster_rcnn_resnet101_coco-Copy1.config pointing to the test TFRecord with ground truth and detection information.</p>
  
  <ol start="3">
  <li>I did also try the object_detection/utils/object_detection_evaluation to get the evaluation. This is nothing different than using the 1st approach because it useless the same base functions - evaluator.evaluate()</li>
  </ol>
  
  <p>I would appreciate any help on this.</p>