title: How to add custom evaluation metrics in Tensorflow Object Detection API?
body: <p>I would like to have my custom list of metrics when evaluating an instance segmentation model in Tensorflow's Object Detection API, which can be summarized as follows;</p>
  
  <ul>
  <li>Precision values for IOUs of 0.5-0.95 with increments of 0.05</li>
  <li>Recall values  for IOUs of 0.5-0.95 with increments of 0.05</li>
  <li>AUC values for precision and recall between 0-1 with increments of 0.05</li>
  </ul>
  
  <p>What I've currently tested is modifying the already existing coco evaluation metrics by tweaking some code in the PythonAPI of pycocotools and the additional metrics file within Tensorflow's research model. Currently the default output values for COCO evaluation are the following</p>
  
  <pre><code>Precision/mAP
  Precision/mAP@.50IOU
  Precision/mAP@.75IOU
  Precision/mAP (small)
  Precision/mAP (medium)
  Precision/mAP (large)
  Recall/AR@1
  Recall/AR@10
  Recall/AR@100
  Recall/AR@100 (small)
  Recall/AR@100 (medium)
  Recall/AR@100 (large)
  </code></pre>
  
  <p>So I decided first to use <code>coco_detection_metrics</code> in my <code>eval_config</code> field inside the .config file used for training</p>
  
  <pre><code>eval_config: {
    metrics_set: "coco_detection_metrics"
  }
  </code></pre>
  
  <p>And edit <code>cocoeval.py</code> and <code>cocotools.py</code> multiple times (proportional to the number of values) by adding more items to the stats list and stats sumary dictionary in order to get the desired result. For demonstration purposes, I am only going to show one example by adding precision at IOU=0.55 on top of precision at IOU=0.5.</p>
  
  <p>So, this is the modified method of the <code>COCOeval</code> class inside <a href="https://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocotools/cocoeval.py" rel="nofollow noreferrer">cocoeval.py</a></p>
  
  <pre><code>def _summarizeDets():
      stats[1] = _summarize(1, iouThr=.5, maxDets=self.params.maxDets[2])
      stats[12] = _summarize(1, iouThr=.5, maxDets=self.params.maxDets[2])
  </code></pre>
  
  <p>and the edited methods under the <code>COCOEvalWrapper</code> class inside <a href="https://github.com/tensorflow/models/blob/master/research/object_detection/metrics/coco_tools.py" rel="nofollow noreferrer">coco_tools.py</a></p>
  
  <pre><code>summary_metrics = OrderedDict([
      ('Precision/mAP@.50IOU', self.stats[1]),
      ('Precision/mAP@.55IOU', self.stats[12])
  for category_index, category_id in enumerate(self.GetCategoryIdList()):
      per_category_ap['Precision mAP@.50IOU ByCategory/{}'.format( category)] = self.category_stats[1][category_index]
      per_category_ap['Precision mAP@.55IOU ByCategory/{}'.format( category)] = self.category_stats[12][category_index]
  </code></pre>
  
  <p>It would be useful to know a more efficient way to deal with my problem and easily request a list of custom evaluation metrics without having to tweak the already existing COCO files. Ideally, my <strong>primary</strong> goal is to</p>
  
  <ul>
  <li>Be able to create a custom console output based on the metrics provided at the beginning of the question</li>
  </ul>
  
  <p>and my secondary goals would be to</p>
  
  <ul>
  <li>Export the metrics with their respective values in JSON format</li>
  <li>Visualize the three graphs in Tensorboard</li>
  </ul>