title: TensorFlow object detection API evaluate training performance
body: <p>I have been using <a href="https://github.com/tensorflow/models/tree/master/research/object_detection" rel="nofollow noreferrer">Tensorflow Object Detection API</a> on my own dataset. While training, I want to know how well the NN is learning from the Training set. So, I want to run an evaluation on both training and eval set and get accuracy (mAP) respectively during the training sessions. </p>
  
  <p>My config file:</p>
  
  <pre><code>model {
    faster_rcnn {
      num_classes: 50
      image_resizer {
        fixed_shape_resizer {
          height: 960
          width: 960
        }
      }
      number_of_stages: 3
      feature_extractor {
        type: 'faster_rcnn_resnet101'
        first_stage_features_stride: 8
      }
      first_stage_anchor_generator {
        grid_anchor_generator {
          scales: [0.25, 0.5, 1.0, 2.0]
          aspect_ratios: [0.5, 1.0, 2.0]
          height_stride: 8
          width_stride: 8
        }
      }
      first_stage_atrous_rate: 2
      first_stage_box_predictor_conv_hyperparams {
        op: CONV
        regularizer {
          l2_regularizer {
            weight: 0.0
          }
        }
        initializer {
          truncated_normal_initializer {
            stddev: 0.00999999977648
          }
        }
      }
      first_stage_nms_score_threshold: 0.0
      first_stage_nms_iou_threshold: 0.699999988079
      first_stage_max_proposals: 100
      first_stage_localization_loss_weight: 2.0
      first_stage_objectness_loss_weight: 1.0
      initial_crop_size: 14
      maxpool_kernel_size: 2
      maxpool_stride: 2
      second_stage_box_predictor {
        mask_rcnn_box_predictor {
          use_dropout: false
          dropout_keep_probability: 1.0
          fc_hyperparams {
            op: FC
            regularizer {
              l2_regularizer {
                weight: 0.0
              }
            }
            initializer {
              variance_scaling_initializer {
                factor: 1.0
                uniform: true
                mode: FAN_AVG
              }
            }
          }
          conv_hyperparams {
            op: CONV
            regularizer {
              l2_regularizer {
                weight: 0.0
              }
            }
            initializer {
              truncated_normal_initializer {
                stddev: 0.00999999977648
              }
            }
          }
          predict_instance_masks: true
          mask_height: 33
          mask_width: 33
          mask_prediction_conv_depth: 0
          mask_prediction_num_conv_layers: 4
        }
      }
      second_stage_post_processing {
        batch_non_max_suppression {
          score_threshold: 0.300000011921
          iou_threshold: 0.600000023842
          max_detections_per_class: 100
          max_total_detections: 100
        }
        score_converter: SOFTMAX
      }
      second_stage_localization_loss_weight: 2.0
      second_stage_classification_loss_weight: 1.0
      second_stage_mask_prediction_loss_weight: 4.0
    }
  }
  train_config: {
    batch_size: 1
    optimizer {
      momentum_optimizer: {
        learning_rate: {
          manual_step_learning_rate {
            initial_learning_rate: 0.003
            schedule {
              step: 3000
              learning_rate: 0.00075
            }
            schedule {
              step: 6000
              learning_rate: 0.000300000014249
            }
            schedule {
              step: 15000
              learning_rate: 0.000075
            }
            schedule {
              step: 18000
              learning_rate: 0.0000314249
            }
            schedule {
              step: 900000
              learning_rate: 2.99999992421e-05
            }
            schedule {
              step: 1200000
              learning_rate: 3.00000010611e-06
            }
          }
        }
        momentum_optimizer_value: 0.899999976158
      }
      use_moving_average: false
    }
    gradient_clipping_by_norm: 10.0
    fine_tune_checkpoint: "./mask_rcnn_resnet101_atrous_coco/model.ckpt"
    from_detection_checkpoint: true
    num_steps: 200000
    data_augmentation_options {
      random_horizontal_flip {
      }
    }
  }
  train_input_reader: {
    label_map_path: "./map901_label_map.pbtxt"
    load_instance_masks: true
    mask_type: PNG_MASKS
    tf_record_input_reader {
      input_path: ["./my_coco_train.record-?????-of-00005"]
    }
  }
  eval_config: {
    num_examples: 8000
    max_evals: 100
    num_visualizations: 25
  }
  eval_input_reader: {
    label_map_path: "./map901_label_map.pbtxt"
    shuffle: false
    load_instance_masks: true
    mask_type: PNG_MASKS
    num_readers: 1
    tf_record_input_reader {
      input_path: ["./my_coco_val.record-?????-of-00001"]
    }
  }
  </code></pre>
  
  <p>I ran the script with these parameters</p>
  
  <pre><code>python model_main.py --alsologtostderr \
    --pipeline_config_path=${PIPELINE_CONFIG_PATH} \
    --model_dir=${TRAIN_DIR} \
    --num_train_steps=24000 \
    --sample_1_of_n_eval_on_train_examples=25 \
    --num_eval_steps=100 \
    --sample_1_of_n_eval_examples=1 
  </code></pre>
  
  <p>I think this will run an evaluation of Eval examples. To evaluate training data (to check how many features captured from training) I have added 
  <code>--eval_training_data=True</code> to the parameters. </p>
  
  <p>I can not add the "eval_training_data" on the go. I need to run 2 different training sessions.</p>
  
  <p>Interestingly, with "eval_training_data" parameter added I got, </p>
  
  <pre><code> Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.165
   Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.281
   Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.167
   Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.051
   Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.109
   Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.202
   Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.164
   Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.202
   Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.202
   Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.057
   Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.141
   Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.236
  </code></pre>
  
  <p>Whereas without "eval_training_data" I got</p>
  
  <pre><code> Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.168
   Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.283
   Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.173
   Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.049
   Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.108
   Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.208
   Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.170
   Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.208
   Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.208
   Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.056
   Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.139
   Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.248
  </code></pre>
  
  <p>I just got confused. My questions are:</p>
  
  <ol>
  <li>Does not "eval_training_data" force object detection API to run evaluation on the Training set? </li>
  <li>Why in my case both scores are almost the same, in some cases eval score is better? </li>
  <li>Which parameters need to add so that it evaluates both training and eval set separately during a training session and print it?</li>
  </ol>