title: TensorFlow - ValueError: Checkpoint version should be V2
body: <ul>
  <li>GPU: NVIDIA GEFORCE RTX 2060</li>
  <li>GPU: 16GB RAM, 6 processor cores</li>
  <li>TensorFlow: 2.3.1</li>
  <li>Python: 3.8.6</li>
  <li>CUDA: 10.1</li>
  <li>cuDNN: 7.6</li>
  </ul>
  <p>I am training a Mask R-CNN Inception ResNet V2 1024x1024 algorithm (on my computer's GPU), as downloaded from the <a href="https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md" rel="nofollow noreferrer">TensorFlow 2 Detection Model Zoo.</a> I am training this algorithm on my custom dataset, which I have labeled using <a href="https://github.com/tzutalin/labelImg" rel="nofollow noreferrer">Label-img</a> . When I train the model using the Anaconda command <code>python model_main_tf2.py --model_dir=models/my_faster_rcnn --pipeline_config_path=models/my_faster_rcnn/pipeline.config</code>, I get the following error:</p>
  <pre><code>Traceback (most recent call last):
    File &quot;model_main_tf2.py&quot;, line 113, in &lt;module&gt;
      tf.compat.v1.app.run()
    File &quot;C:\user\anaconda3\envs\object_detection_api\lib\site-packages\tensorflow\python\platform\app.py&quot;, line 40, in run
      _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
    File &quot;C:\user\anaconda3\envs\object_detection_api\lib\site-packages\absl\app.py&quot;, line 303, in run
      _run_main(main, args)
    File &quot;C:\user\anaconda3\envs\object_detection_api\lib\site-packages\absl\app.py&quot;, line 251, in _run_main
      sys.exit(main(argv))
    File &quot;model_main_tf2.py&quot;, line 104, in main
      model_lib_v2.train_loop(
    File &quot;C:\user\anaconda3\envs\object_detection_api\lib\site-packages\object_detection\model_lib_v2.py&quot;, line 564, in train_loop
      load_fine_tune_checkpoint(detection_model,
    File &quot;C:\user\anaconda3\envs\object_detection_api\lib\site-packages\object_detection\model_lib_v2.py&quot;, line 348, in load_fine_tune_checkpoint
      raise ValueError('Checkpoint version should be V2')
  ValueError: Checkpoint version should be V2
  </code></pre>
  <p>What is the code needed to resolve this error? (Below are some scripts referenced in the error):</p>
  <p>model_main_tf2.py:</p>
  <pre><code># Lint as: python3
  # Copyright 2020 The TensorFlow Authors. All Rights Reserved.
  #
  # Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);
  # you may not use this file except in compliance with the License.
  # You may obtain a copy of the License at
  #
  #     http://www.apache.org/licenses/LICENSE-2.0
  #
  # Unless required by applicable law or agreed to in writing, software
  # distributed under the License is distributed on an &quot;AS IS&quot; BASIS,
  # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  # See the License for the specific language governing permissions and
  # limitations under the License.
  # ==============================================================================
  
  r&quot;&quot;&quot;Creates and runs TF2 object detection models.
  
  For local training/evaluation run:
  PIPELINE_CONFIG_PATH=path/to/pipeline.config
  MODEL_DIR=/tmp/model_outputs
  NUM_TRAIN_STEPS=10000
  SAMPLE_1_OF_N_EVAL_EXAMPLES=1
  python model_main_tf2.py -- \
    --model_dir=$MODEL_DIR --num_train_steps=$NUM_TRAIN_STEPS \
    --sample_1_of_n_eval_examples=$SAMPLE_1_OF_N_EVAL_EXAMPLES \
    --pipeline_config_path=$PIPELINE_CONFIG_PATH \
    --alsologtostderr
  &quot;&quot;&quot;
  from absl import flags
  import tensorflow.compat.v2 as tf
  from object_detection import model_lib_v2
  
  flags.DEFINE_string('pipeline_config_path', None, 'Path to pipeline config '
                      'file.')
  flags.DEFINE_integer('num_train_steps', None, 'Number of train steps.')
  flags.DEFINE_bool('eval_on_train_data', False, 'Enable evaluating on train '
                    'data (only supported in distributed training).')
  flags.DEFINE_integer('sample_1_of_n_eval_examples', None, 'Will sample one of '
                       'every n eval input examples, where n is provided.')
  flags.DEFINE_integer('sample_1_of_n_eval_on_train_examples', 5, 'Will sample '
                       'one of every n train input examples for evaluation, '
                       'where n is provided. This is only used if '
                       '`eval_training_data` is True.')
  flags.DEFINE_string(
      'model_dir', None, 'Path to output model directory '
                         'where event and checkpoint files will be written.')
  flags.DEFINE_string(
      'checkpoint_dir', None, 'Path to directory holding a checkpoint.  If '
      '`checkpoint_dir` is provided, this binary operates in eval-only mode, '
      'writing resulting metrics to `model_dir`.')
  
  flags.DEFINE_integer('eval_timeout', 3600, 'Number of seconds to wait for an'
                       'evaluation checkpoint before exiting.')
  
  flags.DEFINE_bool('use_tpu', False, 'Whether the job is executing on a TPU.')
  flags.DEFINE_string(
      'tpu_name',
      default=None,
      help='Name of the Cloud TPU for Cluster Resolvers.')
  flags.DEFINE_integer(
      'num_workers', 1, 'When num_workers &gt; 1, training uses '
      'MultiWorkerMirroredStrategy. When num_workers = 1 it uses '
      'MirroredStrategy.')
  flags.DEFINE_integer(
      'checkpoint_every_n', 1000, 'Integer defining how often we checkpoint.')
  flags.DEFINE_boolean('record_summaries', True,
                       ('Whether or not to record summaries during'
                        ' training.'))
  
  FLAGS = flags.FLAGS
  
  
  def main(unused_argv):
    flags.mark_flag_as_required('model_dir')
    flags.mark_flag_as_required('pipeline_config_path')
    tf.config.set_soft_device_placement(True)
  
    if FLAGS.checkpoint_dir:
      model_lib_v2.eval_continuously(
          pipeline_config_path=FLAGS.pipeline_config_path,
          model_dir=FLAGS.model_dir,
          train_steps=FLAGS.num_train_steps,
          sample_1_of_n_eval_examples=FLAGS.sample_1_of_n_eval_examples,
          sample_1_of_n_eval_on_train_examples=(
              FLAGS.sample_1_of_n_eval_on_train_examples),
          checkpoint_dir=FLAGS.checkpoint_dir,
          wait_interval=300, timeout=FLAGS.eval_timeout)
    else:
      if FLAGS.use_tpu:
        # TPU is automatically inferred if tpu_name is None and
        # we are running under cloud ai-platform.
        resolver = tf.distribute.cluster_resolver.TPUClusterResolver(
            FLAGS.tpu_name)
        tf.config.experimental_connect_to_cluster(resolver)
        tf.tpu.experimental.initialize_tpu_system(resolver)
        strategy = tf.distribute.experimental.TPUStrategy(resolver)
      elif FLAGS.num_workers &gt; 1:
        strategy = tf.distribute.experimental.MultiWorkerMirroredStrategy()
      else:
        strategy = tf.compat.v2.distribute.MirroredStrategy()
  
      with strategy.scope():
        model_lib_v2.train_loop(
            pipeline_config_path=FLAGS.pipeline_config_path,
            model_dir=FLAGS.model_dir,
            train_steps=FLAGS.num_train_steps,
            use_tpu=FLAGS.use_tpu,
            checkpoint_every_n=FLAGS.checkpoint_every_n,
            record_summaries=FLAGS.record_summaries)
  
  if __name__ == '__main__':
    tf.compat.v1.app.run()
  </code></pre>
  <p>pipeline.config file:</p>
  <pre><code># Mask R-CNN with Inception Resnet v2 (no atrous)
  # Sync-trained on COCO (with 8 GPUs) with batch size 16 (1024x1024 resolution)
  # Initialized from Imagenet classification checkpoint
  # TF2-Compatible, *Not* TPU-Compatible
  #
  # Achieves XXX mAP on COCO
  
  model {
    faster_rcnn {
      number_of_stages: 3
      num_classes: 1
      image_resizer {
        fixed_shape_resizer {
          height: 1024
          width: 1024
          # pad_to_max_dimension: true
        }
      }
      feature_extractor {
        type: 'faster_rcnn_inception_resnet_v2_keras'
      }
      first_stage_anchor_generator {
        grid_anchor_generator {
          scales: [0.25, 0.5, 1.0, 2.0]
          aspect_ratios: [0.5, 1.0, 2.0]
          height_stride: 16
          width_stride: 16
        }
      }
      first_stage_box_predictor_conv_hyperparams {
        op: CONV
        regularizer {
          l2_regularizer {
            weight: 0.0
          }
        }
        initializer {
          truncated_normal_initializer {
            stddev: 0.01
          }
        }
      }
      first_stage_nms_score_threshold: 0.0
      first_stage_nms_iou_threshold: 0.7
      first_stage_max_proposals: 300
      first_stage_localization_loss_weight: 2.0
      first_stage_objectness_loss_weight: 1.0
      initial_crop_size: 17
      maxpool_kernel_size: 1
      maxpool_stride: 1
      second_stage_box_predictor {
        mask_rcnn_box_predictor {
          use_dropout: false
          dropout_keep_probability: 1.0
          fc_hyperparams {
            op: FC
            regularizer {
              l2_regularizer {
                weight: 0.0
              }
            }
            initializer {
              variance_scaling_initializer {
                factor: 1.0
                uniform: true
                mode: FAN_AVG
              }
            }
          }
          mask_height: 33
          mask_width: 33
          mask_prediction_conv_depth: 0
          mask_prediction_num_conv_layers: 4
          conv_hyperparams {
            op: CONV
            regularizer {
              l2_regularizer {
                weight: 0.0
              }
            }
            initializer {
              truncated_normal_initializer {
                stddev: 0.01
              }
            }
          }
          predict_instance_masks: true
        }
      }
      second_stage_post_processing {
        batch_non_max_suppression {
          score_threshold: 0.0
          iou_threshold: 0.6
          max_detections_per_class: 100
          max_total_detections: 100
        }
        score_converter: SOFTMAX
      }
      second_stage_localization_loss_weight: 2.0
      second_stage_classification_loss_weight: 1.0
      second_stage_mask_prediction_loss_weight: 4.0
      resize_masks: false
    }
  }
  
  train_config: {
    batch_size: 1
    num_steps: 200000
    optimizer {
      momentum_optimizer: {
        learning_rate: {
          cosine_decay_learning_rate {
            learning_rate_base: 0.008
            total_steps: 200000
            warmup_learning_rate: 0.0
            warmup_steps: 5000
          }
        }
        momentum_optimizer_value: 0.9
      }
      use_moving_average: false
    }
    gradient_clipping_by_norm: 10.0
    fine_tune_checkpoint: &quot;pre-trained-models/mask_rcnn_inception_resnet_v2_1024x1024_coco17_gpu-8/checkpoint/ckpt-0&quot;
    data_augmentation_options {
      random_horizontal_flip {
      }
    }
  }
  
  train_input_reader: {
    tf_record_input_reader {
      input_path: &quot;annotations/train.record&quot;
    }
    load_instance_masks: true
    mask_type: PNG_MASKS
  }
  
  eval_config: {
    metrics_set: &quot;coco_detection_metrics&quot;
    metrics_set: &quot;coco_mask_metrics&quot;
    eval_instance_masks: true
    use_moving_averages: false
    batch_size: 1
    include_metrics_per_category: true
  }
  
  eval_input_reader: {
    label_map_path: &quot;annotations/label_map.pbtxt&quot;
    shuffle: false
    num_epochs: 1
    tf_record_input_reader {
      input_path: &quot;annotations/test.record&quot;
    }
    load_instance_masks: true
    mask_type: PNG_MASKS
  }
  </code></pre>
  <p>The rest of the python scripts referenced in the error can be found <a href="https://github.com/tensorflow/tensorflow/issues/44843" rel="nofollow noreferrer">here</a>, as they would not fit in a single StackOverflow post.</p>