title: Getting "Graph is finalized and cannot be modified" during evaluation for pre-trained RPN
body: <p>I am building a network that is re-using the region proposal network weights from the matterport Mask R-CNN repo (mask_rcnn_coco.h5), and only the RPN. The code is very similar to what's in the original repository:</p>
  
  <pre><code>    def rpn_graph(self, feature_map, anchors_per_location=3, anchor_stride=1):
          """Builds the computation graph of Region Proposal Network.
          feature_map: backbone features [frames, height, width, depth]
          anchors_per_location: number of anchors per pixel in the feature map
          anchor_stride: Controls the density of anchors. Typically 1 (anchors for
                         every pixel in the feature map), or 2 (every other pixel).
          Returns:
              rpn_class_logits: [frames, H * W * anchors_per_location, 2] Anchor classifier logits (before softmax)
              rpn_probs: [frames, H * W * anchors_per_location, 2] Anchor classifier probabilities.
              rpn_bbox: [frames, H * W * anchors_per_location, (dy, dx, log(dh), log(dw))] Deltas to be
                        applied to anchors.
          """
          # Shared convolutional base of the RPN
          shared = KL.Conv2D(512, (3, 3), padding='same', activation='relu',
                             strides=anchor_stride, name='rpn_conv_shared', trainable=False)(feature_map)
  
          # Anchor Score. [batch, height, width, anchors per location * 2].
          x = KL.Conv2D(2 * anchors_per_location, (1, 1), padding='valid',
                        activation='linear', name='rpn_class_raw', trainable=False)(shared)
  
          # Reshape to [batch, anchors, 2]
          rpn_class_logits = KL.Lambda(lambda t: tf.reshape(t, [tf.shape(t)[0], -1, 2]))(x)
  
          # Softmax on last dimension of BG/FG.
          rpn_probs = KL.Activation("softmax", name="rpn_class_xxx")(rpn_class_logits)
  
          # Bounding box refinement. [batch, H, W, anchors per location * depth]
          # where depth is [x, y, log(w), log(h)]
          x = KL.Conv2D(anchors_per_location * 4, (1, 1), padding="valid", activation='linear',
                        name='rpn_bbox_pred', trainable=False)(shared)
  
          # Reshape to [batch, anchors, 4]
          rpn_bbox = KL.Lambda(lambda t: tf.reshape(t, [tf.shape(t)[0], -1, 4]))(x)
          return [rpn_class_logits, rpn_probs, rpn_bbox]
  
      def build_rpn_model(self):
          """Builds a Keras model of the Region Proposal Network.
          It wraps the RPN graph so it can be used multiple times with shared
          weights.
  
          Returns a Keras Model object. The model outputs, when called, are:
          rpn_class_logits: [batch, H * W * anchors_per_location, 2] Anchor classifier logits (before softmax)
          rpn_probs: [batch, H * W * anchors_per_location, 2] Anchor classifier probabilities.
          rpn_bbox: [batch, H * W * anchors_per_location, (dy, dx, log(dh), log(dw))] Deltas to be
                      applied to anchors.
          """
          input_feature_map = KL.Input(shape=[None, None, self.DEPTH], name="input_rpn_feature_map")
          outputs = self.rpn_graph(
              input_feature_map,
              self.ANCHORS_PER_LOCATION,
              self.ANCHOR_STRIDE
          )
  
          model = KM.Model([input_feature_map], outputs, name="rpn_model")
          self.load_pre_trained_rpn_weights(model)
  
          return model
  
      def load_pre_trained_rpn_weights(self, model):
          """
          We load the weights for the region proposal network from the pre-trained mask-rcnn model
  
          :param model: Keras model for the region proposal network
          :return:
          """
  
          weights = h5py.File(self.PATH_TO_MASK_RCNN_WEIGHTS, mode='r')
          saving.load_weights_from_hdf5_group_by_name(weights, [model])
  
          weights.close()
  </code></pre>
  
  <p>Then I initialize the RPN in <strong>init</strong> to use later on:</p>
  
  <pre><code>    ...
      self.region_proposal_network = self.build_rpn_model()
      ...
  </code></pre>
  
  <p>Training is running well and I can see the checkpoint is saved. But then, when the model is loaded for evaluation I get (using <code>tf.estimator.train_and_evaluate</code>)</p>
  
  <pre><code>    RuntimeError: Graph is finalized and cannot be modified.
  </code></pre>
  
  <p>Any tips that could help me out?</p>