title: Triplet model for image retrieval from the Keras pretrained network
body: <p>I want to implement a model for image retrieval. The model will be trained with a triplet loss function (same as facenet or similar architectures). My idea was to use a pretrained classification model from Keras (e.g. resnet50), and make it a triple architecture. This is my model in Keras:</p>
  
  <pre><code>resnet_input = Input(shape=(224,224,3))
  resnet_model = ResNet50(weights='imagenet', include_top = False, input_tensor=resnet_input)
  net = resnet_model.output
  
  net = Flatten(name='flatten')(net) 
  net = Dense(512, activation='relu', name='embded')(net)
  net = Lambda(l2Norm, output_shape=[512])(net)
  
  base_model = Model(resnet_model.input, net, name='resnet_model')
  
  input_shape=(224,224,3)
  input_anchor = Input(shape=input_shape, name='input_anchor')
  input_positive = Input(shape=input_shape, name='input_pos')
  input_negative = Input(shape=input_shape, name='input_neg')
  
  net_anchor = base_model(input_anchor)
  net_positive = base_model(input_positive)
  net_negative = base_model(input_negative)
  
  positive_dist = Lambda(euclidean_distance, name='pos_dist')([net_anchor, net_positive])
  negative_dist = Lambda(euclidean_distance, name='neg_dist')([net_anchor, net_negative])
  
  stacked_dists = Lambda( 
              lambda vects: K.stack(vects, axis=1),
              name='stacked_dists'
  )([positive_dist, negative_dist])
  
  model = Model([input_anchor, input_positive, input_negative], stacked_dists, name='triple_siamese')
  
  def triplet_loss(_, y_pred):
      margin = K.constant(1)
      return K.mean(K.maximum(K.constant(0), K.square(y_pred[0]) - K.square(y_pred[1]) + margin))
  
  def accuracy(_, y_pred):
      return K.mean(y_pred[0] &lt; y_pred[1])
  
  def l2Norm(x):
      return  K.l2_normalize(x, axis=-1)
  
  def euclidean_distance(vects):
      x, y = vects
      return K.sqrt(K.maximum(K.sum(K.square(x - y), axis=1, keepdims=True), K.epsilon()))
  </code></pre>
  
  <p>The model should predict a feature vector for every image. The distances (euclidean in this case) between these vectors should be close to zero if the images are from the same class, or close to one if they're not.</p>
  
  <p>I already tried different learning steps, batch sizes, different margins in the loss function, choosing different output layers from the original resnet model, different layers added to the end of the resnet, training just the new added layers vs training the whole model. I also tried to use this resnet model without pretrained weights, <strong>the result was still the same, around 0.5 accuracy and 1.0 loss</strong>, no matter what I did. (The input images were preprocessed in the way which was expected for this model with <code>keras.applications.resnet50.preprocess_input</code>)</p>
  
  <p>I didn't do any hard negative mining which may result in slow convergence, but still, 0.5 accuracy in this case (check the function) is random predictions.</p>
  
  <p>So I started thinking maybe I miss something really important here (it's a quite difficult architecture). So please, if you notice something wrong or suspicious in my implementation I'd be really glad for that.</p>