title: Regularisers in Keras vs. Caffe
body: <p>I've edited an AlexNet in KERAS, trying to learn a single class (i.e. dogs/faces or not dog/faces).
  I have training samples of <strong>correct</strong> images so the Ytrain is <strong>[1, 0]</strong>, and <strong>incorrect</strong> images so the Ytrain is <strong>[0,1]</strong>.
  The goal is to know if the image is a dog (for example).</p>
  
  <p>while training the loss was very big ~ 100,000,000</p>
  
  <p>so i deleted the regularisers lines and it worked...</p>
  
  <pre><code>activity_regularizer=ActivityRegularizer(l1=1, l2=1),
  W_regularizer=WeightRegularizer(l1=2.0,l2=0.0)))
  </code></pre>
  
  <p>In Caffe the model works fine with these lines in the convolution layers:</p>
  
  <pre><code>param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  </code></pre>
  
  <p>Here is the model in Keras(image size 3,68,56):</p>
  
  <pre><code>    Xtrain = self.Xtrain
      Ytrain = self.Ytrain
  
      batch_size = 10
      nb_classes = 2
      nb_epoch = 3
  
      # input image dimensions
      img_rows, img_cols = np.shape(self.Xtrain)[2], np.shape(self.Xtrain)[3]
      #########################
      ###   AlexNet - Start  ##
      #########################
      print("Defining AlexNet...")
  
      model = Sequential()
      model.add(ZeroPadding2D((1,1),input_shape=(3,img_rows,img_cols)))
      model.add(Convolution2D(96, 11, 11, border_mode='valid',
                              init='glorot_normal',
                              activation='relu',
                              activity_regularizer=ActivityRegularizer(l1=1, l2=1),
                              W_regularizer=WeightRegularizer(l1=2.0,l2=0.0)))
      model.add(BatchNormalization())
      model.add(MaxPooling2D(pool_size=(3, 3),strides=(2,2)))
  
      model.add(ZeroPadding2D((1,1)))
      model.add(Convolution2D(256, 5, 5, border_mode='valid',
                              init='glorot_normal',
                              activation='relu',
                              activity_regularizer=ActivityRegularizer(l1=1, l2=1),
                              W_regularizer=WeightRegularizer(l1=2,l2=0)))
      model.add(BatchNormalization())
      model.add(MaxPooling2D(pool_size=(3, 3),strides=(2,2)))
  
      model.add(ZeroPadding2D((1,1)))
      model.add(Convolution2D(384, 3, 3, border_mode='valid',
                              init='glorot_normal',
                              activation='relu',
                              activity_regularizer=ActivityRegularizer(l1=1, l2=1),
                              W_regularizer=WeightRegularizer(l1=2,l2=0)))
  
      model.add(ZeroPadding2D((1,1)))
      model.add(Convolution2D(384, 3, 3, border_mode='valid',
                              init='glorot_normal',
                              activation='relu',
                              activity_regularizer=ActivityRegularizer(l1=1, l2=1),
                              W_regularizer=WeightRegularizer(l1=2,l2=0)))
  
      model.add(ZeroPadding2D((1,1)))
      model.add(Convolution2D(256, 3, 3, border_mode='valid',
                              init='glorot_normal',
                              activation='relu',
                              activity_regularizer=ActivityRegularizer(l1=1, l2=1),
                              W_regularizer=WeightRegularizer(l1=2,l2=0)))
      model.add(MaxPooling2D(pool_size=(3, 3),strides=(2,2)))
  
      model.add(Flatten())
  
      model.add(Dense(4096,
                      activation='relu',
                      init='glorot_normal',
                      activity_regularizer=ActivityRegularizer(l1=1, l2=1),
                      W_regularizer=WeightRegularizer(l1=2,l2=0)))
      model.add(Dropout(0.5))
  
      model.add(Dense(4096,
                      activation='relu',
                      init='glorot_normal',
                      activity_regularizer=ActivityRegularizer(l1=1, l2=1),
                      W_regularizer=WeightRegularizer(l1=2,l2=0)))
      model.add(Dropout(0.5))
  
  
      model.add(Dense(nb_classes,
                      activation='softmax',
                      activity_regularizer=ActivityRegularizer(l1=1, l2=1),
                      W_regularizer=WeightRegularizer(l1=2,l2=0)))
  
      #######################
      ###   AlexNet - End  ##
      #######################
  
      add = Adadelta(lr=0.1, rho=0.95, epsilon=1e-06, decay=0.995)
      print("Compiling AlexNet...")
      model.compile(loss='categorical_crossentropy', optimizer=add)
  
      print("Fitting AlexNet...")
      model.fit(Xtrain, Ytrain, batch_size=batch_size, nb_epoch=nb_epoch,
        show_accuracy=True, verbose=1, shuffle=True) 
  </code></pre>
  
  <p>Y is that?? </p>