My first machine learning experiment , model not converging , tips? I wanted to recreate the model mentioned in this paper:https://arxiv.org/pdf/1610.09204v1.pdf . I am using keras with tensorflow backend, and a gtx 1050ti.

I am an ML beginner, and thought this would be a good way to get a hands on feel for things. However, My model is not converging(loss is same as first epoch).
This is what I read from that paper:


  The first convolutional layer re- ceives an input of 56px by 56px
  images with RGB channels. It uses 32 filters of size 5×5×3, stride 1
  and then sampled with max pooling of size 2 × 2, stride 1. The second
  convolutional layer has 64 filters of size 5×5×32, stride 1 and a max
  pooling of size 2 × 2, stride 1. The results of the second max pooling
  provide the first fully-connected layer with a vector of length 12,544
  (14 × 14 × 64) which are used by 512 neurons. The final
  fully-connected output layer uses a 20-wide softmax [21] which
  represents the probability of each respective 20 class labels. This
  architecture is similar to the LeNet model [3], but with using
  rectified linear unit (ReLU) [22] activation functions instead of
  sigmoid activation functions. We also use dropout [23], a technique to
  prevent overfitting, with a keep probability of 0.5 for the
  fully-connected layers.


and my code is:

model = Sequential()
model.add(Convolution2D(32, 5, 5, border_mode='same', input_shape=(70,52, 3))) model.add(Activation("relu")) model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Convolution2D(64, 5, 5, border_mode='same')) model.add(Activation("relu")) model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Flatten()) model.add(Dense(output_dim=512)) model.add(Activation("relu"))
model.add(Dense(output_dim=2))
model.add(Activation("softmax")) model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(x_train, y_train, nb_epoch=70, batch_size=500,verbose=1)


the full code can be found here : https://gist.github.com/harveyslash/5c98f9fdab0d53a2a48f477a52d8588d
I have scrapped the data from goodreads
Help appreciated !

EDIT

I forgot to actually ask what i wanted. Since its my first experiment, i would like to ask what are some things that I should do to make my model converge.
You may try Stochastic Gradient Descent optimizer with a learning rate decay and nesterov momentum. You can also try a different batch_size. Also you are missing drop out layers between the fully connected layers which the authors used.

Try

...
# flatten the conv layers
model.add(Flatten())
model.add(Dropout(0.5))

# fully connected 512
model.add(Dense(output_dim=512)) 
model.add(Activation("relu"))
model.add(Dropout(0.5))

# fully connected output layer
model.add(Dense(output_dim=2))
model.add(Activation("softmax")) 

# compile
model.compile(loss='categorical_crossentropy', 
              optimizer=SGD(lr=10e-4, 
                            decay=10e-6, 
                            momentum=0.99, 
                            nesterov=True), 
              metrics=['accuracy'])

# train
model.fit(x_train, y_train, 
          nb_epoch=70, 
          batch_size=64,
          verbose=1)


In my experience this usually helps a lot. If you get this setting to converge, you may try rmsprop and adam.