title: Keras model learning to give wrong answers training accuracy raising to 0.8 to then deacrease dramatically down to 0.1
body: <p>I'm going through the book "Deep Learning with Python" from François Chollet.</p>
  
  <p>In section 7.9, there is an example neural network using Conv1D layers to process the IMDB dataset. What surprises me is that it starts to learn and both training and validation accuracy improve, but after a couple epochs, both the training as well as the validation accuracy drop.</p>
  
  <p>Seeing the validation accuracy drop does not surprises me, it's quite typical of overfitting, but <strong>what I don't understand is how the training accuracy can drop down to as low as 12%. It's almost as if the network was learning the opposite of what it should.</strong></p>
  
  <p>The code is:</p>
  
  <pre><code>import keras
  from keras import layers
  from keras.datasets import imdb
  from keras.preprocessing import sequence
  from keras.optimizers import Adam, RMSprop
  
  max_features = 2000
  max_len = 500
  
  (x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features)
  x_train = sequence.pad_sequences(x_train, maxlen=max_len)
  x_test = sequence.pad_sequences(x_test, maxlen=max_len)
  
  model = keras.models.Sequential()
  model.add(layers.Embedding(max_features, 128,
                             input_length=max_len,
                             name='embed'))
  model.add(layers.Conv1D(32, 7, activation='relu'))
  model.add(layers.MaxPooling1D(5))
  model.add(layers.Conv1D(32, 7, activation='relu'))
  model.add(layers.GlobalMaxPooling1D())
  model.add(layers.Dense(1))
  
  model.summary()
  
  model.compile(optimizer=RMSprop(),
                loss='binary_crossentropy',
                metrics=['acc'])
  
  history = model.fit(x_train, y_train,
                      epochs=20,
                      batch_size=128,
                      validation_split=0.2,
                      # callbacks=callbacks,
                      )
  </code></pre>
  
  <p>And the training results are as follows:</p>
  
  <pre><code>Train on 20000 samples, validate on 5000 samples
  Epoch 1/20
  20000/20000 [==============================] - 4s 212us/step - loss: 0.7043 - acc: 0.6076 - val_loss: 0.4488 - val_acc: 0.8166
  Epoch 2/20
  20000/20000 [==============================] - 3s 151us/step - loss: 0.4509 - acc: 0.8179 - val_loss: 0.6575 - val_acc: 0.7594
  Epoch 3/20
  20000/20000 [==============================] - 3s 151us/step - loss: 0.4082 - acc: 0.7923 - val_loss: 0.4759 - val_acc: 0.7874
  Epoch 4/20
  20000/20000 [==============================] - 3s 152us/step - loss: 0.3633 - acc: 0.7526 - val_loss: 0.5385 - val_acc: 0.7356
  Epoch 5/20
  20000/20000 [==============================] - 3s 154us/step - loss: 0.3333 - acc: 0.7235 - val_loss: 0.5658 - val_acc: 0.7056
  Epoch 6/20
  20000/20000 [==============================] - 3s 152us/step - loss: 0.2793 - acc: 0.6868 - val_loss: 0.5790 - val_acc: 0.6494
  Epoch 7/20
  20000/20000 [==============================] - 3s 151us/step - loss: 0.2433 - acc: 0.6408 - val_loss: 0.6710 - val_acc: 0.5726
  Epoch 8/20
  20000/20000 [==============================] - 3s 149us/step - loss: 0.2061 - acc: 0.5789 - val_loss: 1.7192 - val_acc: 0.3538
  Epoch 9/20
  20000/20000 [==============================] - 3s 151us/step - loss: 0.1769 - acc: 0.5144 - val_loss: 0.8144 - val_acc: 0.4416
  Epoch 10/20
  20000/20000 [==============================] - 3s 151us/step - loss: 0.1507 - acc: 0.4365 - val_loss: 1.1555 - val_acc: 0.3682
  Epoch 11/20
  20000/20000 [==============================] - 3s 152us/step - loss: 0.1395 - acc: 0.3675 - val_loss: 1.1440 - val_acc: 0.3412
  Epoch 12/20
  20000/20000 [==============================] - 3s 156us/step - loss: 0.1241 - acc: 0.3159 - val_loss: 1.8202 - val_acc: 0.2686
  Epoch 13/20
  20000/20000 [==============================] - 3s 155us/step - loss: 0.1225 - acc: 0.2756 - val_loss: 1.0667 - val_acc: 0.2944
  Epoch 14/20
  20000/20000 [==============================] - 3s 152us/step - loss: 0.1183 - acc: 0.2422 - val_loss: 1.1143 - val_acc: 0.2794
  Epoch 15/20
  20000/20000 [==============================] - 3s 151us/step - loss: 0.1153 - acc: 0.2142 - val_loss: 1.1599 - val_acc: 0.2686
  Epoch 16/20
  20000/20000 [==============================] - 3s 153us/step - loss: 0.1150 - acc: 0.1930 - val_loss: 1.2467 - val_acc: 0.2544
  Epoch 17/20
  20000/20000 [==============================] - 3s 151us/step - loss: 0.1145 - acc: 0.1766 - val_loss: 1.1953 - val_acc: 0.2492
  Epoch 18/20
  20000/20000 [==============================] - 3s 153us/step - loss: 0.1115 - acc: 0.1508 - val_loss: 1.4812 - val_acc: 0.2226
  Epoch 19/20
  20000/20000 [==============================] - 3s 156us/step - loss: 0.1119 - acc: 0.1355 - val_loss: 1.2690 - val_acc: 0.2288
  Epoch 20/20
  20000/20000 [==============================] - 3s 155us/step - loss: 0.1127 - acc: 0.1248 - val_loss: 1.2903 - val_acc: 0.2148
  </code></pre>
  
  <p>I could of course stop it early while the validation accuracy is at the top, but what I would is to understand how the training accuracy can drop so dramatically. And more surprisingly, how it can drop below 0.5, being a softmax 0/1 type output layer, I would expect the accuracy to be around 0.5. It really looks like it learns to give wrong answers. </p>