title: My network only predicts one class
body: <p>I am trying to do a six-class classification using CNNs. The first problem I have is that the validation loss and accuracy start high. It seems that until it reaches a lower value neither training, not validation accuracy is calculated since it is stuck in the same value</p>
  
  <p><img src="https://i.stack.imgur.com/JEfpF.png" alt="(training start)">.</p>
  
  <p>Then, when it does start calculating the accuracy it goes terribly bad: </p>
  
  <p><img src="https://i.stack.imgur.com/hvBZc.png" alt="half training">. I am using the following net:</p>
  
  
  
  <pre class="lang-python prettyprint-override"><code>inp = Input(shape=input_shape)
  out = Conv2D(16, (5, 5),activation = 'relu', kernel_initializer='glorot_normal', kernel_regularizer=regularizers.l2(0.01), padding='same')(inp)
  out = MaxPooling2D(pool_size=(2, 2))(out)
  out = Dropout(0.5)(out)
  
  out = Conv2D(32, (3, 3),activation = 'relu',kernel_initializer='glorot_normal',kernel_regularizer=regularizers.l2(0.01), padding='same')(out)
  out = MaxPooling2D(pool_size=(2, 2))(out)
  out = Dropout(0.5)(out)
  
  out = Conv2D(32, (3, 3),activation = 'relu',kernel_initializer='glorot_normal',kernel_regularizer=regularizers.l2(0.01), padding='same')(out)
  out = Dropout(0.5)(out)
  
  out = Conv2D(64, (3, 3), activation = 'relu',kernel_initializer='glorot_normal',kernel_regularizer=regularizers.l2(0.01), padding='same')(out)
  out = Conv2D(64, (3, 3),activation = 'relu', kernel_initializer='glorot_normal',kernel_regularizer=regularizers.l2(0.01), padding='same')(out)
  out = MaxPooling2D(pool_size=(2, 2))(out)
  
  out = Conv2D(128, (3, 3), activation = 'relu',kernel_initializer='glorot_normal',kernel_regularizer=regularizers.l2(0.01), padding='same')(out)
  out = Conv2D(128, (3, 3),activation = 'relu', kernel_initializer='glorot_normal',kernel_regularizer=regularizers.l2(0.01), padding='same')(out)
  out = MaxPooling2D(pool_size=(2, 2))(out)
  out = Conv2D(256, (3, 3), activation = 'relu',kernel_initializer='glorot_normal',kernel_regularizer=regularizers.l2(0.01), padding='same')(out)
  out = MaxPooling2D(pool_size=(2, 2))(out)
  out = Conv2D(256, (3, 3), activation = 'relu',kernel_initializer='glorot_normal', kernel_regularizer=regularizers.l2(0.01), padding='same')(out)
  out = MaxPooling2D(pool_size=(2, 2))(out)
  out = Flatten()(out)
  out = Dropout(0.5)(out)
  dense1 = Dense(6, activation="softmax")(out)
  model = Model(inputs = inp, outputs = dense1)
  </code></pre>
  
  <p>I have checked that the labels are correct, and the images are fine as well. The network's ouput is always the same class (which by the way is the class with fewer images).</p>
  
  <p>I am using Adam optimizer with <code>lr = 1e-5</code></p>