title: Validation loss bounces randomly when training a Keras model regardless of the used optimiser
body: <p>I am retraining InceptionV3 model on 200 images and I am using Adam optimiser:  </p>
  
  <pre><code>opt = Adam(lr=0.0001, decay=0.0001 / 100)
  </code></pre>
  
  <p>I noticed the loss bounces specially the validation. I thought that is down to the learning rate as I saw in some answers like <a href="https://stackoverflow.com/questions/56430566/transfer-learning-val-loss-strange-behaviour">Transfer Learning - Val_loss strange behaviour</a>, and also <a href="https://stackoverflow.com/questions/38728895/why-is-it-possible-to-have-low-loss-but-also-very-low-accuracy-in-a-convolutio,but">Why is it possible to have low loss, but also very low accuracy, in a convolutional neural network?</a> they were not helpful. </p>
  
  <p>so I used RMSprop, but I had the same behaviour. Here is how the performance looks like:</p>
  
  <p><a href="https://i.stack.imgur.com/HAmSN.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/HAmSN.png" alt="enter image description here"></a></p>
  
  <p>Any suggestions why am I experiencing this and how to tackle it? </p>