title: Why is the training accuracy fluctuating?
body: <p>I'm working with a video classification of 5 classes and using TimeDistributed CNN model in Google Colab platform. The training dataset contains 80 videos containing 5 frames each. The validation dataset contains 20 videos containing 5 frames each. The batch size I used is 64. So, in total, I'm working with 100 videos. I compiled the model using Adam optimizer and categorical cross_entropy loss.</p>
  
  <pre><code>model = Sequential()
  
  input_shape=(5, 128, 128, 3)
  
  model.add(TimeDistributed(Conv2D(32, (3, 3), strides=(1, 1),
              activation='relu', padding='same'), input_shape=input_shape))
  model.add(TimeDistributed(MaxPooling2D((2, 2))))
  model.add(TimeDistributed(Conv2D(64, (3, 3), strides=(1, 1),
              activation='relu', padding='same')))
  model.add(TimeDistributed(Conv2D(128, (3, 3), strides=(1, 1),
              activation='relu', padding='same')))
  model.add(TimeDistributed(BatchNormalization()))
  model.add(TimeDistributed(MaxPooling2D((2, 2))))
  model.add(TimeDistributed(Flatten()))
  
  model.add(GRU(64, return_sequences=False))
  
  model.add(BatchNormalization())
  
  model.add((Dense(128, activation='relu')))
  model.add(Dense(5, activation='softmax'))
  
  from tensorflow.keras.optimizers import Adam
  
  model.compile(loss='categorical_crossentropy',
                optimizer=Adam(lr=0.0001),
                metrics=['accuracy'])
  </code></pre>
  
  <p>But, after fitting this model with the dataset, the training accuracy curve is fluctuating like this:</p>
  
  <p><a href="https://i.stack.imgur.com/0qLYb.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/0qLYb.png" alt="here, 50 epoch is used"></a></p>
  
  <p>Can anyone help me out to understand the reason behind this fluctuation?</p>