title: What is causing large jumps in training accuracy and loss between epochs?
body: <p>In training a neural network in Tensorflow 2.0 in python, I'm noticing that training accuracy and loss change dramatically between epochs.  I'm aware that the metrics printed are an average over the entire epoch, but accuracy seems to drop significantly after each epoch, despite the average always increasing.  </p>
  
  <p>The loss also exhibits this behavior, dropping significantly each epoch but the average increases.  Here is an image of what I mean (from Tensorboard):</p>
  
  <p><a href="https://i.stack.imgur.com/fm6KK.png" rel="noreferrer"><img src="https://i.stack.imgur.com/fm6KK.png" alt="strange training behavior"></a></p>
  
  <p>I've noticed this behavior on all of the models I've implemented myself, so it could be a bug, but I want a second opinion on whether this is normal behavior and if so what does it mean?</p>
  
  <p>Also, I'm using a fairly large dataset (roughly 3 million examples).  Batch size is 32 and each dot in the accuracy/loss graphs represent 50 batches (2k on the graph = 100k batches). The learning rate graph is 1:1 for batches.</p>