title: Why training loss is increased at the beginning of each epoch?
body: <p>I'm training a linear regression model. I use tf.contrib.data to prepare the dataset, shuffle it and serve it in batches:</p>
  
  <pre><code>  dataset = tf.contrib.data.TFRecordDataset(filename)
    dataset = dataset.map(
        _parse_function, num_threads=16, output_buffer_size=100 * batch_size)
    dataset = dataset.repeat(5)
    dataset = dataset.shuffle(buffer_size=100000)
    dataset = dataset.padded_batch(batch_size, padded_shapes=([None], [None]))
    iterator = dataset.make_initializable_iterator()
    x_inputs, y_ = iterator.get_next()
  </code></pre>
  
  <p>The following is our training loss:
  <img src="https://i.stack.imgur.com/CJryo.png" alt="training loss"></p>
  
  <p>It is very strange that at the beginning of each epoch (iteration=100k) we have a pulse in the training loss. If the training process continues, we see the same pattern at the beginning of the following epochs.</p>