title: How to efficiently shuffle a large tf.data.Dataset when using tf.estimator.train_and_evaluate?
body: <p>The <a href="https://www.tensorflow.org/api_docs/python/tf/estimator/train_and_evaluate" rel="nofollow noreferrer"><code>tf.estimator.train_and_evaluate</code></a> documentation makes it clear that the input dataset must be properly shuffled for the training to see all examples:</p>
  
  <blockquote>
    <p>Overfitting: In order to avoid overfitting, it is recommended to set up the training input_fn to shuffle the training data properly. It is also recommended to train the model a little longer, say multiple epochs, before performing evaluation, as the input pipeline starts from scratch for each training. It is particularly important for local training and evaluation.</p>
  </blockquote>
  
  <p>In my application, I would like to uniformly sample examples from the full <a href="https://www.tensorflow.org/api_docs/python/tf/data/Dataset" rel="nofollow noreferrer"><code>tf.data.Dataset</code></a> with arbitrary evaluation frequency and <a href="https://www.tensorflow.org/api_docs/python/tf/data/Dataset#shuffle" rel="nofollow noreferrer"><code>shuffle()</code></a>'s buffer size. Otherwise, the training can at most see the first:</p>
  
  <pre><code>(steps_per_second * eval_delay * batch_size) + buffer_size
  </code></pre>
  
  <p>elements, effectively discarding the rest. Is there an efficient way to work around that without loading the complete dataset in the system memory?</p>
  
  <p>I considered <a href="https://www.tensorflow.org/api_docs/python/tf/data/Dataset#shard" rel="nofollow noreferrer">sharding</a> the dataset based on the buffer size, but if the evaluation does not occur frequently, it will iterate on the same shard multiple times (a <code>repeat()</code> closes the pipeline). Ideally, I would like to move to another shard after a complete iteration over the dataset, is that possible?</p>
  
  <p>Thanks for any pointers!</p>