What is the benefit of splitting tfrecord file into shards?
Splitting TFRecord files into shards helps you shuffle large datasets that won't fit into memory.

Imagine you have millions of training examples saved on disk and you want to repeatedly run them through a training process. Furthermore, suppose that for each repetition of the training data (i.e. each epoch) you want to load the data in a completely random order.

One approach is to have one file per training example and generate a list of all filenames. Then at the beginning of each epoch you shuffle the list of filenames and load the individual files. The problem with this approach is that you are loading millions of files from random locations on your disk. This can be slow especially on a hard disk drive. Even a RAID 0 array will not help with speed if you are loading millions of small files from random locations. The problem gets even worse if you are accessing the files over a network connection.

Another approach is to read the training examples in sequence from one large TFRecord file and shuffle the examples in memory using a shuffle buffer. However, the shuffle buffer typically cannot be larger than the DDR memory available to your CPU. And if the shuffle buffer is significantly smaller than your dataset, then it may not adequately shuffle the data. The data may be "locally" shuffled but not "globally" shuffled. That is, examples from the beginning of the dataset may not be shuffled with examples from the end of the dataset.

A good solution is to use a balanced combination of the above two approaches by splitting your dataset into multiple TFRecord files (called shards). During each epoch you can shuffle the shard filenames to obtain global shuffling and use a shuffle buffer to obtain local shuffling. A good balance will make the shards large enough to prevent disk speed issues but will keep the shards small enough to allow for adequately shuffling by a shuffle buffer.

Here are the exact steps:


Randomly place all training examples into multiple TFRecord files (shards).
At the beginning of each epoch, shuffle the list of shard filenames.
Read training examples from the shards and pass the examples through a shuffle buffer. Typically, the shuffle buffer should be larger than the shard size to ensure good shuffling across shards.
Pass the shuffled examples into your training process.