title: Tensorflow: Modern way to load large data
body: <p>I want to train a convolutional neural network (using tf.keras from Tensorflow version 1.13) using numpy arrays as input data. The training data (which I currently store in a single &gt;30GB '.npz' file) does not fit in RAM all at once. <strong>What is the best way to save and load large data-sets into a neural network for training?</strong> Since I didn't manage to find a good answer to this (surely ubiquitous?) problem, I'm hoping to hear one here. Thank you very much in advance for any help!</p>
  <h3>Sources</h3>
  <p>Similar questions seem to have been asked many times (e.g. <a href="https://stackoverflow.com/questions/49169016/training-classifier-from-tfrecords-in-tensorflow">training-classifier-from-tfrecords-in-tensorflow</a>, <a href="https://stackoverflow.com/questions/40467990/tensorflow-synchronize-readings-from-tfrecord">tensorflow-synchronize-readings-from-tfrecord</a>, <a href="https://stackoverflow.com/questions/51357223/how-to-load-data-parallelly-in-tensorflow">how-to-load-data-parallelly-in-tensorflow</a>) but are several years old and usually contain no conclusive answer.</p>
  <p>My current understanding is that using TFRecord files is a good way to approach this problem. The most promising tutorial I found so far explaining how to use TFRecord files with keras is <a href="https://medium.com/@moritzkrger/speeding-up-keras-with-tfrecord-datasets-5464f9836c36" rel="noreferrer">medium.com</a>. Other helpful sources were <a href="http://machinelearninguru.com/deep_learning/data_preparation/tfrecord/tfrecord.html" rel="noreferrer">machinelearninguru.com</a> and <a href="https://medium.com/mostly-ai/tensorflow-records-what-they-are-and-how-to-use-them-c46bc4bbb564" rel="noreferrer">medium.com_source2</a> and sources therin.</p>
  <p>The official tensorflow documentation and tutorials (on <a href="https://www.tensorflow.org/api_docs/python/tf/data/Dataset" rel="noreferrer">tf.data.Dataset</a>, <a href="https://www.tensorflow.org/guide/datasets" rel="noreferrer">Importing Data</a>, <a href="https://www.tensorflow.org/tutorials/load_data/tf_records" rel="noreferrer">tf_records</a> etc.) did not help me. In particular, several of the examples given there didn't work for me even without modifications.</p>
  <h3>My Attempt at using TFRecord files</h3>
  <p>I'm assuming TFRecords are a good way to solve my problem but I'm having a hard time using them. Here is an example I made based on the tutorial <a href="https://medium.com/@moritzkrger/speeding-up-keras-with-tfrecord-datasets-5464f9836c36" rel="noreferrer">medium.com</a>. I stripped down the code as much as I could.</p>
  <pre><code># python 3.6, tensorflow 1.13.
  # Adapted from https://medium.com/@moritzkrger/speeding-up-keras-with-tfrecord-datasets-5464f9836c36
  import tensorflow as tf
  import numpy as np
  from tensorflow.python import keras as keras
  
  
  # Helper functions (see also https://www.tensorflow.org/tutorials/load_data/tf_records)
  def _int64_feature(value):
      return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))
  
  
  def _bytes_feature(value):
      return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))
  
  
  def writeTFRecords():
      number_of_samples = 100  # create some random data to play with
      images, labels = (np.random.sample((number_of_samples, 256, 256, 1)), np.random.randint(0, 30, number_of_samples))
  
      writer = tf.python_io.TFRecordWriter(&quot;bla.tfrecord&quot;)
  
      for index in range(images.shape[0]):
          image = images[index]
          label = labels[index]
  
          feature = {'image':  _bytes_feature(tf.compat.as_bytes(image.tostring())),
                     'label':  _int64_feature(int(label))}
  
          example = tf.train.Example(features=tf.train.Features(feature=feature))
          writer.write(example.SerializeToString())
      writer.close()
  
  
  def loadTFRecord(data_path):
      with tf.Session() as sess:
          feature = {'train/image': tf.FixedLenFeature([], tf.string),
                     'train/label': tf.FixedLenFeature([], tf.int64)}
          # Create a list of filenames and pass it to a queue
          filename_queue = tf.train.string_input_producer([data_path], num_epochs=1)
          # Define a reader and read the next record
          reader = tf.TFRecordReader()
          _, serialized_example = reader.read(filename_queue)
          # Decode the record read by the reader
          features = tf.parse_single_example(serialized_example, features=feature)
          # Convert the image data from string back to the numbers
          image = tf.decode_raw(features['train/image'], tf.float32)
  
          # Cast label data into int32
          label = tf.cast(features['train/label'], tf.int32)
          # Reshape image data into the original shape
          image = tf.reshape(image, [256, 256, 1])
  
          return image, label  # I'm not 100% sure that's how this works...
  
  
  # ######### generate a TFRecords file in the working directory containing random data. #################################
  writeTFRecords()
  # ######## Load the TFRecords file and use it to train a simple example neural network. ################################
  image, label = loadTFRecord(&quot;bla.tfrecord&quot;)
  
  model_input = keras.layers.Input(tensor=image)
  model_output = keras.layers.Flatten(input_shape=(-1, 256, 256, 1))(model_input)
  model_output = keras.layers.Dense(16, activation='relu')(model_output)
  
  train_model = keras.models.Model(inputs=model_input, outputs=model_output)
  train_model.compile(optimizer=keras.optimizers.RMSprop(lr=0.0001),  
                      loss='mean_squared_error',
                      target_tensors=[label])
  
  print(&quot;\n \n start training \n \n&quot;) # Execution gets stuck on fitting
  train_model.fit(epochs=1, steps_per_epoch=10)  # no output or error messages.
  
  </code></pre>
  <p>The code creates a TFRecord file and starts fitting, then just gets stuck with no output or error messages. I don't know what the problem is or how I could try to fix it.</p>