title: Parallelize tf.from_generator using tf.contrib.data.parallel_interleave
body: <p>I have a bunch of JSON array files (AVRO to be accurate) and each of them yield multiple samples for training a Keras Model. Using ideas from <a href="https://stackoverflow.com/a/47414078/1585523">@GPhilo</a> and from <a href="https://stackoverflow.com/a/47874044/1585523">@jsimsa</a>, I was able to come up with this to parallelize my input pipeline. Unable to figure out how to design the <code>generator(n)</code> to divide the work of processing files. The code fails inside <code>parse_file(f)</code> as the function expects a string file path and not a <code>Tensor</code>,</p>
  
  <pre><code>N = num_cores = 2
  files_to_process = ["f1.avro", "f2.avro", "f3.avro"]
  shuffle_size = prefetch_buffer = 1000
  batch_size = 512
  
  def generator(n):
      size = math.ceil(len(files_to_process) / N)
      start_index = n * size
      end_index = start_index + size
  
      def gen():
          # for f in files_to_process[start_index:end_index]:
          for f in tf.slice(files_to_process, start_index, size):
              yield f
  
      return gen
  
  def dataset(n):
      return tf.data.Dataset.from_generator(generator(n), (tf.string,))
  
  def process_file(f):
      examples_x, examples_y = parse_file(f)
      return examples_x, examples_y
  
  ds = tf.data.Dataset.range(N)
  ds = ds.apply(tf.contrib.data.parallel_interleave(dataset, cycle_length=N))
  ds = ds.map(process_file, num_parallel_calls=N)
  ds = ds.prefetch(prefetch_buffer)
  ds = ds.flat_map(lambda *x: tf.data.Dataset.from_tensor_slices(x))
  ds = ds.batch(batch_size).shuffle(shuffle_size)
  
  ...
  myTfKerasModel.fit( ds.make_one_iterator(), NUM_TRAIN_SAMPLES // batch_size )
  </code></pre>
  
  <ul>
  <li>What is the correct way to design <code>generator(n)</code> here</li>
  <li>Is this an optimized way to design my input pipeline using <code>parallel_interleave</code> and <code>flat_map</code></li>
  </ul>