How to parallelise python input pipeline in Distributed Tensorflow
Recently Google has released Tensorflow Extended (TFX). It essentially consists of:

A set of operators which each use Apache Beam to do data distribution (They call them components).
A standardization of both data and parameter format (what they call protobuf)
Automated dependency management of operators (workflow/orchestration)
Tracking of runs.  This allows the system to skip operations that have already been performed under the same conditions.

I would suggest either take a look at TFX.  Or, for a more modest leap, look at Apache Beam.
I have a non trivial input pipeline, which consists of reading ground truth and raw data and performing preprocessing on them, written in Python. It takes a long time to run the input pipeline for a single sample so I have multiple processes (from python multiprocessing package) running in parallel and queues to perform the operation quickly and prefetch data. The output is then fed to my network using feed_dict. The overhead of this process in my training loop is 2 orders of magnitude less than the actual tf.Session.run() time.
I'm trying to move to the tf.data API, by wrapping with tf.py_func my read+preprocess functions but it runs slowly, probably due to GIL, even when increasing the number of multiple calls.
I want to scale up my training to multiple machines and am not sure how data fetching behaves in such a case, also there's the performance issue for a single machine as well :)
So, basically my question is:
How to run python functions in tf.data api input pipeline in parallel on multiple CPU cores?