title: Dataset `from_generator` poor performance with buffered staging area
body: <p><strong>Short version:</strong><br>
  Context: The start of my training data input pipeline is a <code>tf.data.Dataset.from_generator</code> call.</p>
  
  <p>I am running into unexpectedly poor performance when using a buffered staging area rather than streaming data directly through a python generator.  The buffered staging area is filled by multiple producers (in C++), and consumed via python generator. Performance is substantially better when streaming the data through the python generator in a single-thread synthesize-data-as-needed fashion, despite the higher raw throughput of the buffered generator. I'm not sure why this is the case.</p>
  
  <p><strong>Context &amp; (semi)Specifics:</strong><br>
  We are using synthetic images created in C++ as sample data for training. The data synthesis program is thread-safe, so throughput can be improved with concurrent generation. Python (CPython) has its nasty GIL, so we implemented a buffered staging area in C++. This staging area is filled by multiple producers, and consumed from via our python generator (C++ -> python transition using <code>ctypes</code>). </p>
  
  <p>The intent behind this program structure is to improve the speed of data IO by minimizing the work being done in the python generator.</p>
  
  <p>This works great in simple cases when testing the raw throughput of the python generator. It also functions appropriately when working within the context of a trivial Tensorflow program, which just creates a dataset from this generator and iterates through N examples. </p>
  
  <p>When plugged into our actual training pipeline, however, the buffered generator appears to function far less appropriately. </p>
  
  <p>The producer threads fill the buffer very quickly into the lifetime of the Tensorflow training program, and are just waiting around most of the time (not busy waiting). My understanding is that IO in this pipeline would be trivial once the buffered area is full, as all the python generator needs to do is pull one sample from its nice supply of pre-formatted samples and <code>yield</code> it on over to the Tensorflow runtime (via <code>tf.data.Dataset.from_generator</code>). </p>
  
  <p>The issue is that when this generator is plugged into our <em>actual</em> Tensorflow training program, throughput is substantially lower than it had been in the past with static data, or with an unbuffered version of the data generator (buffered version is about 40% as fast as unbuffered version when used in non-toy TF runtime). The buffered staging area has terrible throughput irrespective of the number of producer threads. </p>
  
  <p>I'm currently at a loss for the source of this behavior, and I was wondering if anybody out there had any insight into what could be going on. As far as I can tell, the generator function is behaving as it should, with an appropriate amount of threads doing an appropriate amount of work at the appropriate time. On the Tensorflow side of things, however, I have a less clear picture of how exactly things are working. </p>
  
  <p><strong>Other info:</strong><br>
  I'm thinking it could have something to do with the following note provided from the Tensorflow Dataset API (subheading: from_generator):</p>
  
  <blockquote>
    <p>NOTE: If generator depends on mutable global variables or other external state, be aware that the runtime may invoke generator multiple times (in order to support repeating the Dataset) and at any time between the call to Dataset.from_generator() and the production of the first element from the generator. Mutating global variables or external state can cause undefined behavior, and we recommend that you explicitly cache any external state in generator before calling Dataset.from_generator().</p>
  </blockquote>
  
  <p>There is some external state associated with this python generator, seeing as how it's essentially popping from a MPSC buffer (which is being filled in separate threads), but I don't quite understand how/why this state in particular could be causing problems. </p>
  
  <p>I recognize that a custom Tensorflow op is another option, but I'd also like to get to the root cause of this issue if possible.</p>
  
  <p>I can include some source code if requested, but I was more so concerned that there could be a flaw in the underlying approach. </p>
  
  <p><strong>My question is:</strong><br>
  Is there something inherently wrong with this program design, or is this behavior likely just a result of some bug in my code?</p>
  
  <p>Thanks for the help, and let me know if any more details are required. </p>