title: How to maximize CPU utilization in tensorflow GPU training with Dataset API?
body: <p>I am using the new dataset api to train a simple feed-forward DL model. I am interested in maximizing training speed. Since my network size isn't huge, as expected I see low GPU utilization. That is fine. But what I don't understand is why CPU usage is also far from 100%. I am using a multi-cpu/gpu core machine. Currently I get up to 140 steps / sec where batch_size = 128. If I cache the dataset I can get up to 210 steps (after initial scan). So I expect that with sufficient prefetching, I should be able to reach the same speed without caching. However with various prefetching and prefetch_to_device parameters, I cannot get more than 140 steps / sec. I also set num_parallel_calls to the number of cpu cores, which improves by about 20%.</p>
  
  <p>Ideally I'd like the prefetching thread to be on a disjoint cpu core from the rest of the input pipeline, so that whatever benefit it provides is strictly additive. But from the cpu usage profiling I suspect that the prefetching and input processing occur on every core:</p>
  
  <p><a href="https://i.stack.imgur.com/tM1XE.png" rel="noreferrer"><img src="https://i.stack.imgur.com/tM1XE.png" alt="enter image description here"></a></p>
  
  <p>Is there a way to have more control over cpu allocation? I have tried prefetch(1), prefetch(500), and several other values (right after batch or at the end of the dataset construction), as well as in combination with prefetch_to_device(gpu_device, batch_size = None, 1, 500, etc). So far prefetch(500) without prefetch_to_device works the best.  </p>
  
  <p>Why doesn't prefetch try to exhaust all the cpu power on my machine? What are other possible bottlenecks in training speed?</p>
  
  <p>Many thanks!</p>