title: Tensorflow on GPU uses too much memory
body: <p>I'm trying to train a very basic, small LSTM model in Tensorflow on a GTX 1080 (although I've tried other cards, too). Depending on some parameters (like the hidden state size), I get a <code>ResourceExhaustedError</code> after a pretty regular number of iterations.</p>
  
  <p>The model isn't much more than a an embedding matrix (ca. 5000*300) and a single-layer LSTM, with a final dense projection at every timestep. I have tried with batch sizes as small as 1 and a hidden state size of just 20, but still I run out of memory on an 8G card, with a total training data size of 5M. </p>
  
  <p>I can't wrap my head around why this is happening. I've obviously tried other suggestions to related problems discussed on Stackoverflow, incl. reducing the <code>per_process_gpu_memory_fraction</code> in the TF GPU options, but to no avail.</p>
  
  <p>See code here: </p>
  
  <ul>
  <li><a href="https://pastebin.com/1MSUrt15" rel="nofollow noreferrer">https://pastebin.com/1MSUrt15</a> (main training script)</li>
  <li><a href="https://pastebin.com/U1tYbM8A" rel="nofollow noreferrer">https://pastebin.com/U1tYbM8A</a> (model definition)</li>
  </ul>
  
  <p>[This doesn't include some utility scripts, so won't run alone. I also deleted some functions for the sake of shortness. The code is designed for multi-task learning, which introduces some overhead here, but the memory problems persist in single-task setups.]</p>
  
  <p>PS: one thing that I know I'm doing not 100% efficiently is storing all training data as a <code>numpy</code> array, then sampling from there and using TF's <code>feed_dict</code> to provide the data to my model. This, I believe, can slow down computation to some degree, but shouldn't cause such severe memory issues, right?</p>