title: Tensorflow GPU application crashes Jupyter notebook kernel
body: <p>We are running Tensorflow applications on GPU using multiple Jupyter notebooks.  Every once in a while one of the runs crashes the notebook, with the simple notification that "The kernel has crashed...".</p>
  
  <p>When we placed the code into a python .py file, the stderr output was</p>
  
  <pre><code>F tensorflow/core/kernels/conv_ops_3d.cc:369] Check failed:   stream-&gt;parent()-&gt;GetConvolveAlgorithms(&amp;algorithms)
  Aborted
  </code></pre>
  
  <p>In another run the stderr reported:</p>
  
  <pre><code>F tensorflow/core/common_runtime/gpu/gpu_util.cc:296] GPU-&gt;CPU Memcpy failed
  </code></pre>
  
  <p>The problem is that the tensorflow applications are grabbing a lot of memory.  In Linux you can run <code>top</code> to see what is going on.  On our machine we saw that each tensorflow process was grabbing <code>0.55t</code>!</p>
  
  <p>When you run the process inside a Jupyter notebook and do not shutdown the notebook, the notebook does not release the memory.  At some point you will run a process that cannot access memory and it will die.  If you are running inside a notebook it will only tell you that the kernel has died.</p>
  
  <p>Can anyone help with this?</p>