<p dir="auto">NOTE: Only file GitHub issues for bugs and feature requests.  All other topics will be closed.</p>
<p dir="auto">For general support from the community, see <a href="https://stackoverflow.com/questions/tagged/tensorflow" rel="nofollow">StackOverflow</a>.<br>
To make bugs and feature requests more easy to find and organize, we close issues that are deemed<br>
out of scope for GitHub Issues and point people to StackOverflow.</p>
<p dir="auto">For bugs or installation issues, please provide the following information.<br>
The more information you provide, the more easily we will be able to offer<br>
help and advice.</p>
<h3 dir="auto">What related GitHub issues or StackOverflow threads have you found by searching the web for your problem?</h3>
<p dir="auto">None seem to apply, this is a new issue just introduced with 1.0.0-rc1</p>
<h3 dir="auto">Environment info</h3>
<p dir="auto">Operating System:<br>
Mac<br>
Installed version of CUDA and cuDNN:<br>
(please attach the output of <code class="notranslate">ls -l /path/to/cuda/lib/libcud*</code>):</p>
<p dir="auto">If installed from binary pip package, provide:</p>
<ol dir="auto">
<li>A link to the pip package you installed:</li>
</ol>
<p dir="auto">export TF_BINARY_URL=<a href="https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-1.0.0rc1-py3-none-any.whl" rel="nofollow">https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-1.0.0rc1-py3-none-any.whl</a></p>
<ol start="2" dir="auto">
<li>
<p dir="auto">The output from <code class="notranslate">python -c "import tensorflow; print(tensorflow.__version__)"</code>.<br>
1.0.0-rc1<br>
If installed from source, provide</p>
</li>
<li>
<p dir="auto">The commit hash (<code class="notranslate">git rev-parse HEAD</code>)</p>
</li>
<li>
<p dir="auto">The output of <code class="notranslate">bazel version</code></p>
</li>
</ol>
<h3 dir="auto">If possible, provide a minimal reproducible example (We usually don't have time to read hundreds of lines of your code)</h3>
<p dir="auto">TFLearn has incorrect implementations of <code class="notranslate">predict</code> and <code class="notranslate">predict_proba</code>, per the <a href="http://scikit-learn.org/stable/developers/contributing.html#coding-guidelines" rel="nofollow">sklearn developer's guide</a>.  The current (v1.0.0-rc1) version's <code class="notranslate">predict</code> actually performs as would be expected for <code class="notranslate">predict_proba</code> and <code class="notranslate">SKCompat</code> does not include a <code class="notranslate">predict_proba</code>.</p>
<p dir="auto">SKFLOW/TFLearn's compatibility with sklearn has been broken in various ways since when v0.8 (which worked) was refactored into SKCompat (see <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="198242109" data-permission-text="Title is private" data-url="https://github.com/tensorflow/tensorflow/issues/6584" data-hovercard-type="issue" data-hovercard-url="/tensorflow/tensorflow/issues/6584/hovercard" href="https://github.com/tensorflow/tensorflow/issues/6584">#6584</a>).</p>
<p dir="auto">The following code shows how the accuracy function fails due to this issue.  This could be fixed with an argmax call, but the core problem is that SKCompat does not properly implement the sklearn functions.</p>
<p dir="auto">Note, this code is based on the Contrib TF Learn example here:  <a href="https://www.tensorflow.org/tutorials/tflearn/" rel="nofollow">https://www.tensorflow.org/tutorials/tflearn/</a></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from sklearn import cross_validation
from sklearn import metrics
import tensorflow as tf

def main(unused_argv):
  # Load dataset.
  iris = tf.contrib.learn.datasets.load_dataset('iris')
  x_train, x_test, y_train, y_test = cross_validation.train_test_split(
      iris.data, iris.target, test_size=0.2, random_state=42)

  # Build 3 layer DNN with 10, 20, 10 units respectively.
  feature_columns = tf.contrib.learn.infer_real_valued_columns_from_input(
      x_train)
  classifier = tf.contrib.learn.DNNClassifier(
      feature_columns=feature_columns, hidden_units=[10, 20, 10], n_classes=3)
  classifier = tf.contrib.learn.SKCompat(classifier)

  # Fit and predict.
  classifier.fit(x_train, y_train, steps=200)

  # This outputs something with shape [30,3] should output single class predictions
  # See http://scikit-learn.org/stable/developers/contributing.html#coding-guidelines
  # See predict vs predict_proba
  #predictions = classifier.predict(x_test)
  #score = metrics.accuracy_score(y_test, predictions)
  #print('Accuracy: {0:f}'.format(score))

  # predict_proba does not work either:
  classifier.predict_proba(x_test)
  # Generates a: AttributeError: 'SKCompat' object has no attribute 'predict_proba'

if __name__ == '__main__':
  tf.app.run()"><pre class="notranslate"><code class="notranslate">from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from sklearn import cross_validation
from sklearn import metrics
import tensorflow as tf

def main(unused_argv):
  # Load dataset.
  iris = tf.contrib.learn.datasets.load_dataset('iris')
  x_train, x_test, y_train, y_test = cross_validation.train_test_split(
      iris.data, iris.target, test_size=0.2, random_state=42)

  # Build 3 layer DNN with 10, 20, 10 units respectively.
  feature_columns = tf.contrib.learn.infer_real_valued_columns_from_input(
      x_train)
  classifier = tf.contrib.learn.DNNClassifier(
      feature_columns=feature_columns, hidden_units=[10, 20, 10], n_classes=3)
  classifier = tf.contrib.learn.SKCompat(classifier)

  # Fit and predict.
  classifier.fit(x_train, y_train, steps=200)

  # This outputs something with shape [30,3] should output single class predictions
  # See http://scikit-learn.org/stable/developers/contributing.html#coding-guidelines
  # See predict vs predict_proba
  #predictions = classifier.predict(x_test)
  #score = metrics.accuracy_score(y_test, predictions)
  #print('Accuracy: {0:f}'.format(score))

  # predict_proba does not work either:
  classifier.predict_proba(x_test)
  # Generates a: AttributeError: 'SKCompat' object has no attribute 'predict_proba'

if __name__ == '__main__':
  tf.app.run()
</code></pre></div>
<h3 dir="auto">What other attempted solutions have you tried?</h3>
<p dir="auto"><a href="http://scikit-learn.org/stable/developers/contributing.html#coding-guidelines" rel="nofollow">http://scikit-learn.org/stable/developers/contributing.html#coding-guidelines</a></p>
<h3 dir="auto">Logs or other output that would be helpful</h3>
<p dir="auto">(If logs are large, please upload as attachment or provide link).</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="/Users/jeff/anaconda/envs/tf-latest/bin/python /Users/jeff/Dropbox/school/teaching/wustl/classes/T81_558_deep_learning/code/test3.py
/Users/jeff/anaconda/envs/tf-latest/lib/python3.5/site-packages/sklearn/cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
  &quot;This module will be removed in 0.20.&quot;, DeprecationWarning)
WARNING:tensorflow:float64 is not supported by many models, consider casting to float32.
WARNING:tensorflow:Using temporary folder as model directory: /var/folders/bs/_w74fpx157v3vs82q0fwnflr0000gn/T/tmpl7bwdnzy
WARNING:tensorflow:float64 is not supported by many models, consider casting to float32.
WARNING:tensorflow:From /Users/jeff/anaconda/envs/tf-latest/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/estimators/head.py:1362: scalar_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.
Instructions for updating:
Please switch to tf.summary.scalar. Note that tf.summary.scalar uses the node name instead of the tag. This means that TensorFlow will automatically de-duplicate summary names based on the scope they are created in. Also, passing a tensor or list of tags to a scalar summary op is no longer supported.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
WARNING:tensorflow:float64 is not supported by many models, consider casting to float32.
Traceback (most recent call last):
  File &quot;/Users/jeff/Dropbox/school/teaching/wustl/classes/T81_558_deep_learning/code/test3.py&quot;, line 52, in &lt;module&gt;
    tf.app.run()
  File &quot;/Users/jeff/anaconda/envs/tf-latest/lib/python3.5/site-packages/tensorflow/python/platform/app.py&quot;, line 44, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File &quot;/Users/jeff/Dropbox/school/teaching/wustl/classes/T81_558_deep_learning/code/test3.py&quot;, line 44, in main
    score = metrics.accuracy_score(y_test, predictions)
  File &quot;/Users/jeff/anaconda/envs/tf-latest/lib/python3.5/site-packages/sklearn/metrics/classification.py&quot;, line 172, in accuracy_score
    y_type, y_true, y_pred = _check_targets(y_true, y_pred)
  File &quot;/Users/jeff/anaconda/envs/tf-latest/lib/python3.5/site-packages/sklearn/metrics/classification.py&quot;, line 72, in _check_targets
    check_consistent_length(y_true, y_pred)
  File &quot;/Users/jeff/anaconda/envs/tf-latest/lib/python3.5/site-packages/sklearn/utils/validation.py&quot;, line 181, in check_consistent_length
    &quot; samples: %r&quot; % [int(l) for l in lengths])
ValueError: Found input variables with inconsistent numbers of samples: [30, 3]

Process finished with exit code 1"><pre class="notranslate"><code class="notranslate">/Users/jeff/anaconda/envs/tf-latest/bin/python /Users/jeff/Dropbox/school/teaching/wustl/classes/T81_558_deep_learning/code/test3.py
/Users/jeff/anaconda/envs/tf-latest/lib/python3.5/site-packages/sklearn/cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
  "This module will be removed in 0.20.", DeprecationWarning)
WARNING:tensorflow:float64 is not supported by many models, consider casting to float32.
WARNING:tensorflow:Using temporary folder as model directory: /var/folders/bs/_w74fpx157v3vs82q0fwnflr0000gn/T/tmpl7bwdnzy
WARNING:tensorflow:float64 is not supported by many models, consider casting to float32.
WARNING:tensorflow:From /Users/jeff/anaconda/envs/tf-latest/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/estimators/head.py:1362: scalar_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.
Instructions for updating:
Please switch to tf.summary.scalar. Note that tf.summary.scalar uses the node name instead of the tag. This means that TensorFlow will automatically de-duplicate summary names based on the scope they are created in. Also, passing a tensor or list of tags to a scalar summary op is no longer supported.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
WARNING:tensorflow:float64 is not supported by many models, consider casting to float32.
Traceback (most recent call last):
  File "/Users/jeff/Dropbox/school/teaching/wustl/classes/T81_558_deep_learning/code/test3.py", line 52, in &lt;module&gt;
    tf.app.run()
  File "/Users/jeff/anaconda/envs/tf-latest/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 44, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "/Users/jeff/Dropbox/school/teaching/wustl/classes/T81_558_deep_learning/code/test3.py", line 44, in main
    score = metrics.accuracy_score(y_test, predictions)
  File "/Users/jeff/anaconda/envs/tf-latest/lib/python3.5/site-packages/sklearn/metrics/classification.py", line 172, in accuracy_score
    y_type, y_true, y_pred = _check_targets(y_true, y_pred)
  File "/Users/jeff/anaconda/envs/tf-latest/lib/python3.5/site-packages/sklearn/metrics/classification.py", line 72, in _check_targets
    check_consistent_length(y_true, y_pred)
  File "/Users/jeff/anaconda/envs/tf-latest/lib/python3.5/site-packages/sklearn/utils/validation.py", line 181, in check_consistent_length
    " samples: %r" % [int(l) for l in lengths])
ValueError: Found input variables with inconsistent numbers of samples: [30, 3]

Process finished with exit code 1
</code></pre></div>
<p dir="auto">Have I written custom code: Forked network arch for FasterRCNN from <code class="notranslate">https://github.com/endernewton/tf-faster-rcnn</code><br>
OS Platform and Distribution: Ubuntu 16.04.4, x86_64 GNU/Linux<br>
TensorFlow installed from:<br>
Bazel version: 0.5.2<br>
CUDA/cuDNN version: CUDA V8.0.61, release 8.0<br>
GPU model and memory: <code class="notranslate">Nvidia Tesla K80</code>, 12 GB memory<br>
Exact procedure to reproduce: Load <code class="notranslate">graph</code> and <code class="notranslate">net</code> in GPU memory, use <code class="notranslate">tf.reset_default_graph()</code> followed by <code class="notranslate">sess.close()</code>, GPU memory not freed as seen through <code class="notranslate">nvidia-smi</code></p>
<hr>
<p dir="auto">Possible duplicate but re-opening here since it doesn't appear to have been resolved &amp; there's no way to re-open the previously filed ones:</p>
<p dir="auto">Versions Used/Tried:</p>
<ul dir="auto">
<li>System: aws EC2 (ubuntu) (with p2.xlarge elastic GPU instance)</li>
<li>GPU: <code class="notranslate">NVIDIA Tesla K80</code></li>
<li>Tensorflow versions: 1.3.0, 1.5.5 (tried on both)</li>
<li>CUDA Version: 8.0, 9.0 (tried on both)</li>
</ul>
<p dir="auto">Calling <code class="notranslate">tf.reset_graph()</code> and <code class="notranslate">sess.close()</code> doesn't free GPU as seen using <code class="notranslate">nvidia-smi</code>.<br>
It might be possible that nvidia-smi doesn't update but in that case, there wouldn't be resource errors on subsequent code trying to run on the GPU as seen here:<br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/15898956/38519614-6ad30aec-3c0e-11e8-895e-b9b2d20dab17.png"><img src="https://user-images.githubusercontent.com/15898956/38519614-6ad30aec-3c0e-11e8-895e-b9b2d20dab17.png" alt="image" style="max-width: 100%;"></a></p>
<p dir="auto">What i tried after this was to use a subprocess within my script to kill the previous processes occupying resources on the GPU as seen here:<br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/15898956/38519591-55817534-3c0e-11e8-8bae-51d91e20b98e.png"><img src="https://user-images.githubusercontent.com/15898956/38519591-55817534-3c0e-11e8-8bae-51d91e20b98e.png" alt="image" style="max-width: 100%;"></a><br>
Although this works, it seems like incredibly bad practice &amp; I shouldn't be doing this.</p>
<p dir="auto">What I tried then was to use the <code class="notranslate">numba</code> host API for interacting with the GPU to shut down processes i.e. clear <code class="notranslate">current_context</code> and then clear deallocations from the GPU like this:<br>
<code class="notranslate">from numba import cuda current_context = get_context(devnum=0) current_context.reset() cuda.current_context().deallocations.clear()</code></p>
<p dir="auto">When this didn't work out, as a last resort, i tried to use this:<br>
<code class="notranslate">from numba import cuda cuda.gpus[0].numba.cuda.cudadrv.devices.reset()</code><br>
which works but results in a substantial memory leak everytime it runs which implies that after running my code a few times, the leak accumulated to a large enough value to again give me  resource errors.</p>
<hr>
<p dir="auto">Context: I am trying to deploy a deep learning model using a flask API. Since this is a pipeline with multiple computation graphs, I cannot afford to keep all of those in memory so i need to do something like this:</p>
<ol dir="auto">
<li>Upload data &amp; store it</li>
<li>Build Graph</li>
<li>Run Inference on stored data</li>
<li>Remove graph from memory</li>
<li>Build new graph</li>
<li>Run Inference on stored data again<br>
.<br>
.<br>
....and so on</li>
</ol>
<hr>
<p dir="auto">What I suspect might be going on under the hood:<br>
It's possible that <code class="notranslate">tf.reset_graph()</code> frees memory but doesn't remove the actual process holding onto that chunk of memory in the GPU in which case it makes sense for <code class="notranslate">nvidia-smi</code> to still show me the PID occupying memory in the GPU. But shouldn't <code class="notranslate">tf.reset_graph()</code> followed by <code class="notranslate">sess.close()</code> be freeing the GPU entirely?</p>
<hr>
<p dir="auto">Any help on the issue will be appreciated.</p>