<p dir="auto">Hi,<br>
I'm using Tensorflow <a href="http://ci.tensorflow.org/view/Nightly/job/nigntly-matrix-linux-gpu/TF_BUILD_CONTAINER_TYPE=GPU,TF_BUILD_IS_OPT=OPT,TF_BUILD_IS_PIP=PIP,TF_BUILD_PYTHON_VERSION=PYTHON2,label=gpu-working/lastSuccessfulBuild/" rel="nofollow">last succeed build #85</a></p>
<p dir="auto">I succeed to train a classifier with batch normalization and saving it with classifier.save(logdir) and restoring the classifier with<br>
classifier = tf.contrib.skflow.TensorFlowEstimator.restore(logdir)</p>
<p dir="auto">the classifier is restored but when using prediction classifier.predict(X_test) it seems that the batch normalization is applied as is in Training mode.</p>
<p dir="auto">Looking to the code of batch normalization (batch_norm_ops.py)<br>
<code class="notranslate">is_training = array_ops_.squeeze(ops.get_collection("IS_TRAINING")) mean, variance = control_flow_ops.cond(is_training, update_mean_var, lambda: (ema_mean, ema_var))</code></p>
<p dir="auto">I suppose as the graph is loaded in restore step, the "IS_TRAINING" is defined at the state of True at the saved step. and the cond function used will be update_mean_var at predict step , however the "IS_TRAINING" is changed at _predict step.<br>
you can detect the problem when classifying 2 class problem and adding a batch normalization before the softmax the mean of the softmax of any tested data will be 0.</p>
<p dir="auto">Maybe a solution will be to use a placeholder in place of the variable is_training</p>
<p dir="auto">Or I'm missing something ?</p>
<p dir="auto"><g-emoji class="g-emoji" alias="arrow_forward" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/25b6.png">▶</g-emoji> Operating System:<br>
ubunutu 14.04 LTS</p>
<p dir="auto"><g-emoji class="g-emoji" alias="arrow_forward" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/25b6.png">▶</g-emoji> Installed version of CUDA and cuDNN:<br>
CUDA 7.5.18 / cuDNN 7.0(v4)<br>
NVIDIA-SMI 352.39     Driver Version: 352.39<br>
GeForce GTX TITAN X</p>
<p dir="auto"><g-emoji class="g-emoji" alias="arrow_forward" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/25b6.png">▶</g-emoji> I installed tensorflow with next command line. And that is a link of tensorflow Github, 'Linux GPU, python 2' whl file.</p>
<p dir="auto">sudo pip install --upgrade <a href="http://ci.tensorflow.org/view/Nightly/job/nigntly-matrix-linux-gpu/TF_BUILD_CONTAINER_TYPE=GPU,TF_BUILD_IS_OPT=OPT,TF_BUILD_IS_PIP=PIP,TF_BUILD_PYTHON_VERSION=PYTHON2,label=gpu-working/lastSuccessfulBuild/artifact/pip_test/whl/tensorflow-0.8.0-cp27-none-linux_x86_64.whl" rel="nofollow">http://ci.tensorflow.org/view/Nightly/job/nigntly-matrix-linux-gpu/TF_BUILD_CONTAINER_TYPE=GPU,TF_BUILD_IS_OPT=OPT,TF_BUILD_IS_PIP=PIP,TF_BUILD_PYTHON_VERSION=PYTHON2,label=gpu-working/lastSuccessfulBuild/artifact/pip_test/whl/tensorflow-0.8.0-cp27-none-linux_x86_64.whl</a></p>
<p dir="auto"><g-emoji class="g-emoji" alias="arrow_forward" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/25b6.png">▶</g-emoji> TensorFlow version: 0.8.0<br>
(outputs of python -c "import tensorflow; print(tensorflow.<strong>version</strong>)" )</p>
<hr>
<h3 dir="auto">Steps to reproduce</h3>
<ol dir="auto">
<li>Save and Restore a tfmodel modifying [tensorflow/tensorflow/examples/skflow/mnist.py]<br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/4004593/14917371/a93e2882-0e59-11e6-8ff0-8154c0cd318f.png"><img src="https://cloud.githubusercontent.com/assets/4004593/14917371/a93e2882-0e59-11e6-8ff0-8154c0cd318f.png" alt="image" style="max-width: 100%;"></a></li>
<li>And see the accuracy is not deterministic but stochastic after restored.<br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/4004593/14917401/d8e70342-0e59-11e6-86f6-aa778b66ede2.png"><img src="https://cloud.githubusercontent.com/assets/4004593/14917401/d8e70342-0e59-11e6-86f6-aa778b66ede2.png" alt="image" style="max-width: 100%;"></a></li>
</ol>
<h3 dir="auto">What have you tried?</h3>
<p dir="auto">When I try to predict MNIST data after save() and restore() the 'tfmodel' with skflow function, the dropout shows stochastic outcomes. And when I remove the dropout, there is no stochasticity.</p>
<hr>
<p dir="auto">I guess when the model has been restored, the collections of a graph about DROPOUT are may not properly loaded. So some codes on<br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/4004593/14917208/dcbbfcc6-0e58-11e6-8b4f-8188a99fda8a.png"><img src="https://cloud.githubusercontent.com/assets/4004593/14917208/dcbbfcc6-0e58-11e6-8b4f-8188a99fda8a.png" alt="image" style="max-width: 100%;"></a><br>
▶ feed_dict = {prob: 1.0 for prob in dropouts}<br>
the function of "_predict()" at [tensorflow/contrib/learn/python/learn/estimators/base.py] may dont' work.</p>
<p dir="auto">If you run following code after restore tfmodel, then you will see that there is no 'droputs' key.<br>
<g-emoji class="g-emoji" alias="arrow_forward" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/25b6.png">▶</g-emoji> print (classifier._graph.get_all_collection_keys())</p>