<p dir="auto"><strong>System information</strong></p>
<ul dir="auto">
<li>Have I written custom code (as opposed to using a stock example script provided in TensorFlow):Yes</li>
<li>OS Platform and Distribution (e.g., Linux Ubuntu 16.04):Win 10</li>
<li>TensorFlow installed from (source or binary):binary</li>
<li>TensorFlow version (use command below):2.0.0-alpha0</li>
<li>Python version:3.6</li>
</ul>
<p dir="auto"><strong>Describe the current behavior</strong><br>
The value estimate became an array instead of scalar float after 8000+ steps</p>
<p dir="auto"><strong>Describe the expected behavior</strong><br>
The value estimate should be a scalar float</p>
<p dir="auto"><strong>Code to reproduce the issue</strong><br>
<a href="https://github.com/SetoKaiba/ml-agents/tree/tf2">https://github.com/SetoKaiba/ml-agents/tree/tf2</a></p>
<p dir="auto"><strong>Other info / logs</strong><br>
After some deep dig, I found the problem is here. It uses the tf session to estimate the value. The value became an array instead of scalar float after 8000+ steps<br>
<a href="https://github.com/SetoKaiba/ml-agents/blob/tf2/ml-agents/mlagents/trainers/ppo/policy.py#L198">https://github.com/SetoKaiba/ml-agents/blob/tf2/ml-agents/mlagents/trainers/ppo/policy.py#L198</a></p>
<p dir="auto"><strong>System information</strong></p>
<ul dir="auto">
<li>Have I written custom code (as opposed to using a stock example script provided in TensorFlow):Yes</li>
<li>OS Platform and Distribution (e.g., Linux Ubuntu 16.04):Win 10</li>
<li>TensorFlow installed from (source or binary):binary</li>
<li>TensorFlow version (use command below):2.0.0-alpha0</li>
<li>Python version:3.6</li>
</ul>
<p dir="auto"><strong>Describe the current behavior</strong><br>
The session return an ndarray [[value_estimate]] instead of value_estimate.</p>
<p dir="auto"><strong>Describe the expected behavior</strong><br>
The session should return a scalar float. Just value_estimate. It should not be in an ndarray.<br>
Workaround: get the actual scalar float in the ndarray if it is not a float<br>
<a href="https://github.com/SetoKaiba/ml-agents/blob/tf2/ml-agents/mlagents/trainers/ppo/policy.py#L199-L200">https://github.com/SetoKaiba/ml-agents/blob/tf2/ml-agents/mlagents/trainers/ppo/policy.py#L199-L200</a></p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="if not isinstance(value_estimate, float):
            value_estimate = value_estimate[0][0]"><pre class="notranslate"><span class="pl-k">if</span> <span class="pl-c1">not</span> <span class="pl-en">isinstance</span>(<span class="pl-s1">value_estimate</span>, <span class="pl-s1">float</span>):
            <span class="pl-s1">value_estimate</span> <span class="pl-c1">=</span> <span class="pl-s1">value_estimate</span>[<span class="pl-c1">0</span>][<span class="pl-c1">0</span>]</pre></div>
<p dir="auto"><strong>Code to reproduce the issue</strong><br>
<a href="https://github.com/SetoKaiba/ml-agents/blob/tf2/ml-agents/mlagents/trainers/ppo/policy.py#L199-L200">https://github.com/SetoKaiba/ml-agents/blob/tf2/ml-agents/mlagents/trainers/ppo/policy.py#L199-L200</a><br>
The problem can be reproduced without the two lines workaround.</p>
<p dir="auto"><strong>Other info / logs</strong><br>
The value_estimate is calculated by the session and model.value. Code is lines below.<br>
<a href="https://github.com/SetoKaiba/ml-agents/blob/tf2/ml-agents/mlagents/trainers/ppo/policy.py#L198">https://github.com/SetoKaiba/ml-agents/blob/tf2/ml-agents/mlagents/trainers/ppo/policy.py#L198</a></p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="value_estimate = self.sess.run(self.model.value, feed_dict)"><pre class="notranslate"><span class="pl-s1">value_estimate</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-s1">sess</span>.<span class="pl-en">run</span>(<span class="pl-s1">self</span>.<span class="pl-s1">model</span>.<span class="pl-s1">value</span>, <span class="pl-s1">feed_dict</span>)</pre></div>
<p dir="auto">self.model.value is defined like this. The previous layer is dense layer with one units.<br>
And the tf.identity should just return a scalar float.<br>
The code is running correctly in TF 1.x.<br>
But it's working incorrectly in TF 2.0.<br>
<a href="https://github.com/SetoKaiba/ml-agents/blob/tf2/ml-agents/mlagents/trainers/models.py#L298-L299">https://github.com/SetoKaiba/ml-agents/blob/tf2/ml-agents/mlagents/trainers/models.py#L298-L299</a></p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="        value = tf.layers.dense(hidden_value, 1, activation=None)
        self.value = tf.identity(value, name=&quot;value_estimate&quot;)"><pre class="notranslate">        <span class="pl-s1">value</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">layers</span>.<span class="pl-en">dense</span>(<span class="pl-s1">hidden_value</span>, <span class="pl-c1">1</span>, <span class="pl-s1">activation</span><span class="pl-c1">=</span><span class="pl-c1">None</span>)
        <span class="pl-s1">self</span>.<span class="pl-s1">value</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">identity</span>(<span class="pl-s1">value</span>, <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">"value_estimate"</span>)</pre></div>