title: tf and tf.keras Dense layer shows completely different behavior in my setup
body: <p>While using tensorflow 1.14, I noticed some very strange behavior when using tf.layers.Dense vs tf.keras.layers.Dense. People on Stackoverflow say that these two layers are exactly the same, and I basically would agree, but having a look at the discounted reward while training an AC agent results in the following graph:</p>
  
  <p><img src="https://code.julien.li/so-tf-problem.png" alt="tf vs tf.keras"></p>
  
  <p>The arguments are exactly the same. Repeated runs lead to the same result (see differently colored data in image). As far as I understand the code, one of the Dense layers inherits from the other: <a href="https://github.com/tensorflow/tensorflow/blob/v1.14.0/tensorflow/python/keras/layers/core.py#L913" rel="nofollow noreferrer">tf.keras.layers.core</a> and <a href="https://github.com/tensorflow/tensorflow/blob/v1.14.0/tensorflow/python/layers/core.py#L33" rel="nofollow noreferrer">tf.layers.core</a>.</p>
  
  <p>Is anyone able to explain this behavior?</p>