title: Precison issue with sigmoid activation function for Tensorflow/Keras 2.3.1
body: <p>Assuming the following forward pass in a classic ANN
  (Based on <a href="https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/" rel="nofollow noreferrer">https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/</a>):</p>
  <blockquote>
  <p><img src="https://chart.googleapis.com/chart?cht=tx&amp;chl=net_%7Bh1%7D%20%3D%20w_1%20*%20i_1%20%2B%20w_2%20*%20i_2%20%2B%20b_1%20*%201" alt="net_{h1} = w_1 * i_1 + w_2 * i_2 + b_1 * 1" /></p>
  </blockquote>
  <blockquote>
  <p><img src="https://chart.googleapis.com/chart?cht=tx&amp;chl=net_%7Bh1%7D%20%3D%200.15%20*%200.05%20%2B%200.2%20*%200.1%20%2B%200.35%20*%201%20%3D%200.3775" alt="net_{h1} = 0.15 * 0.05 + 0.2 * 0.1 + 0.35 * 1 = 0.3775" /></p>
  </blockquote>
  <p>Now let's use a sigmoid activation on that, I get:</p>
  <blockquote>
  <p><img src="https://chart.googleapis.com/chart?cht=tx&amp;chl=out_%7Bh1%7D%20%3D%20%5Cfrac%7B1%7D%7B1%2Be%5E%7B-net_%7Bh1%7D%7D%7D%20%3D%20%5Cfrac%7B1%7D%7B1%2Be%5E%7B-0.3775%7D%7D%20%3D%200.593269992" alt="out_{h1} = \frac{1}{1+e^{-net_{h1}}} = \frac{1}{1+e^{-0.3775}} = 0.593269992" /></p>
  </blockquote>
  <p>So far so good, now let's check the result of this calculation in python:</p>
  <pre><code>1 / (1+ math.exp(-0.3775)) # ... = 0.5932699921071872, OK
  </code></pre>
  <p>However this is double precision and since Keras uses float32 let's calculate the same thing but with float32, I get:</p>
  <pre><code>w1 = np.array(0.15, dtype=np.float32)
  i1 = np.array(0.05, dtype=np.float32)
  w2 = np.array(0.2, dtype=np.float32)
  i2 = np.array(0.1, dtype=np.float32)
  b1 = np.array(0.35, dtype=np.float32)
  k1 = np.array(1, dtype=np.float32)
  k2= np.array(1, dtype=np.float32)
  
  n1 = w1 * i1 + w2 * i2 + b1 * k1
  np.array(k2 / (1 + np.exp(-n1)), dtype=np.float32) # ---&gt;array(0.59327, dtype=float32)
  </code></pre>
  <p>Ok so normally I should be able to get 0.59327 for our <em><strong>out_{h1}</strong></em> using Keras float32 framework.</p>
  <p>Let's now try to get this result using Keras:</p>
  <pre><code>from tensorflow.keras.models import Sequential
  from tensorflow.keras.layers import Dense
  from tensorflow.keras.optimizers import SGD
  import numpy as np
  
  
  model = Sequential()
  
  model.add(Dense(2,activation='sigmoid'))                    # Build only one layer (enough for example)
  model.compile(optimizer = SGD(lr = 0.5),loss='mse')         # Don't really care for this example, simply for compilation
  input = np.array([[0.05, 0.10]])                            # Set input, same as the ones provided in the example
  output = np.array([[0.01, 0.99]])                           # Don't really care for this example since we want to check activation only
  
  
  
  weights = [np.array([[0.15,  0.20 ],                        # Required for set_weights
          [0.25,  0.30]], dtype=np.float32),
   np.array([0.35,  0.35], dtype=np.float32)]
  model.build(input.shape)                                    # Required for set_weights
  model.set_weights(weights)                                  # Set the weights to be the same as the one provided in the example
  
  
  model.predict(np.array([[0.05, 0.10]], dtype=np.float32))   # This can be seen as out_{h1}
                                                              # array([[0.5944759 , 0.59628266]], dtype=float32)
                                                              # NOK: 0.5944759 != 0.59327
  </code></pre>
  <p>Can someone explain to me why I get 0.5944759 instead of 0.59327? The result seem far from the expected ouput and if possible provide an example of calculation and/or a way to get the expected output of 0.59327.</p>
  <p>Please note this example was done using:</p>
  <ul>
  <li>tensorflow                2.3.1</li>
  <li>numpy                     1.18.5</li>
  <li>python                    3.8.12</li>
  </ul>
  <p>Thx for your help.</p>