title: Difference in the accuracy of an ANN whose weights are initialized by using np.random.randn and np.random.rand
body: <p>Using a tutorial on neural-networks from scratch, I created an ANN with 1 hidden layer for the MNIST dataset. In the tutorial they use <code>np.random.rand() - 0.5</code> to initialize the weights and the neural network works fine. I did the same, but when instead of using <code>rand()-0.5</code> function if I use <code>np.random.randn()</code>, the final accuracy drops significantly.</p>
  <p>I don't understand why that would happen because in <a href="https://stackoverflow.com/questions/47240308/differences-between-numpy-random-rand-vs-numpy-random-randn-in-python">this</a> question, the response was that using randn is better.</p>
  <p>Code for the network:</p>
  <pre><code>def init_params():
      W1 = np.random.rand(10, 784) - 0.5
      b1 = np.random.rand(10, 1) - 0.5
      W2 = np.random.rand(10, 10) - 0.5
      b2 = np.random.rand(10, 1) - 0.5
      return W1, b1, W2, b2
  
  def ReLU(Z):
      return np.maximum(0, Z)
  
  def dReLU(Z):
      return Z &gt; 0
  
  def softmax(Z):
      A = np.exp(Z) / sum(np.exp(Z))
      return A
  
  def forward_prop(W1, b1, W2, b2, X):
      Z1 = W1.dot(X) + b1
      A1 = ReLU(Z1)
      Z2 = W2.dot(A1) + b2
      A2 = softmax(Z2)
      return Z1, A1, Z2, A2
  
  def one_hot(Y):
      one_hot_Y = np.zeros((Y.size, Y.max() + 1))
      one_hot_Y[np.arange(Y.size), Y] = 1
      one_hot_Y = one_hot_Y.T
      return one_hot_Y
  
  def back_prop(Z1, A1, Z2, A2, W2, X, Y):
      m = Y.size
      one_hot_Y = one_hot(Y)
      dZ2 = A2 - one_hot_Y
      dW2 = 1 / m * np.dot(dZ2, A1.T)
      db2 = 1 / m * np.sum(dZ2)
      dZ1 = np.dot(W2.T, dZ2)
      dW1 = 1 / m * np.dot(dZ1, X.T)
      db1 = 1 / m *np.sum(dZ1)
      return dW1, db1, dW2, db2
  
  def update_params(W1, b1, W2, b2, dW1, db1, dW2, db2, alpha):
      W1 = W1 - alpha * dW1
      b1 = b1 - alpha * db1
      W2 = W2 - alpha * dW2
      b2 = b2 - alpha * db2
      return W1, b1, W2, b2
  
  def get_predictions(A2):
      return np.argmax(A2, 0)
  
  def get_accuracy(predictions, Y):
      print(predictions, Y)
      return np.sum(predictions == Y) / Y.size
  
  def gradient_descent(X, Y, iterations, alpha):
      W1, b1, W2, b2 = init_params()
      
      for i in range(iterations):
          Z1, A1, Z2, A2 = forward_prop(W1, b1, W2, b2, X)
          dW1, db1, dW2, db2 = back_prop(Z1, A1, Z2, A2, W2, X, Y)
          W1, b1, W2, b2 = update_params(W1, b1, W2, b2, dW1, db1, dW2, db2, alpha)
          if (i % 10 == 0):
              print('Iteration: ', i)
              print('Accuracy: ', get_accuracy(get_predictions(A2), Y))
      return W1, b1, W2, b2
  </code></pre>
  <p>Number of epochs = 500
  Learning rate = 0.15</p>
  <p>Accuracy of using <code>np.random.rand() - 0.5</code> after 500 epochs: <strong>80.6%</strong></p>
  <p>Accuracy of using <code>np.random.randn()</code> after 500 epochs: <strong>55.2%</strong></p>
  <p>It also seems that the accuracy stops changing after the 100th epoch for the method that uses <code>np.random.randn()</code></p>
  <p>The data I am using is mnist digit classification data</p>