neural network xor gate classification
I've solved the problem with three changes:


The weights are too small at the beginning, remove the scaling
I think the biases should also be randomly initialized:


w1 = np.random.randn(S1, S0) 
b1 = np.random.randn(S1, 1)
w2 = np.random.randn(S2, S1) 
b2 = np.random.randn(S2, 1)


Your are using a fixed learning rate of 0.03, change it to use the learning rate, and you can also increase it:



    learning_rate = 0.1 ...

    w1 = w1 - dW1 * learning_rate
    w2 = w2 - dW2 * learning_rate
    b1 = b1 - dB1 * learning_rate
    b2 = b2 - dB2 * learning_rate

The image below shpws what I get for the progression of J:



Just for curiosity, if we change the sigmoid function with the ReLU, take a look to what happens to J (in the code I didn't change the name of the defs...). The learning with ReLU is much faster.

:


    def sigmoid(z):
    return np.maximum(0,z)



    def sigmoid_derivative(x):
    x[x &lt; = 0] = 0
    x[x &gt; 0]  = 1
    return x