Why can it be that my neural network is predicting the contrary? I did the coursera deep learning course where as an assignment you have to complete a few functions of a neural net. Everything worked great so I tried to implement it from scratch.

Did it all and when I tested it it behaved really strange. I removed the hidden layers and left only one neuron on the output one so its basically logistic regression.

I'm training it with the following:

X = [[-3], [-2], [3], [2]]

Y = [[0], [0], [1.], [1.]]

but when I use that same X to check, it predicts aprox. [[1.], [1.], [0], [0]]

the bias is close to 0 and the single weight is -27. After each iteration the cost increases from 0.7 to 29.9 where it stays for the last 7 epochs.

If I change Y to [1, 1, [0], [0]] the prediction is 0.5 for all instances, and both bias and weight is close to 0. With this Y configuration the cost stays at 0.69 through all iterations.

If I change the update line from w -= alpha * grad to += the situation is the same but all the way round.

This is my gradrient function

    def gradients(self, X, Y):
    dws = []
    dbs = []
    h, activations, zs = self.forward(X)
    da = - (np.divide(Y, h) - np.divide(1 - Y, 1 - h)) #dJ/dOutput
    for d in np.arange(1, len(self.weights))[::-1]:
        a = activations[d] #output for layer d
        z = zs[d] #linear for layer d
        w = self.weights[d] #weights for layer w

        dz = da * a * (1-a)
        m = w.shape[1]
        dw = 1./m * dz.dot(activations[d].T)
        db = 1./m * np.sum(dz, axis=1, keepdims=True)
        da = w.T.dot(dz)
        dws.insert(0, dw)
        dbs.insert(0, db)
    dws.insert(0, None)
    dbs.insert(0, None)
    return dws, dbs


I'm using an empty weight at 0 so its easier to match the layer's activations, zs and weights.

For all layers the activation function is sigmoid. Once it works I'll see if I change the hidden to ReLu.

Here is the whole code:
GitHub
After being all day trying to figure this out I had to post this question to find clarity.
The problem is here

dw = 1./m * dz.dot(activations[d].T)


it should be 

dw = 1./m * dz.dot(activations[d-1].T)