Neural network diverging instead of converging
With neural networks, you always need to randomly initialize your weights to break symmetry.
If you don't use a non-linear activation function in the hidden units, then you might as well have stayed with a single layer. Your network is now just a composition of two linear functions, which is of course just another linear function.
That learning rate seems excessively small. If I'm using a fixed learning rate I normally find a value somewhere between 0.1 and 0.0001 to work well for most problems. This is obviously problem dependent so take my experience for what it is.
In addition to checking your code against the math you've learned, when doing gradient based optimization it can be very helpful to analytically calculate the gradients required using finite differences and compare them to the values you're computing in your code. See here for a discussion of how you can do this. I've caught many errors this way and seeing these types of tests pass always helps me feel much more sure of code correctness.