Help with backpropagation equations for a simple neural network with Sigmoid activation I'm trying to create a simple neural network with 2 input neurons, 1 hidden layer with 3 neurons and an output layer with single neuron and to train it for XOR problem.
 As per my understanding of neural nets, I'm planning to have in each neuron a weighted linear function of inputs and a Sigmoid activation function. So for example, the first input neuron with input x will compute c=ax+b, a being a parameter and b being its bias. Then it will feed 
c to Sigmoid function to compute z=1/(1+exp(-c)) which will be its output. Please correct me if I'm wrong in any of this. 

Now, I'm trying to follow this article to create back-propagation differential equations. It says that dc = z * (1-z) * dz where dz is the gradient of the green edge and then da = x * dc where da is the gradient computed in this iteration of backpropagation.

My question is that there are three zs (one corresponding to each outward edge) and their value(weight) will be identical (z=1/(1+exp(-c))). And I assume, then all values and gradients in the three neurons in hidden layer will also be identical. Then what's the point of having three neurons in hidden layer? What causes the randomness which can lead to three different gradients across each of the three green edges? And if they actually are different, how can I compute dc which is simply z * (1-z) * dz but there are three zs and three dzs?
The gradient is actually calculated w.r.t to the weights so that each neuron would have a different 'change in the weight' according to its previous weight. In other words, not all the 3 neurons in the hidden layer will be penalized with the same value because the initial weights at t=0 are different for these neurone. 

It is well explained here: https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/ 

I hope this short answer helped!