Back propagation on matrix of weights
The back propagation must be done in two steps.


Transfer derivative

Given the output value, you must calculate its slope. This is simple as you are using the sigmoid activation function. So the derivative can be calculated as follows




# Calculating slope
def transfer_derivative(output):
    return output * (1.0 - output)



Back propagate the error

Calculate the errors for the various layers of the network and update the weights and bias.




# Error calculation

# For output layer
error = (expected - output) * transfer_derivative(output)

# Hidden layers
error = (weight_k * error_j) * transfer_derivative(output)


Where error_j is the error signal from the jth neuron in the output layer, weight_k is the weight that connects the kth neuron to the current neuron and output is the output for the current neuron.

And finally updating weights



weight = weight + learning_rate * error * input


Here is a detailed explanation of implementing neural networks from scratch using Python along with sample code. Give it a read.