A good reference for the back propagation algorithm? I'm trying to learn more about the fundamentals of neural networks. I feel like I understand the basics of back propagation, but I want to solidify the details in my mind.

I was working through Ian Goodfellow's famous Deep Learning text. However, I found their exposition of back propagation unsatisfactory. They model the computational graph the wrong way, making the weights into the vertices and the operators into the edges. But this makes no sense, as it is the operations (not the variables) which must have an in-degree and out-degree to them. 

I was wondering if there is a better reference, either in a textbook, a paper, or a blog that would rigorously outline the details of back propagation in its full generality.
Perhaps we can forget the graph idea to start and just go through the mathematics of it and then we will map our computations to a type of graph just so that it makes it easier to scale. 

One neuron network is called the perceptron

The first neural networks only had a single neuron which took in some inputs $x$ and then provide an output. A common function used is the sigmoid function 

$\sigma(z) = \frac{1}{1+exp(z)}$

$\sigma(w^Tx) = \frac{1}{1+exp(w^Tx + b)}$

where $w$ is the associated weight for each input $x$ and we have a bias $b$.

How do we train these weights

We will use gradient descent to train the weights based on the output of the sigmoid function and we will use some cost function $C$ and train on batches of data of size $N$. 

$C = \frac{1}{2N} \sum_i^N (\hat{y} - y)^2$

$\hat{y}$ is the predicted class obtained from the sigmoid function and $y$ is the ground truth label. We will use gradient descent to minimize the cost function with respect to the weights $w$. To make life easier we will split the derivative as follows

$\frac{\partial C}{\partial w} = \frac{\partial C}{\partial \hat{y}} \frac{\partial \hat{y}}{\partial w}$.

$\frac{\partial C}{\partial \hat{y}} = \hat{y} - y$

and we have that $\hat{y} = \sigma(w^Tx)$ and the derivative of the sigmoid function is $\frac{\partial \sigma(z)}{\partial z} = \sigma(z)(1-\sigma(z))$ thus we have,

$\frac{\partial \hat{y}}{\partial w} = \frac{1}{1+exp(w^Tx + b)} (1 - \frac{1}{1+exp(w^Tx + b)})$.

So we can then update the weights through gradient descent as 

$w^{new} = w^{old} - \eta \frac{\partial C}{\partial w}$

where $\eta$ is the learning rate.

What if we have 2 layers?

Take a look at the following answer: Compute backpropagation



Why model neural networks as a graph

In graph theory nodes are connected by vertices. Each neuron takes in some inputs and produces an output. The inputs to this node are not the same as the outputs of the previous layer. They are affected by some weight. Thus we can consider these weights to be the vertices of the graph that link nodes together.