How do TensorFlow learning algorithms work?
When training a dense network with a single hidden layer (with $m$ hidden nodes) to do binary classification, we minimize the cost function 
$$L(W_1,W_2) = \sum_{i=1}^N \ell(y_i, \sigma(W_2 a(W_1 \hat x_i)),$$ 
where $x_i \in \mathbb R^d$ is the $i$th feature vector in the training dataset, $\hat x_i \in \mathbb R^{d+1}$ is the augmented feature vector obtained by prepending a $1$ to $x_i$, $W_1$ is an $m \times (d+1)$ matrix containing the weights and biases for the hidden layer, $a$ is a function that applies the ReLU activation function to each of its inputs (and also has an additional output which is equal to $1$), and $W_2$ is a $1 \times m$ matrix containing the weights and bias of the output layer. Here $\sigma$ is the logistic function and $\ell$ is the binary cross-entropy loss function.
Edit: Does mathjax not work on this website?
I am trying to understand how the learning algorithms used by TensorFlow work.
I have read papers on these methods in the past (a few years ago now, so apologies for my poor terminology) and they seem to claim to be able to differentiate the loss function with respect to each value within the AI's weights and biases. This would tell the learning algorithm which direction to move the AI's parameters in to reduce the loss. 
To differentiate the AI's parameters with respect to the loss function, the learning algorithm would first need to generate an equation describing this relationship. My question is: how is this equation generated?