the following theorem provides us with the formulas needed to compute the gradient of typical ci (ththeorem gradient of training loss for given (inputoutputpair (xy)let ( thbe the output of algorithm and let (thloss(yg( th)be an almost-everywhere differentiable loss funcl tion suppose {zl al } = are the vectors obtained during the feed-forward propagation ( xal ( th)thenwe have for lc dl > - wl dl bl and where dl : /zl is computed recursively for dl- dl- > dl with dl sl zl ( proofthe scalar value is obtained from the transitions ( )followed by the mapping ( th loss(yg( th)using the chain rule (see appendix )we have dl sl (xc zl zl (xzl recall that the vector/vector derivative of linear mapping wz is given by wsee ( it follows thatsince zl wl al- bl and al (zl )the chain rule gives zl al- zl dl- > zl- zl- al- hencethe recursive formula ( )dl- zl dl- > dl zl- zl- zl using the {dl }we can now compute the derivatives with respect to the weight matrices and the biases in particularapplying the "scalar/matrixdifferentiation rule ( to zl wl al- bl givesc zl dl > - wl zl wl and zl dl bl bl zl from the theorem we can see that for each pair (xyin the training setwe can compute the gradient /th in sequential mannerby computing dl this procedure is called back-propagation since back-propagation mostly involves simple matrix multiplicationit backpropagation