# Why is the change in cost wrt bias in neural network equal to error in the neuron?

While reading the book on neural networks by Michael Nielson, I had a problem understanding equation (BP3), which is

$$\frac{\partial C}{\partial b_{j}^{l}}=\delta_{j}^{l} \tag{BP3}\label{BP3},$$ which can be translated to plain English as follows

The change in cost, $$C$$, with respect to the bias of a neuron, $$b_{j}^{l}$$, is equal to the error in that neuron, $$\delta_{j}^{l}$$.

Why would this be the case?

This is just an application of the chain rule. The same chapter has "Proof of the four fundamental equations" section, which proves BP1-2, while PB3-4 are left as exercise to the reader. I agree that it's a good exercise indeed, that's why I encourage you to stop here and try to prove it yourself using a chain rule.

Now, if you decided to read further, here's the sketch of a proof.

Recall equations (25) and (29), both definitions:

• $$z^l$$ is a linear transformation of $$a^{l-1}$$: $$z^l = w^l a^{l-1} + b^l$$

• $$\delta^l$$ is a partial derivative of $$C$$ with respect to $$z^l$$: $$\delta^l = \dfrac{\partial C}{\partial z^l}$$

The chain rule itself:

• the partial derivative of $$C$$ with respect to $$b^l$$:

$$\frac{\partial C}{\partial b^l_j} = \sum_k \frac{\partial C}{\partial z^l_k} \frac{\partial z^l_k}{\partial b^l_j}$$

Almost all elements in this sum will be zero, except for one when $$k=j$$. The first term in it is by definition $$\delta^l_j$$, the second term $$\dfrac{\partial z^l_j}{\partial b^l_j}$$ is $$1$$, because $$z$$ is linear with respect to $$b$$.

The derivative with respect to weights is taken the same way, and differs only in the last term: $$z$$ is linear with respect to $$w$$ as well, but with a coefficient $$a^{l-1}$$.
{"found_math": true, "script_math_tex": 0, "script_math_asciimath": 0, "math_annotations": 0, "math_alttext": 0, "mathml": 0, "mathjax_tag": 0, "mathjax_inline_tex": 0, "mathjax_display_tex": 0, "mathjax_asciimath": 0, "img_math": 0, "codecogs_latex": 0, "wp_latex": 0, "mimetex.cgi": 0, "/images/math/codecogs": 0, "mathtex.cgi": 0, "katex": 0, "math-container": 23, "wp-katex-eq": 0, "align": 0, "equation": 0, "x-ck12": 0, "texerror": 0}