Why is the change in cost wrt bias in neural network equal to error in the neuron? While reading the book on neural networks by Michael Nielson, I had a problem understanding equation (BP3), which is
$$
\frac{\partial C}{\partial b_{j}^{l}}=\delta_{j}^{l} \tag{BP3}\label{BP3},
$$
which can be translated to plain English as follows

The change in cost, $C$, with respect to the bias of a neuron, $b_{j}^{l}$, is equal to the error in that neuron, $\delta_{j}^{l}$.

Why would this be the case?
This is just an application of the chain rule. The same chapter has "Proof of the four fundamental equations" section, which proves BP1-2, while PB3-4 are left as exercise to the reader. I agree that it's a good exercise indeed, that's why I encourage you to stop here and try to prove it yourself using a chain rule.

Now, if you decided to read further, here's the sketch of a proof. 

Recall equations (25) and (29), both definitions:


$z^l$ is a linear transformation of $a^{l-1}$: $z^l = w^l a^{l-1} + b^l$
$\delta^l$ is a partial derivative of $C$ with respect to $z^l$: $\delta^l  = \dfrac{\partial C}{\partial z^l}$


The chain rule itself:


the partial derivative of $C$ with respect to $b^l$: 


$$\frac{\partial C}{\partial b^l_j} = \sum_k \frac{\partial C}{\partial z^l_k} \frac{\partial z^l_k}{\partial b^l_j}$$

Almost all elements in this sum will be zero, except for one when $k=j$.
The first term in it is by definition $\delta^l_j$, the second term $\dfrac{\partial z^l_j}{\partial b^l_j}$ is $1$, because $z$ is linear with respect to $b$.

The derivative with respect to weights is taken the same way, and differs only in the last term: $z$ is linear with respect to $w$ as well, but with a coefficient $a^{l-1}$.