Backpropagation - softmax derivative I have a question on the backpropagation in a simple neural network (I am trying to derive the derivative for the backpropagation).

Suppose that the network is simple like so (forward pass):

$$\begin{aligned}z_1 &amp;= xW_1 + b_1  &amp; (1) \\
  a_1 &amp;= \tanh{(z_1)} &amp; (2) \\
z_2 &amp; = a_1W_2 + b_2 &amp; (3)\\
a_2 &amp; = \text{softmax}(z_2)=\hat{y}_n &amp; (4) \\
l_n &amp; = \log{(\hat{y}_n)} &amp; (5) \\
PL_n &amp; = l_n\dot{y}_n^T &amp; (6) \\
L &amp; = -\frac{1}{N} \sum_{n \in N} PL_n &amp; (7)
\end{aligned}
$$

So only one hidden layer. Let the dimensions be the following:

$x$ is $1 \times 2$, $W_1$ is $2 \times 500$, $b_1$ is $1 \times 500$, $a_1$ is $1 \times 500$, $W_2$ is $500 \times 2$, $b_2$ is $1 \times 2$, then $\log{(\hat{y_n})}$ is $1 \times 2$, then $\dot{y}_n^T$ is $2 \times 1$.

Then the backpropagation algorithm (for a single sample), for this staged computation would proceed as follows (equipped with the knowledge that the derivative of $X$ whose dimensions are $N \times M$ will preserve those dimensions):

$$\begin{aligned}
dPL_n &amp; = -1/N &amp; (7) \\
d\dot{y}^T_n &amp; = l_n^T  dPL_n &amp; (6) \\
dl_n &amp; = \dot{y}^T_n dPL_n &amp; (6) \\
d\hat{y}_n &amp; = 1/\hat{y}_n &amp; (5)
\end{aligned}
$$

Backpropagation equation $(5)$ above is a bit of an abuse of notation, but what I am trying to say that it is a vector, whose values are $[1/\hat{y}_n^{(1)}, 1/\hat{y}_n^{(2)}]$. But I am stuck at this step. Because I am not sure about the softmax. I have found this. Which tells me that if $a_2$ is the resulting softmax vector, then I can take following four derivatives: $\frac{\partial a_2^{(1)}}{\partial a_2^{(1)}}$,$\frac{\partial a_2^{(1)}}{\partial a_2^{(2)}}$,$\frac{\partial a_2^{(2)}}{\partial a_2^{(1)}}$,$\frac{\partial a_2^{(2)}}{\partial a_2^{(2)}}$. However, the dimensions of $z_2$ are $1 \times 2$, thus there should only be two partial derivatives. It makes conceptual sense to me, that $dz_2$ should equal to $\left[\frac{\partial a_2^{(1)}}{\partial a_2^{(2)}}, \frac{\partial a_2^{(2)}}{\partial a_2^{(1)}} \right]$, but I cannot motivate my intuition. Also, if I generalize this. Suppose $z_2$ is $1 \times 10$, what symbolic expression for derivative: $dz_2$ would we have then?

(I was motivated to implement my neural network from scratch after reading this post by Stanford; hence this question).

Another resource I am using. I am inclined to believe that $dz_2 = \hat{y}_n - \dot{y}_n$, but why, I do not know ;(

I am also missing $dl_n$ in backprop in $(5)$, the only way to end up at $1 \times 2$, is if we do: $d\hat{y}_n = 1/\hat{y}_n \circ dl_n$.

I have just looked at the porblem in the following way, which did not help me much. We need to find $dz_2  = \frac{\partial L}{ \partial z_2}$, we can rewrite this using backpropagation chain rule $$\begin{align} \frac{\partial L}{ \partial z_2} &amp;= \frac{\partial l_n}{\partial z_2} \frac{\partial L}{\partial l_n} \\ &amp;= \frac{\partial l_n}{\partial z_2} dl_n \\ &amp;= \frac{\partial}{\partial z_2} \log{(\text{softmax}(z_2))} dl_n \\ &amp;=\frac{\partial}{\partial u} \log{u} \frac{\partial u}{\partial z_2} dl_n \\ &amp;= \frac{1}{\text{softmax}(z_2)} \frac{\partial}{\partial z_2} \text{softmax}(z_2) dl_n \end{align}$$

Also, same question on Maths.SE.
The partial derivative of the cost function with respect to the neuron $i$ ,in  the layer $Z_2$ defined in your questions above, is $\frac{\partial L}{\partial z_i}$, where $L$ is the cost function.

please note that the derivation will be on the neuron level not at the layer level, where $z_i$  is the neuron number $i$ in the output layer/softmax layer/ $Z_2$ layer. In other words $Z_2$ layer consists of 2 elements $z_0$ and $z_1$ and the derivation will be w.r.t the neurons/elements.

I re-arrange your equations to keep things as simple as possible: combining equations 5,6,7 yields:  

$L = -\ \sum_{n} \dot{y}_n$ x $ log(y_n)$


Here is a brief derivation


$$\frac{\partial L}{\partial z_i}=-\sum_n\dot{y}_n\frac{\partial \log y_n}{\partial z_i}=-\sum_n\dot{y}_n\frac{1}{y_n}\frac{\partial y_n}{\partial z_i}$$

The softmax function $y_n$ is defined as
$$y_n = \frac{e^{z_n}}{\sum_ke^{z_k}}$$ 

when n=i
$$\frac{\partial y_n}{\partial z_i} = \color{blue}{y_i(1-y_i)}$$
when n &lt;> i 
$$
\frac{\partial y_n}{\partial z_i} = \color{red}{-y_ny_i}$$

now let's substitute $\frac{\partial y_n}{\partial z_i}$

$$\frac{\partial L}{\partial z_i}=-\dot{y}_i\frac{1}{y_i}\color{blue}{y_i(1-y_i)}-\sum_{n\neq i}\dot{y}_n\frac{1}{y_n}\color{red}{({{-y_ny_i}})}$$
$$
\\=-\dot{y}_i(1-y_i)+\sum_{n\neq i}\dot{y}_n({{y_i}})\\=-\dot{y}_i+{\dot{y}_iy_i+\sum_{n\neq i}\dot{y}_n({y_i})}
\\={y_i\left(\sum_n\dot{y}_n\right)}-\dot{y}_i=y_i-\dot{y}_i$$ where $\sum_n\dot{y}_n = 1$


For detailed derivation and explanations please see
https://deepnotes.io/softmax-crossentropy