In gradient descent's update rule, why do we use $\sigma(z^{l-1})\frac{\delta C_0}{ \delta w^{l}}$ instead of $\frac{\delta C_0}{\delta w^{l}}$? I am trying to code a two layered neural network simple NN as I have described here https://itisexplained.com/html/NN/ml/5_codingneuralnetwork/
I am getting stuck on the last step of updating the weights after calculating the gradients for the outer and inner layers via back-propagation
#---------------------------------------------------------------

# Two layered NW. Using from (1) and the equations we derived as explanations
# (1) http://iamtrask.github.io/2015/07/12/basic-python-network/
#---------------------------------------------------------------

import numpy as np
# seed random numbers to make calculation deterministic 
np.random.seed(1)

# pretty print numpy array
np.set_printoptions(formatter={'float': '{: 0.3f}'.format})

# let us code our sigmoid funciton
def sigmoid(x):
    return 1/(1+np.exp(-x))

# let us add a method that takes the derivative of x as well
def derv_sigmoid(x):
   return x*(1-x)

# set learning rate as 1 for this toy example
learningRate =  1

# input x, also used as the training set here
x = np.array([ [0,0,1],[0,1,1],[1,0,1],[1,1,1]  ])

# desired output for each of the training set above
y = np.array([[0,1,1,0]]).T

# Explanaiton - as long as input has two ones, but not three, ouput is One
&quot;&quot;&quot;
Input [0,0,1]  Output = 0
Input [0,1,1]  Output = 1
Input [1,0,1]  Output = 1
Input [1,1,1]  Output = 0
&quot;&quot;&quot;

input_rows = 4
# Randomly initalised weights
weight1 =  np.random.random((3,input_rows))
weight2 =  np.random.random((input_rows,1))

print(&quot;Shape weight1&quot;,np.shape(weight1)) #debug
print(&quot;Shape weight2&quot;,np.shape(weight2)) #debug

# Activation to layer 0 is taken as input x
a0 = x

iterations = 1000
for iter in range(0,iterations):

  # Forward pass - Straight Forward
  z1= x @ weight1
  a1 = sigmoid(z1) 
  z2= a1 @ weight2
  a2 = sigmoid(z2) 

  # Backward Pass - Backpropagation 
  delta2  = (y-a2)
  #---------------------------------------------------------------
  # Calcluating change of Cost/Loss wrto weight of 2nd/last layer
  # Eq (A) ---&gt; dC_dw2 = delta2*derv_sigmoid(z2)
  #---------------------------------------------------------------

  dC_dw2  = delta2 * derv_sigmoid(a2)

  if iter == 0:
    print(&quot;Shape dC_dw2&quot;,np.shape(dC_dw2)) #debug
  
  #---------------------------------------------------------------
  # Calcluating change of Cost/Loss wrto weight of 2nd/last layer
  # Eq (B)---&gt; dC_dw1 = derv_sigmoid(a1)*delta2*derv_sigmoid(a2)*weight2
  # note  delta2*derv_sigmoid(a2) == dC_dw2 
  # dC_dw1 = derv_sigmoid(a1)*dC_dw2*weight2
  #---------------------------------------------------------------
  
  dC_dw1 =  (np.multiply(dC_dw2,weight2.T)) * derv_sigmoid(a1)
  if iter == 0:
    print(&quot;Shape dC_dw1&quot;,np.shape(dC_dw1)) #debug
  

  #---------------------------------------------------------------
  #Gradinent descent
  #---------------------------------------------------------------
 
  #weight2 = weight2 - learningRate*dC_dw2 --&gt; these are what the textbook tells
  #weight1 = weight1 - learningRate*dC_dw1 

  weight2 = weight2 + learningRate*np.dot(a1.T,dC_dw2) # this is what works
  weight1 = weight1 + learningRate*np.dot(a0.T,dC_dw1) 
  

print(&quot;New ouput\n&quot;,a2)


Why is
  weight2 = weight2 + learningRate*np.dot(a1.T,dC_dw2)
  weight1 = weight1 + learningRate*np.dot(a0.T,dC_dw1) 

done instead of
  #weight2 = weight2 - learningRate*dC_dw2
  #weight1 = weight1 - learningRate*dC_dw1 

I am not getting the source of the equation of updating the weights by multiplying with the activation of the previous layer
As per gradient descent, the weight update should be
$$
  W^{l}_{new} = W^{l}_{old} - \gamma * \frac{\delta C_0}{\delta w^{l}}
$$
However, what works in practice is
$$
 W^{l}_{new} = W^{l}_{old} - \gamma * \sigma(z^{l-1})\frac{\delta C_0}{ \delta w^{l}},
$$
where $\gamma$ is the learning rate.
Okay - the answer is here  https://explained.ai/matrix-calculus/#sec6.2 and it is pretty involved. Basically, there is a difference when you derive the equation for one neuron and when you have to do practically for a set of neurons. The answer is matrix calculus. Here goes from what I could make out. Feel free to correct if I am wrong
Gradient Vector/Matrix/2D tensor of Loss function wrto Weight
$$
C = \frac{1}{2} \sum_j (y_j-a^L_j)^2
$$
Assuming a neural net with 2 layers, we have the final Loss as
$$
C = \frac{1}{2} \sum_j (y_j-a^2_j)^2
$$
Where
$$
a^2 = \sigma(w^2.a^1)
$$
We can then write
$$
C = \frac{1}{2} \sum_j v^2 \quad \rightarrow (Eq \;A)
$$
Where
$$
v= y-a^2 
$$
Partial Derivative of Loss function wrto Weight
For the last layer, lets use Chain Rule to split like below
$$
\frac {\partial C}{\partial w^2} = \frac{\partial v^2}{\partial v} * \frac{\partial v}{\partial w^2} \quad \rightarrow (Eq \;B)
$$
$$
 \frac{\partial v^2}{\partial v} =2v \quad \rightarrow (Eq \;B.1)
$$
$$
\frac{\partial v}{\partial w^2}=  \frac{\partial (y-a^2)}{\partial w^2} = 
0-\frac{\partial a^2}{\partial w^2} \quad \rightarrow (Eq \;B.2)
$$
$$
\frac {\partial C}{\partial w^2} = \frac{1}{2} *2v(0-\frac{\partial a^2}{\partial w^2}) \quad \rightarrow (Eq \;B)
$$
 
Now we need to find $\frac{\partial a^2}{\partial w^2}$
Let
$$
a^2= \sigma(sum(w^2 \otimes a^1 )) = \sigma(z^2) 
$$
$$
z^2 = sum(w^2 \otimes a^1 )
$$
$$
z^2 = sum(k^2) \; \text {where} \; k^2=w^2 \otimes a^1 
$$
We now need to derive an intermediate term which we will use later
$$
\frac{\partial z^2}{\partial w^2} =\frac{\partial z^2}{\partial k^2}*\frac{\partial k^2}{\partial w^2}
$$
$$
=\frac {\partial sum(k^2)}{\partial k^2}* \frac {\partial (w^2 \otimes a^1 )} {\partial w^2}
$$
$$
\frac{\partial z^2}{\partial w^2} = (1^{\rightarrow})^T* diag(a^1) =(a^{1})^T \quad \rightarrow (Eq \;B.3)
$$
How the above is, you need to check this  in https://explained.ai/matrix-calculus/#sec6.2
Basically though these are written like scalar here; actually all these are partial differention of vector by vector, or vector by scalar; and a set of vectors can be represented as the matrix here.
Note that the vector dot product $w.a$ when applied on matrices becomes the elementwise multiplication $w^2 \otimes a^1$ (also called Hadamard product)
Going back to  $Eq \;(B.2)$
$$
\frac {\partial a^2}{\partial w^2} = \frac{\partial a^2}{\partial z^2} * \frac{\partial z^2}{\partial w^2}
$$
Using $Eq \;(B.3)$ for the term in left
$$
=  \frac{\partial a^2}{\partial z^2} * (a^{1})^T
$$
$$
=  \frac{\partial \sigma(z^2)}{\partial z^2} * (a^{1})^T
$$
$$
\frac {\partial a^2}{\partial w^2} =   \sigma^{'}(z^2) * (a^{1})^T \quad \rightarrow (Eq \;B.4)
$$
Now lets got back to partial derivative of Loss function wrto to weight
$$
\frac {\partial C}{\partial w^2} = \frac {1}{2}*2v(0-\frac{\partial a^2}{\partial w^2}) \quad \rightarrow (Eq \;B)
$$
Using $Eq \;(B.4)$ to substitute in the last term
$$
= v(0- \sigma^{'}(z^2) * (a^{1})^T)
$$
$$
= v*-1*\sigma^{'}(z^2) * (a^{1})^T
$$
$$
= (y-a^2)*-1*\sigma^{'}(z^2) * (a^{1})^T
$$
$$
\frac {\partial C}{\partial w^2}= (a^2-y)*\sigma^{'}(z^2) * (a^{1})^T
$$