How are the gradients of a Neural Network calculated just by matrix multiplication?
Like the derivative of y = h_relu * w2 being h_relu, which doesn't make much sense to me 


These are partial derivative and depends on w.r.t variable. In the end, we have to complete the chain of partial derivative from output to the respective weights. 

Your Network is -
$$x \xrightarrow[\text{}]{\text{}} w_1 \xrightarrow[\text{}]{\text{  h  }} ReLU \xrightarrow[\text{}]{\text{  h_relu  }} w_2 \xrightarrow[\text{}]{\text{ }} y\_pred$$
By definition ReLU is - $f(x)=\max(0, x).$
Derivative of ReLU - $\frac{d}{dx}\text{ReLU}(x)$
=$1 \forall x &gt; 0$
=$0 \forall x &lt; 0$

In backpropagation -
$\frac{\partial y\_pred}{\partial w_2} $ = $h\_relu$  ;since, y_pred = $h\_relu*w_2$
$\frac{\partial w_2}{\partial h} $ = $1$  ;derivative of ReLU=1,ignoring negative part for simplicity
$\frac{\partial h}{\partial w_1} $ = $x$ ;since, h = $x*w_1$
$\frac{\partial y\_pred}{\partial h\_relu} $ = $w_2$ ;since, y_pred = $h\_relu*w_2$

So, Loss gradient at different weights will be - 
 $L$ at the $y\_pred$
 $L*h\_relu$ at $w_2$ 
 $L*w_2$ at $h\_relu$ 
 $L*h\_relu*1*x$ at $w_1$