RMSprop equation - dividing by a matrix? I've been trying to understand RMSprop for a long time, but there's something that keeps eluding me.
$dW$ and $db$ are matrices (that's what I understand from the element-wise comment), so that must mean that $S_{dW}$ is a matrix (or tensor) as well.
So, in the update rule, they divide a matrix by another matrix? From what I saw in google, no such action exists.
Yes, you are correct, $S_{dW}$, in this case, is the matrix/tensor and it will have the same shape as gradient matrix/tensor. The update rule divides each element in gradient to each gradient in $S_{dW}$ or you can say &quot;element-wise division&quot;.
To avoid the zero variable in $S_{dW}$, the equation usually like this:
$$
W=W-\alpha \frac{dW}{\sqrt{S_{dW}+\epsilon}} \text{  where } \epsilon=10^{-8}
$$
If you want to implement it manually with Python, you can take a look at this link