"Invalid value" in RMSprop implementation from scratch in Python Edit 2: The regularization term (reg_term) is sometimes negative due negatative parameters. Hence S[f&quot;dW{l}&quot;] contains some negative values. I realize the reg_term has to be added before taking the sqrt, like this:
S[f&quot;dW{l}&quot;] = beta2 * S[f&quot;dW{l}&quot;] + (1 - beta2) * (np.square(gradients[f&quot;dW{l}&quot;] + reg_term))

Edit 1: I see that S[f&quot;dW{l}&quot;] contains some negative values. How is this possible when np.square(gradients[f&quot;dW{l}&quot;] always contains positive values?
I have implemented a neural network from scratch which uses mini-batch gradient descent. The network works well. Unfortunately, I can't get my RMSprop implementation to work. I have verified that the network works well with momentum.
I get a RuntimeWarning when training the network with RMSprop: &quot;invalid value encountered in sqrt&quot;. This happens in the &quot;RMSprop&quot; update step.
My implementation of update parameters:
def update_parameters(parameters, gradients, V, S, batch_size, t, learning_rate, reg_param):
    L = len(parameters) // 2
    beta1 = 0.9
    beta2 = 0.999
    epsilon = 1e-8
    
    for l in range(1, L+1):
        reg_term = (reg_param / batch_size) * parameters[f&quot;W{l}&quot;]

        # RMSprop gradients
        S[f&quot;dW{l}&quot;] = beta2 * S[f&quot;dW{l}&quot;] + (1 - beta2) * (np.square(gradients[f&quot;dW{l}&quot;]) + reg_term)
        S[f&quot;db{l}&quot;] = beta2 * S[f&quot;db{l}&quot;] + (1 - beta2) * np.square(gradients[f&quot;db{l}&quot;])
        
        # RMSprop update
        parameters[f&quot;W{l}&quot;] -= learning_rate * (gradients[f&quot;dW{l}&quot;] / (np.sqrt(S[f&quot;dW{l}&quot;])) + epsilon)
        parameters[f&quot;b{l}&quot;] -= learning_rate * (gradients[f&quot;db{l}&quot;] / (np.sqrt(S[f&quot;db{l}&quot;])) + epsilon)

This is how I initialize the parameters:
def init_params_V_and_S(activation_layers):
    params = {}
    V = {}
    S = {}
    L = len(activation_layers)
    
    for l in range(1, L):
        params[f&quot;W{l}&quot;] = np.random.randn(activation_layers[l], activation_layers[l-1]) * np.sqrt(2 / activation_layers[l-1])
        params[f&quot;b{l}&quot;] = np.zeros((activation_layers[l], 1))

        # RMSprop params
        S[f&quot;dW{l}&quot;] = np.zeros((activation_layers[l], activation_layers[l-1]))
        S[f&quot;db{l}&quot;] = np.zeros((activation_layers[l], 1))

    return params, V, S

Any ideas what's causing this?
The regularization term (reg_term) is sometimes negative due negatative parameters. Hence S[f&quot;dW{l}&quot;] contains some negative values. I realize the reg_term has to be added before taking the sqrt, like this:
S[f&quot;dW{l}&quot;] = beta2 * S[f&quot;dW{l}&quot;] + (1 - beta2) * np.square(gradients[f&quot;dW{l}&quot;] + reg_term)