How exactly does SGD with momentum works in keras This is my first question in this community (Artificial Intelligence) and I hope I can get some answers to a troublesome issue of mine.

I am trying to understand Stochastic Gradient Decent (SGD) (with possibly momentum) and modify the keras version to an accumulative one. I have asked a similar question on Stack Overflow but here I am interested in the actual function (and secondarily to the code itself).

For simplicity let's consider I have a neural network with just 10 layers and I am training for 100 iterations (regardless of epochs).
For easier reference I post the keras code here (and also skipped nesterov and constraints lines of code):

def get_updates(self, loss, params):
    grads = self.get_gradients(loss, params)
    self.updates = [K.update_add(self.iterations, 1)]

    lr = self.lr
    if self.initial_decay &gt; 0:
        lr = lr * (1. / (1. + self.decay * K.cast(self.iterations, K.dtype(self.decay))))
    shapes = [K.int_shape(p) for p in params]
    moments = [K.zeros(shape) for shape in shapes]
    self.weights = [self.iterations] + moments
    for p, g, m in zip(params, grads, moments):
        v = self.momentum * m - lr * g  # velocity
        self.updates.append(K.update(m, v))
        new_p = p + v

        self.updates.append(K.update(p, new_p))
    return self.updates


As far as I understand the mechanism of SGD works as follows:


The gradients are updated at every iteration, which in the keras code is performed by calling the get_updates(self, loss, params) function.
There is a gradient that correspond to each layer (in the keras code I think a list of Tensors are produced to keep the gradients).
So, in that sense the for p, g, m in zip(params, grads, moments) loop just iterates through all layers (and their corresponding gradients and moments) and calculates new values.


The really confusing part for me is that if the above function is being called at every iteration then the moments = [K.zeros(shape) for shape in shapes] basically initializes moments to a zero Tensor at every iteration. Wouldn't make more sense if moment was initialized once and then keep updating with new values?

I have noticed that despite my understanding the moments are actually used (as it would be more expected for such a big project to have missed a large bug such as this one). I cannot really understand how though.
I noticed the same issue. Also, it seems that the equation for applying momentum does not exactly match up with the standard one:

$$ v_{d\theta} := \beta v_{d\theta} + (1 - \beta)d\theta \\
 \theta := \theta - \alpha v_{d\theta}$$