title: How Adagrad works in Keras? What does self.weights mean in Keras Optimizer?
body: <p>For example, the implementation of Keras' Adagrad has been:</p>
  
  <pre><code>class Adagrad(Optimizer):
  """Adagrad optimizer.
  It is recommended to leave the parameters of this optimizer
  at their default values.
  # Arguments
      lr: float &gt;= 0. Learning rate.
      epsilon: float &gt;= 0.
      decay: float &gt;= 0. Learning rate decay over each update.
  # References
      - [Adaptive Subgradient Methods for Online Learning and Stochastic Optimization](http://www.jmlr.org/papers/volume12/duchi11a/duchi11a.pdf)
  """
  
  def __init__(self, lr=0.01, epsilon=1e-8, decay=0., **kwargs):
      super(Adagrad, self).__init__(**kwargs)
      self.lr = K.variable(lr)
      self.epsilon = epsilon
      self.decay = K.variable(decay)
      self.initial_decay = decay
      self.iterations = K.variable(0.)
  
  def get_updates(self, params, constraints, loss):
      grads = self.get_gradients(loss, params)
      shapes = [K.get_variable_shape(p) for p in params]
      accumulators = [K.zeros(shape) for shape in shapes]
      self.weights = accumulators
      self.updates = []
  
      lr = self.lr
      if self.initial_decay &gt; 0:
          lr *= (1. / (1. + self.decay * self.iterations))
          self.updates.append(K.update_add(self.iterations, 1))
  
      for p, g, a in zip(params, grads, accumulators):
          new_a = a + K.square(g)  # update accumulator
          self.updates.append(K.update(a, new_a))
          new_p = p - lr * g / (K.sqrt(new_a) + self.epsilon)
          # apply constraints
          if p in constraints:
              c = constraints[p]
              new_p = c(new_p)
          self.updates.append(K.update(p, new_p))
      return self.updates
  </code></pre>
  
  <p>And the Function 'get_update()' seems one step update. However should the accumulators be stored the history information? Why it has been initialized to zeros at each step? How it can be an accumulator through the whole training process?</p>
  
  <p>What does this line do?</p>
  
  <pre><code>self.weights = accumulators
  </code></pre>
  
  <p>It seems self.weights is never been called anymore.</p>