anchor.diff:  $$$
     \begin{eqnarray*}
     \mathbf{w}_{t+1} &=& \mathbf{w}_t - \eta \nabla Q \\
  -   &=& \mathbf{w}_t - \eta \sum_{i=1}^{n} \nabla Q_i(\mathbf{w}) \; ,\\
  +   &=& \mathbf{w}_t - \eta \sum_{i=1}^{d} \nabla Q_i(\mathbf{w}) \; ,\\
     \end{eqnarray*}
   
  -where $\eta > 0$ is a step size. This "batch" update rule has to compute the full objective function in each step, the evaluation time of which is proportional to the size of the training set $n$.
  +where $\eta > 0$ is a step size. This "batch" update rule has to compute the full objective function in each step, the evaluation time of which is proportional to the size of the training set $d$.
   
   Alternatively, in stochastic gradient descent, $Q$ is minimized using
   
   $$$
     \mathbf{w}_{t+1} = \mathbf{w}_t - \eta \nabla Q_i (\mathbf{w}) \; ,
   
  -updating the weights $\mathbf{w}$ in each step using just one sample $i$ randomly chosen from the training set. This is advantageous for big sample sizes, because it makes the evaluation time of each step independent from $n$. Another advantage is that it can process samples on the fly, in an [online learning](http://en.wikipedia.org/wiki/Online_machine_learning) task.
  +updating the weights $\mathbf{w}$ in each step using just one sample $i$ randomly chosen from the training set. This is advantageous for big sample sizes, because it makes the evaluation time of each step independent from $d$. Another advantage is that it can process samples on the fly, in an [online learning](http://en.wikipedia.org/wiki/Online_machine_learning) task.
   
   In practice, instead of $\eta$, the algorithm is used with a decreasing sequence of step sizes $\eta_t$, updated in each step.
   
  
positive.code: $$$
    Q_i(\mathbf{w}) = \left( \mathbf{y}_i - f_{\mathbf{w}} (\mathbf{x}_i) \right)^2 \; ,
  
  where $f_{\mathbf{w}} : \mathbb{R}^n \to \mathbb{R}^m$ is a model function for our data (parametrized by $\mathbf{w}$) and $\mathbf{x}_i \in \mathbb{R}^n$ and $\mathbf{y}_i \in \mathbb{R}^m$ are the input and the expected output of the $i$-th sample in the training set. Finding the parameters $\mathbf{w}$ minimizing $Q(\mathbf{w}) = \sum_{i=1}^{n} Q_i (\mathbf{w})$ thus fits the model function $f_{\mathbf{w}}$ to our data.
  
  We can test this via [fitting a curve](http://en.wikipedia.org/wiki/Curve_fitting)
  
  
negative.code: [Stochastic gradient descent](http://en.wikipedia.org/wiki/Stochastic_gradient_descent) is a [stochastic](http://en.wikipedia.org/wiki/Stochastic) variant of the gradient descent algorithm that is used for minimizing objective functions with the form of a sum
  
  $$$
    Q(\mathbf{w}) = \sum_{i=1}^{d} Q_i(\mathbf{w}) \; ,
  
  where $\mathbf{w}$ is a weight vector parametrizing $Q$. The component $Q_i$ is the contribution of the $i$-th sample to the objective function $Q$, which is to be minimized using a training set of $n$ samples.
  
  