How does the momentum term for backpropagation algorithm work?
Using Backpropagation with momentum in a network with $n$ different weights $W_k$ the $i$-th correction for weight $W_k$ is given by

$\Delta  W_k(i) = -\alpha  \frac{\partial E}{\partial W_k} + \mu  \Delta W_k(i-1)$
where $\frac{\partial E}{\partial W_k} $ is the variation of the loss w.r.t. $W_k$.

Introduction of the momentum rate allows the attenuation of oscillations in the
gradient descent. The geometric idea behind this idea can probably best be understood in terms of an eigenspace analysis in the linear case. If the ratio between lowest and largest eigenvalue is large then performing a gradient descent is slow even if the learning rate large due to the conditioning of the matrix. The momentum introduces some balancing in the update between the eigenvectors associated to lower and larger eigenvalues.

For more detail I refer to 

http://page.mi.fu-berlin.de/rojas/neural/chapter/K8.pdf