SGD, calculating it by hand
You can see in the following link examples in Numpy of the following optimisers


Stochastic Gradient Descent
Stochastic Gradient Descent + momentum
Stochastic Gradient Descent + Nesterov momentum
Adagrad
Adadelta
RMSProp
Adam
AdaMax
Nadam 


Just clone the repository and open the Report.ipyb with Jupyter Notebooks.

https://github.com/ivallesp/awesome-optimizers