is neural networks an online algorithm by nature? I have been doing machine learning for a while, but bits and pieces come together even after some time of practicing.

In neural networks, you adjust the weights by doing one pass (forward pass), and then computing the partial derivatives for the weights (backward pass) after each training example - and subtracting those partial derivatives from the initial weights.

in turn, the calculation of the new weights is mathematically complex (you need to compute the partial derivative of the weights, for which you compute the error at every layer of the neural net - but the input layer).

Is that not by definition an online algorithm, where cost and new weights are calculated after each training example?

Thanks!
There are three training modes for neural networks


stochastic gradient descent: Adjust the weights after every single training example
batch training: Adjust the weights after going through all data (an epoch)
mini-batch training: Adjust the weights after going through a mini-batch. This is usually 128 training examples.


Most of the time, mini-batch training seems to be used.

So the answer is:

No, the neural network learning algorithm is not online algorithm.