Does training of neural networks follow the same order in each epoch?
All the parameters are updated after a batch, there is no notion of order of update.
The batch can or cannot be in the same order. It could lead to overfitting in the sense of the network learn the order of the dataset. An easy turnaround is to shuffle the batch between each epochs.