Is it Normal for a Neural Network Loss to Increase after being trained on an example?
Is your dataset shuffled? 
Otherwise it could be the case that it was predicting one class for the first 99 examples.
If not then LSTM can be tricky to train. Try changing hyper parameters and also I would recommend starting with SimpleRNN, GRU and then LSTM as sometimes a simple network might just do the trick.
I am currently testing an LSTM network. I print the loss of its prediction on a training example before back-propagation and after back-propagation. It would make sense that the after loss should always be less than the before loss because the network was just trained on that example. 
However, I am noticing that around the 100th training example, the network begins to give a more inaccurate prediction after back-propagation than before back-propagating on a training example. 
Is a network expected to always have the before loss be higher than the after loss? If so, are there any reasons this happens?
To be clear, for the first hundred examples, the network seems to be training correctly and doing just fine.