Generalization of RNN/LSTM/GRU... model
Maybe a little bit late to the party, but this might help you in the future:
Neural networks in general are heavily overparameterised functions that are fitted to match the training data. This is true for simple MLPs or more complex RNNs. These functions never encapsulate the &quot;true&quot; function, but only approximate it.
In you data, the spikes are simply outliers (meaning that they cannot be explained by the given, underlying input data). This means that to approximate those outliers, a function of your given inputs is not enough, since you are &quot;averaging out&quot; or ignoring these spkies as outliers during training (I assume you have a large dataset to train on).
See it that way: the given input data to your approximated function is not enough to explain the full &quot;world&quot; that your time series exists in.