Do Recurrent Neural Networks assume stationarity or just a general kind of sequential dependence?
The distribution is stationary, but it doesn't mean the time series should be stationary.
Let's imagine the recurrent network, that returns the n+1 if it was n on previous step. Distribution is not changing with time, but the time-series always goes up.