ngram and RNN prediction rate wrt word index
Recurrent Neural Network (RNN) create a single state vector over time. Thus that curve is to be expected. Initially, the state vector does have enough information to make a quality prediction. Then quickly reaches asymptotic performances. Overall, the predictions are between 15% and 22% correct.
The shape of the graph might be a function of sentence length in the training corpus. Possibly sentences could be between 3 and 7 words long. The drop could be because there is less training data for longer sentences.