How are the observations stored in the RNN that encodes the state?
This research question seems to be analyzed in further details here (section 3) - https://openreview.net/pdf?id=r1lyTjAqYX.

Usually, a sequence is taken as a state to be fed into RNN to compute the final hidden state. One can then ask what initial state should the RNN be seeded with? This paper analyzes three methods with respect to the seed - 


zero initialization: When the RNN is initialized with the zero state 
burn-in: when the sequence is prepended by some preceding observations for RNN to learn a good initial state 
storing the initial hidden state: When the hidden state at the beginning of the sequence is stored