Feeding back hidden state manually in tf.nn.dynamic_rnn (Tensorflow)
There's a lot going on inside an LSTM, so it's easy to get confused. I think you are confusing the state and the weights:


The weights are trained at training time and are not updated during prediction. This is like the weights of any other neural network.
The state updates as the model moved "forward" through the text. It is essentially what enables the LSTM to keep track of where it currently is in a sequence.


The weights have a fixed size, of course, but when generating text the LSTM acts only on the last character and the current state, which is why the sequence length is set to 1 (and if you're only generating one text at a time, the batch length is also 1. you could increase the batch size to generate multiple texts at the same time).