Feeding back hidden state manually in tf.nn.dynamic_rnn (Tensorflow) In many ptb/mini shakespeare  LSTM generator tutorials on web, people make input data, (where every batch is subsequent. for example, sequences in batch_2 are subsequent to sequences in batch_1, and batch_1 is fed right before batch_2) and feed those input data to dynamic_rnn. tutorial: https://r2rt.com/recurrent-neural-networks-in-tensorflow-ii.html   

During training, these tutorials manually feed back the hidden state of LSTM, like codes below. 

batch_size = 32
hidden_state_in = cell.zero_state(batch_size, tf.float32) 
output, hidden_state_out = tf.nn.dynamic_rnn(cell, inputs,initial_state=hidden_state_in)
...
#For loop used in training: ...
    output, hidden_state = sess.run([output, hidden_state_out],
                            feed_dict={hidden_state_in:hidden_state})


What makes me confused, is that when generating or testing ptb/shakespreare, these tutorials feed data, with batchsize==1 and varying seqlength (at training, batchsize was 32 or above).

def generate_characters(g, checkpoint, num_chars, prompt='A', pick_top_chars=None):
    with tf.Session() as sess:
        sess.run(tf.initialize_all_variables())
        g['saver'].restore(sess, checkpoint)
        state = None
        current_char = vocab_to_idx[prompt] 
        chars = [current_char]

        for i in range(num_chars):
             if state is not None:
                feed_dict={g['x']: [[current_char]], g['init_state']: state}
                #shape [BATCHSIZE, SEQLEN] with BATCHSIZE=1 and SEQLEN=1 
            else:
                feed_dict={g['x']: [[current_char]]}
            preds, state = sess.run([g['preds'],g['final_state']], feed_dict)


When using dynamic_rnn with LSTM, the hidden state has shape of  [num_layers, 2, batch_size, state_size]. I just wonder how can we use data with different batch_size at training and testing. If hidden state is fed manually, what kind of trained feature is contained in LSTM graph? Is it cell_state, or the coefficients which are used in building states?

Also, when looking at generating text part of r2rt's tutorial above, first parts of generation seems weird. I guess this is because hidden_state (which is fed to dynamic_rnn's initial_state) at the beginning of generation is yet poorly configured, cause it's right after the first prompt. Am I right about this reason?
There's a lot going on inside an LSTM, so it's easy to get confused. I think you are confusing the state and the weights:


The weights are trained at training time and are not updated during prediction. This is like the weights of any other neural network.
The state updates as the model moved "forward" through the text. It is essentially what enables the LSTM to keep track of where it currently is in a sequence.


The weights have a fixed size, of course, but when generating text the LSTM acts only on the last character and the current state, which is why the sequence length is set to 1 (and if you're only generating one text at a time, the batch length is also 1. you could increase the batch size to generate multiple texts at the same time).