title: Tensorflow save final state of LSTM in dynamic_rnn for prediction
body: <p>I want to save the final state of my LSTM such that it's included when I restore the model and can be used for prediction. As explained below, the Saver only has knowledge of the final state when I use <code>tf.assign</code>. However, this throws an error (also explained below).</p>
  
  <p>During training I always feed the final LSTM state back into the network, as explained in <a href="https://stackoverflow.com/questions/39112622/how-do-i-set-tensorflow-rnn-state-when-state-is-tuple-true">this post</a>. Here are the important parts of the code:</p>
  
  <p>When building the graph:</p>
  
  <pre><code>            self.init_state = tf.placeholder(tf.float32, [
                  self.n_layers, 2, self.batch_size, self.n_hidden
              ])
  
              state_per_layer_list = tf.unstack(self.init_state, axis=0)
  
              rnn_tuple_state = tuple([
                  tf.contrib.rnn.LSTMStateTuple(state_per_layer_list[idx][0],
                                                state_per_layer_list[idx][1])
  
                  for idx in range(self.n_layers)
              ])
  
              outputs, self.final_state = tf.nn.dynamic_rnn(
                  cell, inputs=self.inputs, initial_state=rnn_tuple_state)
  </code></pre>
  
  <p>And during training:</p>
  
  <pre><code>        _current_state = np.zeros((self.n_layers, 2, self.batch_size,
                                     self.n_hidden))
  
              _train_step, _current_state, _loss, _acc, summary = self.sess.run(
                  [
                      self.train_step, self.final_state,
                      self.merged
                  ],
                  feed_dict={self.inputs: _inputs,
                             self.labels:_labels, 
                             self.init_state: _current_state})
  </code></pre>
  
  <p>When I later restore my model from a checkpoint, the final state is not restored as well. As outlined in <a href="https://stackoverflow.com/questions/39112622/how-do-i-set-tensorflow-rnn-state-when-state-is-tuple-true">this post</a> the problem is that the Saver has no knowledge of the new state. The post also suggests a solution, based on <code>tf.assign</code>. Regrettably, I cannot use the suggested</p>
  
  <pre><code>            assign_op = tf.assign(self.init_state, _current_state)
              self.sess.run(assign_op)
  </code></pre>
  
  <p>because self.init state is not a Variable but a placeholder. I get the error </p>
  
  <p>AttributeError: 'Tensor' object has no attribute 'assign'</p>
  
  <p>I have tried to solve this problem for several hours now but I can't get it to work.</p>
  
  <p>Any help is appreciated!</p>
  
  <p><strong>EDIT:</strong></p>
  
  <p>I have changed self.init_state to</p>
  
  <pre><code>            self.init_state = tf.get_variable('saved_state', shape=
              [self.n_layers, 2, self.batch_size, self.n_hidden])
  
              state_per_layer_list = tf.unstack(self.init_state, axis=0)
  
              rnn_tuple_state = tuple([
                  tf.contrib.rnn.LSTMStateTuple(state_per_layer_list[idx][0],
                                                state_per_layer_list[idx][1])
  
                  for idx in range(self.n_layers)
              ])
  
              outputs, self.final_state = tf.nn.dynamic_rnn(
                  cell, inputs=self.inputs, initial_state=rnn_tuple_state)
  </code></pre>
  
  <p>And during training I don't feed a value for self.init_state:</p>
  
  <pre><code>            _train_step, _current_state, _loss, _acc, summary = self.sess.run(
                  [
                      self.train_step, self.final_state,
                      self.merged
                  ],
                  feed_dict={self.inputs: _inputs,
                             self.labels:_labels})
  </code></pre>
  
  <p>However, I still can't run the assignment op. Know I get</p>
  
  <p>TypeError: Expected float32 passed to parameter 'value' of op 'Assign', got (LSTMStateTuple(c=array([[ 0.07291573, -0.06366599, -0.23425588, ...,  0.05307654,</p>