title: How to set different weight variables in multiple rnn layers in tensorflow?
body: <p>Hi I'm writing a "many(input) to many(output)" rnn code to deal with character prediction. For this, I set my hidden layer parameter as follows. Three hidden layers and each of them has 100, 200, and 300 hidden units.</p>
  
  <pre><code>hidden layers= [100,200,300] 
  </code></pre>
  
  <p>My code is this.</p>
  
  <pre><code># parameters.
  hiddenLayers = [100,200,300]
  timeStep = 20 # sequence length
  inputDimension = 38 (# of English alphabet + symbols)
  outputDimension = 38 (# of English alphabet + symbols)
  input_x = tf.placeholder(tyf.float64, [None, timeStep, inputDimension])
  
  # make weights
  w1 = tf.get_variable("w1",[hiddenUnits[0],hiddenUnits[1] ],initializer=tf.random_normal_initializer())
  w2 = tf.get_variable("w2",[hiddenUnits[1],hiddenUnits[2] ],initializer=tf.random_normal_initializer())
  w3 = tf.get_variable("w3",[hiddenUnits[2],outputDimension ],initializer=tf.random_normal_initializer())
  # make biases
  b1 = tf.get_variable("b1",[hiddenUnits[1]], initializer=tf.constant_initializer(0.0))
  b2 = tf.get_variable("b2",[hiddenUnits[2]], initializer=tf.constant_initializer(0.0))
  b3 = tf.get_variable("b3",[outputDimension], initializer=tf.constant_initializer(0.0))
  
  def cell_generator(hiddenUnits):
      return rnn_cell = rnn.BasicLSTMCell(hiddenUnits, forget_bias=1.0)
  
  rnn_cell = rnn.MultiRNNCell([cell_generator(_) for _ in hiddenLayers])
  outputs, states = tf.nn.dynamic_rnn(rnn_cell, input_x, dtype=tf.float64)
  </code></pre>
  
  <p>Now I have 3 hidden layers and each of them has different number of hidden units.
  When I printed out "outputs" and "states" and they say liek this. (I didn't run a session.)</p>
  
  <pre><code>print(outputs)
  Tensor("rnn/transpose:0", shape=(?, 20, 300), dtype=float64)
  
  print(states)
  (LSTMStateTuple(c=&lt;tf.Tensor 'rnn/while/Exit_2:0' shape=(?, 100) dtype=float64&gt;, h=&lt;tf.Tensor 'rnn/while/Exit_3:0' shape=(?, 100) dtype=float64&gt;),
  LSTMStateTuple(c=&lt;tf.Tensor 'rnn/while/Exit_4:0' shape=(?, 200) dtype=float64&gt;, h=&lt;tf.Tensor 'rnn/while/Exit_5:0' shape=(?, 200) dtype=float64&gt;),
  LSTMStateTuple(c=&lt;tf.Tensor 'rnn/while/Exit_6:0' shape=(?, 300) dtype=float64&gt;, h=&lt;tf.Tensor 'rnn/while/Exit_7:0' shape=(?, 300) dtype=float64&gt;))
  </code></pre>
  
  <p>I thought that "output" should hold every output from 3 hidden layers but it has only last hidden layer so I got lost where to multiply 1st and 2nd weights. So now I have questions.</p>
  
  <ol>
  <li>Can I use "states" variables which hold 3 hidden layers so that I can multiply my 3 weights and add 3 biases to each of the last states.</li>
  </ol>
  
  <p>for example...</p>
  
  <pre><code>hidden2 = tf.matmul(state[0],w1) + b1
  hidden3 = tf.matmul(state[1],w2) + b2
  final_output = tf.matmul(state[2],w3) + b3
  # and do the loss calculation for training...
  </code></pre>
  
  <ol start="2">
  <li><p>Is there any way that I can use "output" variable in order to apply my weights and biases to that cell?</p></li>
  <li><p>Or is there any other alternative ways that I can use my weights and biases? Maybe initialize weights and biases in the previous steps(before toss the basicRNNCell to the MultiRNNcell)?</p></li>
  </ol>
  
  <p>I really want to set the different number of hidden layers and units and apply them with my predetermined weight and bias parameters. Please let me know if you have any ideas.</p>
  
  <p>Thanks in advance.</p>