title: Tensorflow RNNs with variable length sequences, padded zeros affect learning
body: <p>I've set up an RNN in tensorflow that takes a variable sequence and makes 1 prediction at the end of the sequence.</p>
  
  <p>I've zero padded my data to a max length of 500 sequences, but many sequences in a batch will be less than 500.</p>
  
  <p>I use dynamic_rnn and pass it the sequence lengths of each sample in the batch:</p>
  
  <pre><code># Get lstm cell output
  m.outputs, m.states = tf.nn.dynamic_rnn(
      cell=lstm_cell,
      dtype=tf.float32,
      sequence_length=m.X_lengths,
      inputs=m.X)
  </code></pre>
  
  <p>Where m.X_lengths is the sequence lengths as a tensor which was set up as a placeholder variable. I pass it in with the <code>feed_dict</code>.</p>
  
  <p>For the cost function, it's sigmoid cross entropy (multi-class classification), and I take the last value from <code>m.outputs</code>, and process that with tf.reduce_mean.</p>
  
  <p>Notably, I did not do any masking of the loss function. My understanding is that masking would only be needed if I were trying to use the sum of all losses from all outputs. But I'm only using the last output.</p>
  
  <p>Now I've added 1000 padded zeros onto my sequences, but the larges sequence length if still just 500, but the batch has 1500 sequence lengths. If the padding is having no effect this will learn the same as without the additional padding. When I train the model with this additional padding learning is negatively affected. Also restricting my sequence lengths to 100 improves the results. </p>
  
  <p>Questions:</p>
  
  <ul>
  <li>Can I pass in a placeholder variable for <code>sequence_lengths</code>?</li>
  <li>Am I correct in understanding that I don't need to mask the losses if I only use the last output from <code>dynamic_nn</code>?</li>
  <li>Are there any red flags in how I thought this out?</li>
  </ul>