title: Dropout on test somehow causes LSTM performance failure
body: <p>I have a problem with TensorFlow where the performance of my LSTM drops dramatically (from 70% to &lt;10%) when I use input dropout. </p>
  
  <p>As I understand it I should set the input_keep_probability to (e.g.) 0.5 during training and then to 1 during testing. This makes perfect sense, but I cant get it to work as intended. If I set the dropout during testing the same as during training my performance improves (this is not the case in the example below, but this needs less code and the point is the same).</p>
  
  <p><a href="http://i.stack.imgur.com/AKbsU.png" rel="nofollow">The accuracy and cost of 3 runs</a></p>
  
  <p>The 'best' line is no dropout, the worst line is [keep_prob@train: 0.5, keep_prob@test: 1] and the center line is [keep_prob@train: 0.5, keep_prob@test: 0.5]. These are cost and accuracy on the test set. They all behave as expected on the train set.</p>
  
  <p>Below is the code that I think is essential. Sadly I cant post the full code or a data sample due to its sensitive nature, but please comment if you need more info.</p>
  
  <pre><code>lstm_size = 512
  numLayers = 4
  numSteps = 15
  
  lstm_cell = tf.nn.rnn_cell.LSTMCell(lstm_size, state_is_tuple=True, forget_bias=1)
  lstm_cell = tf.nn.rnn_cell.DropoutWrapper(lstm_cell, input_keep_prob=input_keep_prob)
  cell = tf.nn.rnn_cell.MultiRNNCell([lstm_cell] * numLayers, state_is_tuple=True)
  
  _inputs = [tf.squeeze(s, [1]) for s in tf.split(1, numSteps, input_data)]
  (outputs, state) = rnn.rnn(cell, _inputs, dtype=tf.float32)
  outputs = tf.pack(outputs)
  
  #transpose so I can put all timesteps through the softmax at once
  outputsTranspose = tf.reshape(outputs, [-1, lstm_size])
  
  softmax_w = tf.get_variable("softmax_w", [lstm_size, nof_classes])                         
  softmax_b = tf.get_variable("softmax_b", [nof_classes]) 
  
  logits = tf.matmul(outputsTranspose, softmax_w) + softmax_b
  
  loss = tf.nn.sparse_softmax_cross_entropy_with_logits(logits, targets)
  cost = tf.reduce_mean(loss)
  
  targetPrediction = tf.argmax(logits, 1)
  accuracy = tf.reduce_mean(tf.cast(tf.equal(targetPrediction, targets), "float"))
  
  """Optimizer"""
  with tf.name_scope("Optimizer") as scope:
      tvars = tf.trainable_variables()
  
      #We clip the gradients to prevent explosion
      grads, _ = tf.clip_by_global_norm(tf.gradients(cost, tvars),maxGradNorm)
      optimizer = tf.train.AdamOptimizer(learning_rate)
      gradients = zip(grads, tvars)
      train_op = optimizer.apply_gradients(gradients)
  
  
  with tf.Session() as sess:
      sess.run(init_op);
  
      for i in range(nofBatches * nofEpochs):
          example_batch, label_batch = sess.run(readTrainDataOp)
  
          result = sess.run([train_op, accuracy, trainSummaries], feed_dict = {input_data: example_batch, targets: label_batch, input_keep_prob:trainInputKeepProbability, batch_size_ph:batch_size})
          #logging
  
          if i % 50 == 0:
              runTestSet()
              #relevant part of runTestSet(): 
              #result = sess.run([cost, accuracy], feed_dict = {input_data: testData, targets: testLabels, input_keep_prob:testInputKeepProbability, batch_size_ph:testBatchSize})
              #logging
  </code></pre>
  
  <p>What am I doing wrong that creates this unexpected behaviour?</p>
  
  <p>EDIT: <s>Here is an image of what an input sample looks like</s> See next link.</p>
  
  <p>The problem also presists with only 1 layer.</p>
  
  <p>EDIT: <a href="https://www.dropbox.com/sh/v68530h790vfk2c/AAAWZ7EsDIVF8dZ87h41-3ipa?dl=0" rel="nofollow">I made an example that reproduces the problem</a>. Just run the python script with the path to test_samples.npy as first argument and the path to the checkpoint as second argument.</p>