title: How to use Dropout and BatchNormalization in LSTM Networks (Keras)
body: <p>I am using LSTM Networks for Multivariate Multi-Timestep predictions.
  So basically <code>seq2seq</code> prediction where a number of <code>n_inputs</code> is fed into the model in order to predict a number of <code>n_outputs</code> of a time series.</p>
  
  <p>My question is how to meaningfully apply <code>Dropout</code> and <code>BatchnNormalization</code> as this appears to be a highly discussed topic for Recurrent and therefore LSTM Networks. Lets stick to Keras as framework for the sake of simplicity.</p>
  
  <p><strong>Case 1: Vanilla LSTM</strong></p>
  
  <pre><code>model = Sequential()
  model.add(LSTM(n_blocks, activation=activation, input_shape=(n_inputs, n_features), dropout=dropout_rate))
  model.add(Dense(int(n_blocks/2)))
  model.add(BatchNormalization())
  model.add(Activation(activation))
  model.add(Dense(n_outputs))
  </code></pre>
  
  <ul>
  <li>Q1: Is it good practice not to use BatchNormalization directly after
  LSTM layers? </li>
  <li>Q2: Is it good practice to use Dropout inside LSTM
  layer? </li>
  <li>Q3: Is the usage of BatchNormalization and Dropout between the
  Dense layers good practice?</li>
  <li>Q4: If I stack multiple LSTM layers, is it a good idea to use BatchNormalization between them?</li>
  </ul>
  
  <p><strong>Case 2: Encoder Decoder like LSTM with TimeDistributed Layers</strong></p>
  
  <pre><code>model = Sequential()
  model.add(LSTM(n_blocks, activation=activation, input_shape=(n_inputs,n_features), dropout=dropout_rate))
  model.add(RepeatVector(n_outputs))
  model.add(LSTM(n_blocks, activation=activation, return_sequences=True, dropout=dropout_rate))
  model.add(TimeDistributed(Dense(int(n_blocks/2)), use_bias=False))
  model.add(TimeDistributed(BatchNormalization()))
  model.add(TimeDistributed(Activation(activation)))
  model.add(TimeDistributed(Dropout(dropout_rate)))
  model.add(TimeDistributed(Dense(1)))
  </code></pre>
  
  <ul>
  <li>Q5: Should <code>BatchNormalozation</code> and <code>Dropout</code> wrapped inside <code>TimeDistributed</code> layers when used between <code>TimeDistributed(Dense())</code> layers, or is it correct to leave them without?</li>
  <li>Q6: Can or should Batchnormalization be applied after, before, or in between the Encoder-Decoder LSTM Blocks?</li>
  <li><p>Q7: If a <code>ConvLSTM2D</code> layer is used as first Layer (Encoder) would this make a difference in the usage of Dropout and BatchNormalization?</p></li>
  <li><p>Q8: should the <code>recurrent_dropout</code> argument be used inside LSTM blocks? If yes should it be combined with normal <code>dropout</code> argument as it is in the example, or should it be exchanged?
  Thank you very much in advance!</p></li>
  </ul>