title: Tensorflow LSTM Dropout Implementation
body: <ul>
  <li>How specifically does tensorflow apply <em>dropout</em> when calling tf.nn.rnn_cell.DropoutWrapper() ?</li>
  </ul>
  
  <p>Everything I read about applying dropout to rnn's references this paper by <a href="https://arxiv.org/pdf/1409.2329.pdf" rel="noreferrer">Zaremba et. al</a> which says don't apply dropout between recurrent connections. Neurons should be dropped out randomly before or after LSTM layers, but not inter-LSTM layers. Ok.</p>
  
  <ul>
  <li>The question I have is how are the neurons turned off with respect to time? </li>
  </ul>
  
  <p>In the paper that everyone cites, it seems that a random 'dropout mask' is applied at each timestep, rather than generating one random 'dropout mask' and reusing it, applying it to all the timesteps in a given layer being dropped out. Then generating a new 'dropout mask' on the next batch.</p>
  
  <p>Further, and probably what matters more at the moment, how does tensorflow do it? I've checked the tensorflow api and tried searching around for a detailed explanation but have yet to find one. </p>
  
  <ul>
  <li>Is there a way to dig into the actual tensorflow source code?</li>
  </ul>